Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions docs/memory-index-rebuild-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# 保留并重建会话记忆

## 问题

保存新的 Embedding 模型或手动重建索引时,`KnowledgeIndexService.invalidateUser`
原先删除当前用户的全部 `memory_vectors`。重建只处理画像薄弱点、个人文档和知识库,
没有重建 `session_summary` / `insight`,因此重建成功后仍然缺少历史会话检索。
同时,个人文档的向量被删除,但文档仍显示 `ready` 和旧分块数。

最近的提取缓存可能仍保留摘要,待处理快照也可能保留正文;但现有索引重建不会
使用它们,已完成会话的投影重试也不会重建被删除的向量。本文修复继续保留的
会话记忆,不承诺恢复旧版本已删除的数据。

## 实现

1. 在同一个 SQLite 事务内,保留当前用户的摘要和洞察行,通过现有 `metadata`
标记向量失效;清理其他派生向量和题目向量缓存,并把个人文档标为
`needs_reindex`,分块数清零。数据库没有个人文档表时也可正常失效。
先完成失效再保存新模型,失效失败时继续使用原来的 provider 配置。
2. `listProfileMemories` 跳过失效行,防止同维度模型切换后误用旧向量。
失效标记写入数据库,服务重启后仍生效。
3. 重建流程新增“历史会话记忆”步骤。读取保留的正文,按会话调用 Embedding,
验证数量、非空维度、同批维度一致性和有限数值。旧行没有会话 ID 时逐行处理。
4. 每个会话在单独事务内原位更新向量并移除失效标记,不删除重插,保留行 ID、
正文、主题、会话归属、其他元数据和原始时间。重建不重跑画像提取,不改变
统计、掌握度和 SM-2。
5. 每次失效使用不同标记。更新时比较读取时的元数据版本,若期间再次失效或
原行被替换/删除,整批回滚并报告重试,避免旧重建重新启用失效向量。
6. Embedding 或 SQLite 失败时保留正文,可以再次重建。清理阶段失败则终止,
返回明确的 fatal 事件;其他步骤继续沿用原来的逐步报错语义。
7. SSE 完成事件增加 `rebuilt.session_memories`,同步更新契约、生成文件与 fixture。
设置页显示步骤失败,避免随后收到 done 就显示“已重建”;更新保留正文的提示。

使用原位更新代替删除再插入,是为了只更新派生的 embedding,同时保持来源、
时间与行身份不变,并能拒绝失效版本的写入。不新增数据库表或列。

## 验收

测试样本每个会话包含一个摘要、一条洞察和一个薄弱点,N 为该样本的会话数。
其他合法提取可能没有摘要,生产环境不要求固定 2N 行。

| 阶段 | 摘要 + 洞察行数 | 历史语义检索 | 个人文档状态 |
| --- | --- | --- | --- |
| 切换前 | 2N | 命中 | `ready` |
| 切换后、重建前 | 2N,失效 | 空 | `needs_reindex`,0 分块 |
| 成功重建后 | 2N,新向量,正文与原始时间不变 | 恢复 | `ready` |
| 记忆步骤失败后 | 2N,失败会话仍失效 | 不使用失败会话的旧向量 | 由文档重建步骤决定 |
| 再次重建成功后 | 2N,无重复行 | 恢复 | `ready` |

运行,不需要 API Key:

```bash
bun test tests-ts/memory-index-rebuild.test.ts
bun run gen:api
bun run check
```

17 个定向用例使用临时文件、真实 SQLite、真实画像和个人文档仓库,通过 Hono
`app.request()` 触发 SSE。覆盖同维度/不同维度切换、手动和重复重建、用户隔离、
服务重启后的失效状态、缺少会话 ID、无历史数据、异常 Embedding、SQLite 事务
回滚、并发再次失效、删除后的写入、取消和鉴权。

## 边界

- 本文说明 Embedding 设置变化与手动索引重建,不涵盖个人归档导入流程。
- 继续沿用现有模型变化判定;完整模型签名、同名模型更换 API 地址和维度身份
未引入新的检测机制。失效标记不能检测未经过失效流程的向量空间变化。
- 保留摘要/洞察正文;`weak_point` 继续从画像重建。
- 保持逐步骤、逐会话提交语义。失败前已完成的会话可以检索,失败会话保持失效;
再次重建会重新处理保留的全部正文。
- 完成事件的 `last_rebuild_at` 延续已有的尝试结束时间语义,不代表所有步骤成功;
调用方应检查步骤错误与 `rebuilt` 标志。新增字段需要客户端与服务端配套升级。
2 changes: 1 addition & 1 deletion docs/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ DeepSeek 模式提供 `none`、`low`、`high`、`max`,保留 JSON 输出配置
- **API**:填写 Base、Key、模型和批大小;完整的 `/v1/embeddings` 地址会自动归一化。
- **本地**:填写 Transformers.js/ONNX 模型 ID 或本地路径;默认 `Xenova/bge-m3`,首次测试自动下载。

更换模型会让已有向量失效。保存后使用“重建索引”,等待知识库、个人资料和记忆向量全部完成;简历面试读取全文,不依赖 Embedding。
更换模型会让已有向量失效。会话摘要与洞察正文会保留,重建成功前不参与检索;个人文档显示“需要重建”。保存后使用“重建索引”,等待历史会话记忆、薄弱点、个人资料和知识库全部完成。失败步骤会显示错误,可再次重建;简历面试读取全文,不依赖 Embedding。

## 服务密钥

Expand Down
1 change: 1 addition & 0 deletions frontend/src/api/schema.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4763,6 +4763,7 @@ export interface components {
/** @constant */
done: true;
rebuilt: {
session_memories: boolean;
weak_points: boolean;
personal_documents: boolean;
topics: string[];
Expand Down
13 changes: 11 additions & 2 deletions frontend/src/pages/Settings.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -464,10 +464,19 @@ export default function Settings() {
setReindexError("");
setReindexDone(false);
setReindexProgress(null);
let rebuildFailed = false;
try {
await rebuildEmbeddingIndex({
onProgress: (p) => setReindexProgress(p),
onProgress: (p) => {
setReindexProgress(p);
if (p.status === "error") {
rebuildFailed = true;
setNeedsReindex(true);
setReindexError(`${p.label}重建失败: ${p.error}`);
}
},
onDone: (d) => {
if (rebuildFailed) return;
setNeedsReindex(false);
setReindexProgress(null);
setReindexDone(true);
Expand Down Expand Up @@ -753,7 +762,7 @@ export default function Settings() {
</div>
<div className="text-[13px] text-dim mb-6">
你自己的 Embedding,仅对你生效;用于题库、知识库、个人资料库和记忆的向量化,必须配置。简历会直接读取全文,不使用 Embedding。
<span className="text-amber-500/90">更换模型后请点下方「更新向量索引」重建(会清空并重算向量,历史会话记忆向量无法恢复)。</span>
<span className="text-amber-500/90">更换模型后请点下方「更新向量索引」重建。历史会话记忆正文会保留,重建成功后恢复检索。</span>
</div>

<div className="space-y-2.5 mb-5">
Expand Down
4 changes: 4 additions & 0 deletions packages/contracts/openapi.json
Original file line number Diff line number Diff line change
Expand Up @@ -2494,6 +2494,9 @@
"rebuilt": {
"type": "object",
"properties": {
"session_memories": {
"type": "boolean"
},
"weak_points": {
"type": "boolean"
},
Expand All @@ -2508,6 +2511,7 @@
}
},
"required": [
"session_memories",
"weak_points",
"personal_documents",
"topics"
Expand Down
2 changes: 1 addition & 1 deletion packages/contracts/src/index-rebuild-events.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ export const IndexRebuildProgressSchema = z.discriminatedUnion('status', [
])
export const IndexRebuildDoneSchema = z.object({
done: z.literal(true),
rebuilt: z.object({ weak_points: z.boolean(), personal_documents: z.boolean(), topics: z.array(z.string()) }).strict(),
rebuilt: z.object({ session_memories: z.boolean(), weak_points: z.boolean(), personal_documents: z.boolean(), topics: z.array(z.string()) }).strict(),
// Preserve the current second-precision timestamp without adding a timezone.
last_rebuild_at: z.string(),
}).strict()
Expand Down
2 changes: 1 addition & 1 deletion packages/core/src/knowledge/index-service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ export class KnowledgeIndexService implements KnowledgeIndex, KnowledgeQuery, Ve
}

async invalidateUser(userId: string): Promise<void> {
await Promise.all([this.vectors.deleteChunks(userId), this.vectors.clearQuestionEmbeddings(userId)])
await this.vectors.invalidateUserVectors(userId)
}

invalidateTopic(userId: string, topic: string): Promise<void> {
Expand Down
6 changes: 6 additions & 0 deletions packages/core/src/knowledge/ports.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ export interface KnowledgeIndex {
}

export type VectorChunk = { content: string; embedding: Float32Array }
/** The opaque version prevents a rebuild from overwriting a later invalidation. */
export type SessionMemoryChunk = { id: number; sessionId: string | null; content: string; version: string | null }
export type DrillSessionProjection = {
sessionId: string
questions: Array<Record<string, unknown>>
Expand All @@ -33,6 +35,10 @@ export type DrillSessionProjection = {

export interface KnowledgeVectorRepository {
initialize(): void
/** Invalidate embeddings while retaining session memory source text. */
invalidateUserVectors(userId: string): Promise<void>
listSessionMemoryChunks(userId: string): Promise<SessionMemoryChunk[]>
updateSessionMemoryEmbeddings(userId: string, chunks: readonly { id: number; version: string | null; embedding: Float32Array }[]): Promise<void>
replaceChunks(input: { userId: string; chunkType: string; topic?: string; chunks: Array<{ content: string; source: string; embedding: Float32Array }> }): Promise<void>
listChunks(userId: string, chunkType: string, topic?: string): Promise<VectorChunk[]>
deleteChunks(userId: string, chunkType?: string, topic?: string): Promise<void>
Expand Down
2 changes: 1 addition & 1 deletion packages/core/src/provider/events.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@ export type IndexRebuildProgress = {

export type IndexRebuildEvent =
| IndexRebuildProgress
| { done: true; rebuilt: { weak_points: boolean; personal_documents: boolean; topics: string[] }; last_rebuild_at: string }
| { done: true; rebuilt: { session_memories: boolean; weak_points: boolean; personal_documents: boolean; topics: string[] }; last_rebuild_at: string }
| { fatal: true; error: string }
31 changes: 27 additions & 4 deletions packages/core/src/provider/settings-operations.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
import type { IndexRebuildEvent } from './events.ts'
import { AuthenticationError } from '../kernel/errors.ts'
import { assertContextActive } from '../kernel/context.ts'
import { parseJsonResponse } from '../kernel/json.ts'
import { finiteNumber, identifier, record, requiredText, uniqueIds, StructuredOutputError } from '../kernel/structured-output.ts'
import type { RequestContext } from '../kernel/context.ts'
import type { KnowledgeVectorRepository, KnowledgeStore } from '../knowledge/ports.ts'
import type { KnowledgeVectorRepository, KnowledgeStore, SessionMemoryChunk } from '../knowledge/ports.ts'
import type { PersonalAgentUseCases } from '../personal-agent/ports.ts'
import type { ProfileUseCases } from '../profile/ports.ts'
import { embeddingModeOf, normalizeEmbeddingSettings } from './config.ts'
Expand Down Expand Up @@ -92,15 +93,32 @@ export class SettingsOperationsService implements SettingsOperationsUseCases {
const id = userId(context)
try {
const topics = await this.deps.knowledge.loadTopics(id)
const plan: Array<[string, string]> = [['cleanup', '清理旧向量'], ['weak_points', '记忆 / 薄弱点']]
const plan: Array<[string, string]> = [['cleanup', '清理旧向量'], ['session_memories', '历史会话记忆'], ['weak_points', '记忆 / 薄弱点']]
if (await this.deps.personal.hasDocuments(context)) plan.push(['personal_documents', '个人资料库'])
for (const [key, topic] of Object.entries(topics)) plan.push([`topic:${key}`, `知识库 · ${topic.name || key}`])
const result: { weak_points: boolean; personal_documents: boolean; topics: string[] } = { weak_points: false, personal_documents: false, topics: [] }
const result = { session_memories: false, weak_points: false, personal_documents: false, topics: [] as string[] }
let completed = 0
for (const [key, label] of plan) {
yield { completed, total: plan.length, label, status: 'running' }
try {
await assertContextActive(context)
if (key === 'cleanup') await this.deps.index.invalidateUser(id)
else if (key === 'session_memories') {
const sessions = new Map<string, SessionMemoryChunk[]>()
for (const chunk of await this.deps.vectors.listSessionMemoryChunks(id)) {
const key = chunk.sessionId === null ? `row:${chunk.id}` : `session:${chunk.sessionId}`
const entries = sessions.get(key) || []
entries.push(chunk); sessions.set(key, entries)
}
for (const chunks of sessions.values()) {
const vectors = await this.deps.embeddings.embed(context, chunks.map(chunk => chunk.content))
const dimension = vectors[0]?.length || 0
if (vectors.length !== chunks.length || !dimension || vectors.some(vector => vector.length !== dimension || !vector.every(Number.isFinite))) throw new Error('Invalid memory embeddings')
await assertContextActive(context)
await this.deps.vectors.updateSessionMemoryEmbeddings(id, chunks.map((chunk, index) => ({ id: chunk.id, version: chunk.version, embedding: vectors[index]! })))
}
result.session_memories = true
}
else if (key === 'weak_points') {
const profile = await this.deps.profile.get(context)
const points = profile.weak_points.filter((point) => !point.improved && !point.archived).map((point) => `${point.topic || '综合'}: ${point.point}`)
Expand All @@ -110,7 +128,12 @@ export class SettingsOperationsService implements SettingsOperationsUseCases {
} else if (key === 'personal_documents') { await this.deps.personal.reindexAll(context); result.personal_documents = true }
else if (key.startsWith('topic:')) { const topic = key.slice(6); await this.deps.index.rebuildTopic(context, topic); result.topics.push(topic) }
completed += 1; yield { completed, total: plan.length, label, status: 'done' }
} catch (error) { completed += 1; yield { completed, total: plan.length, label, status: 'error', error: error instanceof Error ? error.message : String(error) } }
} catch (error) {
await assertContextActive(context)
completed += 1; yield { completed, total: plan.length, label, status: 'error', error: error instanceof Error ? error.message : String(error) }
// No step may make retained vectors readable if invalidation failed.
if (key === 'cleanup') throw error
}
}
const now = new Date().toISOString().slice(0, 19)
await this.deps.settings.saveLastReindexAt(id, now)
Expand Down
5 changes: 3 additions & 2 deletions packages/core/src/provider/settings-service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,12 @@ export class SettingsService implements SettingsUseCases {
const userId = this.userId(context)
const [stored, user] = await Promise.all([this.repository.loadProvider(userId), this.users.findById(userId)])
const before = embeddingTarget(resolveEmbeddingConfig(stored.embedding, this.platform))
await this.repository.saveProvider(userId, { llm: value.llm, embedding: value.embedding, services: value.services })
this.indexes.resetEmbeddingClient(userId)
const after = embeddingTarget(resolveEmbeddingConfig(value.embedding, this.platform))
const embeddingChanged = before !== after
// Keep the old provider active if invalidating its vectors fails.
if (embeddingChanged) await this.indexes.invalidateUser(userId)
await this.repository.saveProvider(userId, { llm: value.llm, embedding: value.embedding, services: value.services })
this.indexes.resetEmbeddingClient(userId)
if (user?.is_admin) {
await this.repository.saveSystem(value.system)
this.registration.allowRegistration = value.system.allow_registration
Expand Down
39 changes: 37 additions & 2 deletions packages/db/src/knowledge-vector-repository.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { Database } from 'bun:sqlite'
import type { DrillSessionProjection, KnowledgeVectorRepository, ProfileMemoryEntry, ProfileVectorMemoryPort, VectorChunk } from '@techspar/core'
import type { DrillSessionProjection, KnowledgeVectorRepository, ProfileMemoryEntry, ProfileVectorMemoryPort, SessionMemoryChunk, VectorChunk } from '@techspar/core'

function toBlob(vector: Float32Array): Uint8Array {
const bytes = new Uint8Array(vector.length * 4)
Expand Down Expand Up @@ -140,18 +140,53 @@ export class BunKnowledgeVectorRepository implements KnowledgeVectorRepository,
const allowed = input.chunkTypes?.length ? new Set(input.chunkTypes) : undefined
return rows.flatMap((row) => {
if (allowed && !allowed.has(row.chunk_type)) return []
const details = metadata(row.metadata)
if (details.embedding_stale) return []
return [{
chunkType: row.chunk_type,
content: row.content,
...(row.topic ? { topic: row.topic } : {}),
...(row.session_id ? { sessionId: row.session_id } : {}),
metadata: metadata(row.metadata),
metadata: details,
embedding: fromBlob(row.embedding),
createdAt: row.created_at || new Date(0).toISOString(),
}]
})
}

async invalidateUserVectors(userId: string): Promise<void> {
this.sqlite.transaction(() => {
// Session text is source data, not a disposable index. Keep its original
// attribution and timestamps, but prevent stale embeddings from being read.
this.sqlite.query(`UPDATE memory_vectors
SET metadata = json_set(CASE WHEN json_valid(metadata) AND json_type(metadata) = 'object' THEN metadata ELSE '{}' END, '$.embedding_stale', $generation)
WHERE user_id = $userId AND chunk_type IN ('session_summary', 'insight')`).run({ $userId: userId, $generation: crypto.randomUUID() })
this.sqlite.query("DELETE FROM memory_vectors WHERE user_id = $userId AND chunk_type NOT IN ('session_summary', 'insight')").run({ $userId: userId })
this.sqlite.query('DELETE FROM question_embeddings WHERE user_id = $userId').run({ $userId: userId })
const hasDocuments = this.sqlite.query("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'personal_documents'").get()
if (hasDocuments) this.sqlite.query("UPDATE personal_documents SET status = 'needs_reindex', chunk_count = 0, error = NULL, updated_at = CURRENT_TIMESTAMP WHERE user_id = $userId").run({ $userId: userId })
})()
}

async listSessionMemoryChunks(userId: string): Promise<SessionMemoryChunk[]> {
return this.sqlite.query<SessionMemoryChunk, { $userId: string }>(`
SELECT id, session_id AS sessionId, content, metadata AS version FROM memory_vectors
WHERE user_id = $userId AND chunk_type IN ('session_summary', 'insight') ORDER BY id
`).all({ $userId: userId })
}

async updateSessionMemoryEmbeddings(userId: string, chunks: readonly { id: number; version: string | null; embedding: Float32Array }[]): Promise<void> {
this.sqlite.transaction(() => {
const statement = this.sqlite.query(`UPDATE memory_vectors
SET embedding = $embedding, metadata = json_remove(CASE WHEN json_valid(metadata) AND json_type(metadata) = 'object' THEN metadata ELSE '{}' END, '$.embedding_stale')
WHERE id = $id AND user_id = $userId AND metadata IS $version AND chunk_type IN ('session_summary', 'insight')`)
for (const chunk of chunks) {
const result = statement.run({ $id: chunk.id, $userId: userId, $version: chunk.version, $embedding: toBlob(chunk.embedding) })
if (result.changes !== 1) throw new Error('Session memory changed during index rebuild; retry the rebuild')
}
})()
}

async deleteChunks(userId: string, chunkType?: string, topic?: string): Promise<void> {
if (!chunkType) {
this.sqlite.query('DELETE FROM memory_vectors WHERE user_id = $userId').run({ $userId: userId })
Expand Down
Loading
Loading