338 lines
30 KiB
Markdown
338 lines
30 KiB
Markdown
# Findings
|
||
|
||
Last Updated: 2026-04-01T12:05:23.000Z
|
||
|
||
## Initializer Summary
|
||
|
||
- JIRA: TRUEREF-0023
|
||
- Refresh mode: REFRESH_IF_REQUIRED
|
||
- Result: Refreshed ARCHITECTURE.md and FINDINGS.md. CODE_STYLE.md remained trusted — sqlite-vec, worker-status, and write-worker additions follow the established conventions already documented.
|
||
|
||
## Research Performed
|
||
|
||
- Counted 149 TypeScript/JavaScript source files in the repository-wide scan and verified the live, non-generated source mix as 147 `.ts` files and 2 `.js` files.
|
||
- Read `package.json`, `.prettierrc`, and `eslint.config.js` to verify dependencies, formatting rules, and linting conventions.
|
||
- Read `sqlite-vec.ts`, `sqlite-vec.store.ts`, `vector.search.ts`, `hybrid.search.service.ts`, `schema.ts`, `client.ts`, and startup wiring to verify the accepted sqlite-vec implementation and current retrieval architecture.
|
||
- Read `worker-pool.ts`, `worker-types.ts`, `write-worker-entry.ts`, and `/api/v1/workers/+server.ts` to verify the current worker topology and status surface.
|
||
- Compared `docs/docs_cache_state.yaml` against the live docs and codebase to identify stale cache evidence and architecture drift.
|
||
- Confirmed `CODE_STYLE.md` still matches the codebase: tabs, single quotes, `trailingComma: none`, ESM imports with `node:` built-ins, flat ESLint config, and descriptive PascalCase/camelCase naming remain consistent.
|
||
|
||
## Open Questions For Planner
|
||
|
||
- Verify whether the write-worker protocol should become part of the active indexing flow or remain documented as optional infrastructure only.
|
||
- Verify whether worker-status and SSE event payloads should be documented in a dedicated API reference for external consumers.
|
||
- Verify whether sqlite-vec operational details such as per-profile vec-table lifecycle and backfill behavior should move into a separate persistence document if the subsystem grows further.
|
||
- Assess whether the WorkerPool fallback mode (main-thread execution when worker scripts are missing) still belongs in the runtime contract or should be removed in favour of a hard build requirement.
|
||
|
||
## Planner Notes Template
|
||
|
||
Add subsequent research below this section.
|
||
|
||
### Entry Template
|
||
|
||
- Date:
|
||
- Task:
|
||
- Files inspected:
|
||
- Findings:
|
||
- Risks / follow-ups:
|
||
|
||
### 2026-04-01 — TRUEREF-0023 initializer refresh audit
|
||
|
||
- Task: Refresh only stale or invalid documentation after the accepted sqlite-vec implementation.
|
||
- Files inspected:
|
||
- `docs/docs_cache_state.yaml`
|
||
- `docs/ARCHITECTURE.md`
|
||
- `docs/CODE_STYLE.md`
|
||
- `docs/FINDINGS.md`
|
||
- `package.json`
|
||
- `.prettierrc`
|
||
- `eslint.config.js`
|
||
- `src/hooks.server.ts`
|
||
- `src/lib/server/db/client.ts`
|
||
- `src/lib/server/db/schema.ts`
|
||
- `src/lib/server/db/sqlite-vec.ts`
|
||
- `src/lib/server/search/sqlite-vec.store.ts`
|
||
- `src/lib/server/search/vector.search.ts`
|
||
- `src/lib/server/search/hybrid.search.service.ts`
|
||
- `src/lib/server/pipeline/startup.ts`
|
||
- `src/lib/server/pipeline/worker-pool.ts`
|
||
- `src/lib/server/pipeline/worker-types.ts`
|
||
- `src/lib/server/pipeline/write-worker-entry.ts`
|
||
- `src/routes/api/v1/workers/+server.ts`
|
||
- `scripts/build-workers.mjs`
|
||
- Findings:
|
||
- The trusted cache metadata was no longer reliable as evidence for planning: `docs/docs_cache_state.yaml` still referenced 2026-03-27 hashes while `ARCHITECTURE.md` and `FINDINGS.md` had been edited later.
|
||
- `ARCHITECTURE.md` was stale. It still described only parse and embed worker concurrency, omitted the `sqlite-vec` production dependency, and did not document the current per-profile vec-table storage layer, worker-status endpoint, or write-worker infrastructure.
|
||
- The current retrieval stack uses sqlite-vec concretely: `loadSqliteVec()` bootstraps connections, `SqliteVecStore` manages vec0 tables plus rowid mapping tables, and `VectorSearch` delegates nearest-neighbor lookup to that store instead of brute-force scoring.
|
||
- The worker architecture now includes parse, embed, and write worker protocols in `worker-types.ts`, build-time bundling for all three entries, and a `/api/v1/workers` route that returns `WorkerPool` status snapshots.
|
||
- `CODE_STYLE.md` remained valid and did not require refresh. The observed source and config files still use tabs, single quotes, `trailingComma: none`, flat ESLint config, ESM imports, PascalCase class names, and camelCase helpers exactly as already documented.
|
||
- `FINDINGS.md` itself was stale because the initializer summary still referred to `TRUEREF-0022` instead of the requested `TRUEREF-0023` refresh.
|
||
- Risks / follow-ups:
|
||
- The write-worker protocol exists and is bundled, but the active indexing path is still centered on parse plus optional embed flow. Future documentation should keep distinguishing implemented infrastructure from the currently exercised path.
|
||
- Cache validity should continue to be driven by deterministic hash evidence rather than document timestamps or trust text alone.
|
||
|
||
### 2026-03-27 — FEEDBACK-0001 initializer refresh audit
|
||
|
||
- Task: Refresh only stale documentation after changes to retrieval, formatters, token budgeting, and parser behavior.
|
||
- Files inspected:
|
||
- `docs/docs_cache_state.yaml`
|
||
- `docs/ARCHITECTURE.md`
|
||
- `docs/CODE_STYLE.md`
|
||
- `docs/FINDINGS.md`
|
||
- `package.json`
|
||
- `src/routes/api/v1/context/+server.ts`
|
||
- `src/lib/server/api/formatters.ts`
|
||
- `src/lib/server/api/token-budget.ts`
|
||
- `src/lib/server/search/query-preprocessor.ts`
|
||
- `src/lib/server/search/search.service.ts`
|
||
- `src/lib/server/search/hybrid.search.service.ts`
|
||
- `src/lib/server/mappers/context-response.mapper.ts`
|
||
- `src/lib/server/models/context-response.ts`
|
||
- `src/lib/server/models/search-result.ts`
|
||
- `src/lib/server/parser/index.ts`
|
||
- `src/lib/server/parser/code.parser.ts`
|
||
- `src/lib/server/parser/markdown.parser.ts`
|
||
- Findings:
|
||
- The documentation cache was trusted, but the architecture summary no longer captured current retrieval behavior: query preprocessing now sanitizes punctuation-heavy input for FTS5, semantic mode can bypass FTS entirely, and auto or hybrid retrieval can fall back to vector search when keyword search returns no candidates.
|
||
- Plain-text and JSON context formatting now carry repository and version metadata, and the text formatter emits an explicit no-results section instead of an empty body.
|
||
- Token budgeting now skips individual over-budget snippets and continues evaluating lower-ranked candidates, which changes the response-selection behavior described at the architecture level.
|
||
- Parser coverage now explicitly includes Markdown, code, config, HTML-like, and plain-text inputs, so the architecture summary needed to reflect that broader file-type handling.
|
||
- The conventions documented in CODE_STYLE.md still match the current repository: strict TypeScript, tab indentation, ESM imports, Prettier and ESLint flat config, and pragmatic service-oriented server modules.
|
||
- Risks / follow-ups:
|
||
- Future cache invalidation should continue to distinguish between behavioral changes that affect architecture docs and localized implementation changes that do not affect the style guide.
|
||
- If the public API contract becomes externally versioned, the new context metadata fields likely deserve a dedicated API document instead of only architecture-level coverage.
|
||
|
||
### 2026-03-27 — FEEDBACK-0001 planning research
|
||
|
||
- Task: Plan the retrieval-fix iteration covering FTS query safety, hybrid fallback, empty-result behavior, result metadata, token budgeting, and parser chunking.
|
||
- Files inspected:
|
||
- `package.json`
|
||
- `src/routes/api/v1/context/+server.ts`
|
||
- `src/lib/server/search/query-preprocessor.ts`
|
||
- `src/lib/server/search/search.service.ts`
|
||
- `src/lib/server/search/hybrid.search.service.ts`
|
||
- `src/lib/server/search/vector.search.ts`
|
||
- `src/lib/server/api/token-budget.ts`
|
||
- `src/lib/server/api/formatters.ts`
|
||
- `src/lib/server/mappers/context-response.mapper.ts`
|
||
- `src/lib/server/models/context-response.ts`
|
||
- `src/lib/server/models/search-result.ts`
|
||
- `src/lib/server/parser/code.parser.ts`
|
||
- `src/lib/server/search/search.service.test.ts`
|
||
- `src/lib/server/search/hybrid.search.service.test.ts`
|
||
- `src/lib/server/api/formatters.test.ts`
|
||
- `src/lib/server/parser/code.parser.test.ts`
|
||
- `src/routes/api/v1/api-contract.integration.test.ts`
|
||
- `src/mcp/tools/query-docs.ts`
|
||
- `src/mcp/client.ts`
|
||
- Findings:
|
||
- `better-sqlite3` `^12.6.2` backs the affected search path; the code already uses bound parameters for `MATCH`, so the practical fix belongs in query normalization and fallback handling rather than SQL string construction.
|
||
- `query-preprocessor.ts` only strips parentheses and appends a trailing wildcard. Other code-like punctuation currently reaches the FTS execution path unsanitized.
|
||
- `search.service.ts` sends the preprocessed text directly to `snippets_fts MATCH ?` and already returns `[]` for blank processed queries.
|
||
- `hybrid.search.service.ts` always executes keyword search before semantic branching. In the current flow, an FTS parse failure can abort `auto`, `hybrid`, and `semantic` requests before vector retrieval runs.
|
||
- `vector.search.ts` already preserves `repositoryId`, `versionId`, and `profileId` filtering and does not need architectural changes for this iteration.
|
||
- `token-budget.ts` stops at the first over-budget snippet instead of skipping that item and continuing through later ranked results.
|
||
- `formatContextTxt([], [])` returns an empty string, so `/api/v1/context?type=txt` can emit an empty `200 OK` body today.
|
||
- `context-response.mapper.ts` and `context-response.ts` expose snippet content and breadcrumb/page title but do not identify local TrueRef origin, repository source metadata, or normalized snippet origin labels.
|
||
- `code.parser.ts` splits primarily at top-level declarations; class/object member functions remain in coarse chunks, which limits method-level recall for camelCase API queries.
|
||
- Existing relevant automated coverage is concentrated in the search, formatter, and parser unit tests; `/api/v1/context` contract coverage currently omits the context endpoint entirely.
|
||
- Risks / follow-ups:
|
||
- Response-shape changes must be additive because `src/mcp/client.ts`, `src/mcp/tools/query-docs.ts`, and UI consumers expect the current top-level keys to remain present.
|
||
- Parser improvements should stay inside `parseCodeFile()` and existing chunking helpers to avoid turning this fix iteration into a schema or pipeline redesign.
|
||
|
||
### 2026-03-27 — FEEDBACK-0001 SQLite FTS5 syntax research
|
||
|
||
- Task: Verify the FTS5 query-grammar constraints that affect punctuation-heavy local search queries.
|
||
- Files inspected:
|
||
- `package.json`
|
||
- `src/lib/server/search/query-preprocessor.ts`
|
||
- `src/lib/server/search/search.service.ts`
|
||
- `src/lib/server/search/hybrid.search.service.ts`
|
||
- Findings:
|
||
- `better-sqlite3` is pinned at `^12.6.2` in `package.json`, and the application binds the `MATCH` string as a parameter instead of interpolating SQL directly.
|
||
- The canonical SQLite FTS5 docs state that barewords may contain letters, digits, underscore, non-ASCII characters, and the substitute character; strings containing other punctuation must be quoted or they become syntax errors in `MATCH` expressions.
|
||
- The same docs state that prefix search is expressed by placing `*` after the token or phrase, not inside quotes, which matches the current trailing-wildcard strategy in `query-preprocessor.ts`.
|
||
- SQLite documents that FTS5 is stricter than FTS3/4 about unrecognized punctuation in query strings, which confirms that code-like user input should be normalized before it reaches `snippets_fts MATCH ?`.
|
||
- Based on the current code path, the practical fix remains application-side sanitization and fallback behavior in `query-preprocessor.ts` and `hybrid.search.service.ts`, not SQL construction changes.
|
||
- Risks / follow-ups:
|
||
- Over-sanitizing punctuation-heavy inputs could erase useful identifiers, so the implementation should preserve searchable alphanumeric and underscore tokens while discarding grammar-breaking punctuation.
|
||
- Prefix expansion should remain on the final searchable token only so the fix preserves current query-cost expectations and test semantics.
|
||
|
||
### 2026-03-27 — LINT-0001 planning research
|
||
|
||
- Task: Plan the lint-fix iteration covering the reported ESLint and eslint-plugin-svelte violations across Svelte UI, SvelteKit routes, server modules, and Vitest suites.
|
||
- Files inspected:
|
||
- `package.json`
|
||
- `eslint.config.js`
|
||
- `docs/FINDINGS.md`
|
||
- `prompts/LINT-0001/prompt.yaml`
|
||
- `prompts/LINT-0001/progress.yaml`
|
||
- `src/lib/components/FolderPicker.svelte`
|
||
- `src/lib/components/RepositoryCard.svelte`
|
||
- `src/lib/components/search/SnippetCard.svelte`
|
||
- `src/lib/server/crawler/local.crawler.test.ts`
|
||
- `src/lib/server/embeddings/embedding.service.test.ts`
|
||
- `src/lib/server/embeddings/local.provider.ts`
|
||
- `src/lib/server/embeddings/provider.ts`
|
||
- `src/lib/server/embeddings/registry.ts`
|
||
- `src/lib/server/models/context-response.ts`
|
||
- `src/lib/server/parser/code.parser.ts`
|
||
- `src/lib/server/pipeline/indexing.pipeline.ts`
|
||
- `src/lib/server/search/hybrid.search.service.test.ts`
|
||
- `src/lib/server/search/query-preprocessor.ts`
|
||
- `src/lib/server/services/repository.service.test.ts`
|
||
- `src/lib/server/services/version.service.test.ts`
|
||
- `src/lib/server/services/version.service.ts`
|
||
- `src/routes/+layout.svelte`
|
||
- `src/routes/+page.svelte`
|
||
- `src/routes/api/v1/libs/search/+server.ts`
|
||
- `src/routes/api/v1/settings/embedding/+server.ts`
|
||
- `src/routes/repos/[id]/+page.svelte`
|
||
- `src/routes/search/+page.svelte`
|
||
- `src/routes/settings/+page.svelte`
|
||
- Findings:
|
||
- The project lint stack is ESLint `^9.39.2` with `typescript-eslint` recommended rules and `eslint-plugin-svelte` recommended plus SvelteKit-aware rules, running over Svelte `^5.51.0` and SvelteKit `^2.50.2`.
|
||
- Context7 documentation for `eslint-plugin-svelte` confirms `svelte/no-navigation-without-base` flags root-relative `<a href="/...">` links and `goto('/...')` calls in SvelteKit projects; compliant fixes must use `$app/paths` base-aware links or base-prefixed `goto` calls.
|
||
- Context7 documentation for Svelte 5 confirms event handlers are regular element properties such as `onclick`, while side effects belong in `$effect`; repo memory also records that client-only fetch bootstrap should not be moved indiscriminately into `$effect` when `onMount` or load is the correct lifecycle boundary.
|
||
- Concrete navigation violations already exist in `src/routes/+layout.svelte`, `src/routes/repos/[id]/+page.svelte`, `src/routes/search/+page.svelte`, and `src/lib/components/RepositoryCard.svelte`, each using hard-coded root-relative internal navigation.
|
||
- Static diagnostics currently expose at least one direct TypeScript lint error in `src/lib/server/embeddings/registry.ts`, where `_config` is defined but never used.
|
||
- `src/routes/api/v1/libs/search/+server.ts` imports `json` from `@sveltejs/kit` without using it, making that endpoint a concrete unused-import cleanup target.
|
||
- `src/lib/server/services/version.service.ts` still uses CommonJS `require(...)` to reach git utilities from TypeScript, which is inconsistent with the repository's ESM style and is a likely lint target under the current ESLint stack.
|
||
- The affected Svelte pages and settings UI already use Svelte 5 event-property syntax, so the lint work should preserve that syntax and focus on base-aware navigation, lifecycle correctness, and unused-symbol cleanup rather than regressing to legacy `on:` directives.
|
||
- Existing automated coverage for the lint-touching backend areas already lives in `src/lib/server/crawler/local.crawler.test.ts`, `src/lib/server/embeddings/embedding.service.test.ts`, `src/lib/server/search/hybrid.search.service.test.ts`, `src/lib/server/services/repository.service.test.ts`, and `src/lib/server/services/version.service.test.ts`; route and component changes rely on build and lint validation rather than dedicated browser tests in this iteration.
|
||
- Risks / follow-ups:
|
||
- Base-aware navigation fixes must preserve internal app routing semantics and should not replace intentional external navigation, because SvelteKit `goto(...)` no longer accepts external URLs.
|
||
- Settings and search page lifecycle changes must avoid reintroducing SSR-triggered fetches or self-triggered URL loops; client-only bootstrap logic should remain mounted once and URL-sync effects must stay idempotent.
|
||
|
||
### 2026-03-27 — ROUTING-0001 planning research
|
||
|
||
- Task: Plan the repository-detail routing fix for slash-bearing repository IDs causing homepage SSR failures and invalid `/repos/[id]` navigation.
|
||
- Files inspected:
|
||
- `package.json`
|
||
- `src/lib/components/RepositoryCard.svelte`
|
||
- `src/routes/+page.svelte`
|
||
- `src/routes/+page.server.ts`
|
||
- `src/routes/repos/[id]/+page.server.ts`
|
||
- `src/routes/repos/[id]/+page.svelte`
|
||
- `src/routes/api/v1/api-contract.integration.test.ts`
|
||
- `src/lib/types.ts`
|
||
- Findings:
|
||
- The app is on SvelteKit `^2.50.2` and uses `$app/paths.resolve(...)` for internal navigation, including `resolveRoute('/repos/[id]', { id: repo.id })` in `RepositoryCard.svelte`.
|
||
- SvelteKit’s `[id]` route is a single-segment dynamic parameter. Context7 routing docs show slash-containing values belong to rest parameters like `[...param]`, so raw repository IDs containing `/` are invalid inputs for `resolveRoute('/repos/[id]', ...)`.
|
||
- The repository model intentionally stores slash-bearing IDs such as `/facebook/react`, and the existing API surface consistently treats those IDs as percent-encoded path segments. The integration contract already passes `params.id = encodeURIComponent('/facebook/react')` for `/api/v1/libs/[id]` handlers, which then call `decodeURIComponent(params.id)`.
|
||
- The homepage SSR failure is therefore rooted in UI link generation, not repository listing fetches: rendering `RepositoryCard.svelte` with a raw slash-bearing `repo.id` can throw before page load completes, which explains repeated `500` responses on `/`.
|
||
- The repo detail page currently forwards `params.id` directly into `encodeURIComponent(...)` for downstream API requests. Once detail links are generated as encoded single segments, the page loader and client-side refresh/delete/reindex flows need one normalization step so API calls continue targeting the stored repository ID instead of a doubly encoded value.
|
||
- No existing browser-facing test covers homepage card navigation or `/repos/[id]` loader behavior; the closest current evidence is the API contract test file, which already exercises encoded repository IDs on HTTP endpoints and provides reusable fixtures for slash-bearing IDs.
|
||
- Risks / follow-ups:
|
||
- The fix should preserve the existing `/repos/[id]` route shape instead of redesigning it to a rest route unless a broader navigation contract change is explicitly requested.
|
||
- Any normalization helper introduced for the repo detail page should be reused consistently across server load and client event handlers to avoid mixed encoded and decoded repository IDs during navigation and fetches.
|
||
|
||
### 2026-04-01 — TRUEREF-0023 sqlite-vec replanning research
|
||
|
||
- Task: Replan the rejected libSQL-native vector iteration around sqlite-vec using the current worktree and verified runtime constraints.
|
||
- Files inspected:
|
||
- `package.json`
|
||
- `docs/docs_cache_state.yaml`
|
||
- `prompts/TRUEREF-0023/prompt.yaml`
|
||
- `prompts/TRUEREF-0023/progress.yaml`
|
||
- `prompts/TRUEREF-0023/iteration_0/review_report.yaml`
|
||
- `prompts/TRUEREF-0023/iteration_0/plan.md`
|
||
- `prompts/TRUEREF-0023/iteration_0/tasks.yaml`
|
||
- `src/lib/server/db/client.ts`
|
||
- `src/lib/server/db/index.ts`
|
||
- `src/lib/server/db/schema.ts`
|
||
- `src/lib/server/db/fts.sql`
|
||
- `src/lib/server/db/vectors.sql`
|
||
- `src/lib/server/db/schema.test.ts`
|
||
- `src/lib/server/search/vector.search.ts`
|
||
- `src/lib/server/search/hybrid.search.service.test.ts`
|
||
- `src/lib/server/embeddings/embedding.service.ts`
|
||
- `src/lib/server/embeddings/embedding.service.test.ts`
|
||
- `src/lib/server/pipeline/job-queue.ts`
|
||
- `src/lib/server/pipeline/progress-broadcaster.ts`
|
||
- `src/lib/server/pipeline/progress-broadcaster.test.ts`
|
||
- `src/lib/server/pipeline/worker-pool.ts`
|
||
- `src/lib/server/pipeline/worker-entry.ts`
|
||
- `src/lib/server/pipeline/embed-worker-entry.ts`
|
||
- `src/lib/server/pipeline/worker-types.ts`
|
||
- `src/lib/server/pipeline/startup.ts`
|
||
- `src/lib/server/pipeline/indexing.pipeline.ts`
|
||
- `src/lib/server/pipeline/indexing.pipeline.test.ts`
|
||
- `src/routes/api/v1/jobs/+server.ts`
|
||
- `src/routes/api/v1/jobs/stream/+server.ts`
|
||
- `src/routes/api/v1/jobs/[id]/stream/+server.ts`
|
||
- `src/routes/api/v1/sse-and-settings.integration.test.ts`
|
||
- `src/routes/admin/jobs/+page.svelte`
|
||
- `src/lib/components/IndexingProgress.svelte`
|
||
- `src/lib/components/admin/JobStatusBadge.svelte`
|
||
- `src/lib/components/admin/JobSkeleton.svelte`
|
||
- `src/lib/components/admin/Toast.svelte`
|
||
- `src/lib/components/admin/WorkerStatusPanel.svelte`
|
||
- `scripts/build-workers.mjs`
|
||
- `node_modules/libsql/types/index.d.ts`
|
||
- `node_modules/libsql/index.js`
|
||
- Findings:
|
||
- Iteration 0 already changed the workspace materially: direct DB imports were switched from `better-sqlite3` to `libsql`, the extra WAL-related pragmas were added in the main DB clients and embed worker, composite indexes plus `vec_embedding` were added to the Drizzle schema and migration metadata, `IndexingProgress.svelte` now uses SSE, the admin jobs page was overhauled, and `WorkerPool` now serializes on `(repositoryId, versionId)` instead of repository only.
|
||
- The rejected vector implementation is still invalid in the current tree. `src/lib/server/db/vectors.sql` contains the rejected libSQL-native assumptions, including a dangling `USING libsql_vector_idx(...)` clause with no valid `CREATE INDEX` statement, and `src/lib/server/search/vector.search.ts` still performs full-table JS cosine scoring over `snippet_embeddings` instead of true in-database KNN.
|
||
- `sqlite-vec` is not currently present in `package.json` or the lockfile, and there is no existing `sqliteVec.load(...)`, `db.loadExtension(...)`, `vec0`, or extension bootstrap code anywhere under `src/`.
|
||
- Context7 sqlite-vec docs confirm the supported Node integration path is `import * as sqliteVec from 'sqlite-vec'; sqliteVec.load(db);`, storing vectors in a `vec0` virtual table and querying with `WHERE embedding MATCH ? ORDER BY distance LIMIT ?`. The docs also show vec0 metadata columns can be filtered directly, which fits the repositoryId, versionId, and profileId requirements.
|
||
- Context7 `better-sqlite3` v12.6.2 docs confirm `db.loadExtension(path)` exists. The installed `libsql` package in this workspace also exposes `loadExtension(path): this` in `node_modules/libsql/types/index.d.ts` and `loadExtension(...args)` in `node_modules/libsql/index.js`, so extension loading is not obviously blocked by the driver API surface alone.
|
||
- The review report remains the only verified runtime evidence for the current libsql path: `vector_from_float32(...)` is unavailable and `libsql_vector_idx` DDL is rejected in this environment. That invalidates the original native-vector approach but does not by itself prove sqlite-vec extension loading succeeds through the current `libsql` package alias, so the replan must include explicit connection-bootstrap and test coverage for real extension loading on the main DB client and worker-owned connections.
|
||
- Two iteration-0 deliverables referenced in the rejected plan do not exist in the current worktree: `src/lib/server/pipeline/write-worker-entry.ts` and `src/routes/api/v1/workers/+server.ts`. `scripts/build-workers.mjs` and the admin `WorkerStatusPanel.svelte` already reference those missing paths, so iteration 1 must either create them or revert those dangling references as part of a consistent plan.
|
||
- The existing admin/SSE work is largely salvageable. `src/routes/api/v1/jobs/stream/+server.ts`, `src/routes/api/v1/jobs/[id]/stream/+server.ts`, `src/lib/server/pipeline/progress-broadcaster.ts`, `src/lib/components/IndexingProgress.svelte`, `src/lib/components/admin/JobSkeleton.svelte`, `src/lib/components/admin/Toast.svelte`, and `src/lib/components/admin/WorkerStatusPanel.svelte` provide a usable foundation, but `src/routes/admin/jobs/+page.svelte` still contains `confirm(...)` and the queue API still only supports exact `repository_id = ?` and single-status filtering.
|
||
- The existing tests still encode the rejected pre-sqlite-vec model: `embedding.service.test.ts`, `schema.test.ts`, `hybrid.search.service.test.ts`, and `indexing.pipeline.test.ts` seed and assert against `snippet_embeddings.embedding` blobs only. The sqlite-vec replan therefore needs new DB bootstrap helpers, vec-table lifecycle assertions, and vector-search tests that validate actual vec0 writes and filtered KNN queries.
|
||
- Risks / follow-ups:
|
||
- The current worktree is dirty with iteration-0 partial changes and generated migration metadata, so iteration-1 tasks must explicitly distinguish keep/revise/revert work to avoid sibling tasks fighting over the same files.
|
||
- Because the current `libsql` package appears to expose `loadExtension`, the replan should avoid assuming an immediate full revert to upstream `better-sqlite3`; instead it should sequence a driver/bootstrap compatibility decision around actual sqlite-vec extension loading behavior with testable acceptance criteria.
|
||
|
||
### 2026-04-01 — TRUEREF-0023 iteration-2 current-worktree verification
|
||
|
||
- Task: Replan iteration 2 against the post-iteration-1 workspace state so the first validation unit no longer leaves a known vec_embedding mismatch behind.
|
||
- Files inspected:
|
||
- `package.json`
|
||
- `package-lock.json`
|
||
- `scripts/build-workers.mjs`
|
||
- `src/lib/server/db/client.ts`
|
||
- `src/lib/server/db/index.ts`
|
||
- `src/lib/server/db/schema.ts`
|
||
- `src/lib/server/db/vectors.sql`
|
||
- `src/lib/server/db/migrations/0006_yielding_centennial.sql`
|
||
- `src/lib/server/db/schema.test.ts`
|
||
- `src/lib/server/embeddings/embedding.service.ts`
|
||
- `src/lib/server/embeddings/embedding.service.test.ts`
|
||
- `src/lib/server/search/vector.search.ts`
|
||
- `src/lib/server/search/hybrid.search.service.ts`
|
||
- `src/lib/server/search/hybrid.search.service.test.ts`
|
||
- `src/lib/server/pipeline/job-queue.ts`
|
||
- `src/lib/server/pipeline/worker-pool.ts`
|
||
- `src/lib/server/pipeline/worker-entry.ts`
|
||
- `src/lib/server/pipeline/embed-worker-entry.ts`
|
||
- `src/lib/server/pipeline/worker-types.ts`
|
||
- `src/lib/server/pipeline/indexing.pipeline.ts`
|
||
- `src/lib/server/pipeline/indexing.pipeline.test.ts`
|
||
- `src/lib/server/pipeline/startup.ts`
|
||
- `src/lib/server/pipeline/progress-broadcaster.ts`
|
||
- `src/routes/api/v1/jobs/+server.ts`
|
||
- `src/routes/api/v1/jobs/stream/+server.ts`
|
||
- `src/routes/api/v1/sse-and-settings.integration.test.ts`
|
||
- `src/routes/admin/jobs/+page.svelte`
|
||
- `src/lib/components/IndexingProgress.svelte`
|
||
- `src/lib/components/admin/JobStatusBadge.svelte`
|
||
- `src/lib/components/admin/JobSkeleton.svelte`
|
||
- `src/lib/components/admin/Toast.svelte`
|
||
- `src/lib/components/admin/WorkerStatusPanel.svelte`
|
||
- `prompts/TRUEREF-0023/iteration_1/plan.md`
|
||
- `prompts/TRUEREF-0023/iteration_1/tasks.yaml`
|
||
- Findings:
|
||
- Iteration 1 already completed the direct-driver reset in the working tree: `package.json` and `package-lock.json` now contain real `better-sqlite3` plus `sqlite-vec`, and the current production/test files read in this pass import `better-sqlite3`, not `libsql`.
|
||
- The remaining failing intermediate state is exactly the schema/write mismatch called out in the review report: `src/lib/server/db/schema.ts` and `src/lib/server/db/migrations/0006_yielding_centennial.sql` still declare `vec_embedding`, `src/lib/server/db/index.ts` still executes `vectors.sql`, and `src/lib/server/embeddings/embedding.service.ts` still inserts `(embedding, vec_embedding)` into `snippet_embeddings`.
|
||
- `src/lib/server/db/vectors.sql` is still invalid startup SQL. It contains a dangling `USING libsql_vector_idx(...)` clause with no enclosing `CREATE INDEX`, so leaving it in the initialization path keeps the rejected libSQL-native design alive.
|
||
- The first iteration-1 task boundary was therefore wrong for the current baseline: the package/import reset is already present, but it only becomes a valid foundation once the relational `vec_embedding` artifacts and `EmbeddingService` insert path are cleaned up in the same validation unit.
|
||
- The current search path is still the pre-sqlite-vec implementation. `src/lib/server/search/vector.search.ts` reads every candidate embedding blob and scores in JavaScript; no `vec0`, `sqliteVec.load(db)`, or sqlite-vec KNN query exists anywhere under `src/` yet.
|
||
- The write worker and worker-status backend are still missing in the live tree even though they are already referenced elsewhere: `scripts/build-workers.mjs` includes `src/lib/server/pipeline/write-worker-entry.ts`, `src/lib/components/admin/WorkerStatusPanel.svelte` fetches `/api/v1/workers`, and `src/routes/api/v1/jobs/stream/+server.ts` currently has no worker-status event source.
|
||
- The admin jobs page remains incomplete but salvageable: `src/routes/admin/jobs/+page.svelte` still uses `confirm(...)` and `alert(...)`, while `JobSkeleton.svelte`, `Toast.svelte`, `WorkerStatusPanel.svelte`, `JobStatusBadge.svelte`, and `IndexingProgress.svelte` already provide the intended UI foundation.
|
||
- `src/lib/server/pipeline/job-queue.ts` still only supports exact `repository_id = ?` and single `status = ?` filtering, so API-side filter work remains a separate backend task and does not need to block the vector-storage implementation.
|
||
- Risks / follow-ups:
|
||
- Iteration 2 task decomposition must treat the current dirty code files from iterations 0 and 1 as the validation baseline, otherwise the executor will keep rediscovering pre-existing worktree drift instead of new task deltas.
|
||
- The sqlite-vec bootstrap helper and the relational cleanup should be planned as one acceptance unit before any downstream vec0, worker-status, or admin-page tasks, because that is the smallest unit that removes the known broken intermediate state.
|