mozempk/trueref
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9525c58e9af280e452b560c9719e82fee4d8a923
trueref/docs/FINDINGS.md
Giancarmine Salucci 9525c58e9a feat(TRUEREF-0023): add sqlite-vec search pipeline
2026-04-01 14:09:19 +02:00

30 KiB
Raw Blame History

Findings

Last Updated: 2026-04-01T12:05:23.000Z

Initializer Summary

  • JIRA: TRUEREF-0023
  • Refresh mode: REFRESH_IF_REQUIRED
  • Result: Refreshed ARCHITECTURE.md and FINDINGS.md. CODE_STYLE.md remained trusted — sqlite-vec, worker-status, and write-worker additions follow the established conventions already documented.

Research Performed

  • Counted 149 TypeScript/JavaScript source files in the repository-wide scan and verified the live, non-generated source mix as 147 .ts files and 2 .js files.
  • Read package.json, .prettierrc, and eslint.config.js to verify dependencies, formatting rules, and linting conventions.
  • Read sqlite-vec.ts, sqlite-vec.store.ts, vector.search.ts, hybrid.search.service.ts, schema.ts, client.ts, and startup wiring to verify the accepted sqlite-vec implementation and current retrieval architecture.
  • Read worker-pool.ts, worker-types.ts, write-worker-entry.ts, and /api/v1/workers/+server.ts to verify the current worker topology and status surface.
  • Compared docs/docs_cache_state.yaml against the live docs and codebase to identify stale cache evidence and architecture drift.
  • Confirmed CODE_STYLE.md still matches the codebase: tabs, single quotes, trailingComma: none, ESM imports with node: built-ins, flat ESLint config, and descriptive PascalCase/camelCase naming remain consistent.

Open Questions For Planner

  • Verify whether the write-worker protocol should become part of the active indexing flow or remain documented as optional infrastructure only.
  • Verify whether worker-status and SSE event payloads should be documented in a dedicated API reference for external consumers.
  • Verify whether sqlite-vec operational details such as per-profile vec-table lifecycle and backfill behavior should move into a separate persistence document if the subsystem grows further.
  • Assess whether the WorkerPool fallback mode (main-thread execution when worker scripts are missing) still belongs in the runtime contract or should be removed in favour of a hard build requirement.

Planner Notes Template

Add subsequent research below this section.

Entry Template

  • Date:
  • Task:
  • Files inspected:
  • Findings:
  • Risks / follow-ups:

2026-04-01 — TRUEREF-0023 initializer refresh audit

  • Task: Refresh only stale or invalid documentation after the accepted sqlite-vec implementation.
  • Files inspected:
    • docs/docs_cache_state.yaml
    • docs/ARCHITECTURE.md
    • docs/CODE_STYLE.md
    • docs/FINDINGS.md
    • package.json
    • .prettierrc
    • eslint.config.js
    • src/hooks.server.ts
    • src/lib/server/db/client.ts
    • src/lib/server/db/schema.ts
    • src/lib/server/db/sqlite-vec.ts
    • src/lib/server/search/sqlite-vec.store.ts
    • src/lib/server/search/vector.search.ts
    • src/lib/server/search/hybrid.search.service.ts
    • src/lib/server/pipeline/startup.ts
    • src/lib/server/pipeline/worker-pool.ts
    • src/lib/server/pipeline/worker-types.ts
    • src/lib/server/pipeline/write-worker-entry.ts
    • src/routes/api/v1/workers/+server.ts
    • scripts/build-workers.mjs
  • Findings:
    • The trusted cache metadata was no longer reliable as evidence for planning: docs/docs_cache_state.yaml still referenced 2026-03-27 hashes while ARCHITECTURE.md and FINDINGS.md had been edited later.
    • ARCHITECTURE.md was stale. It still described only parse and embed worker concurrency, omitted the sqlite-vec production dependency, and did not document the current per-profile vec-table storage layer, worker-status endpoint, or write-worker infrastructure.
    • The current retrieval stack uses sqlite-vec concretely: loadSqliteVec() bootstraps connections, SqliteVecStore manages vec0 tables plus rowid mapping tables, and VectorSearch delegates nearest-neighbor lookup to that store instead of brute-force scoring.
    • The worker architecture now includes parse, embed, and write worker protocols in worker-types.ts, build-time bundling for all three entries, and a /api/v1/workers route that returns WorkerPool status snapshots.
    • CODE_STYLE.md remained valid and did not require refresh. The observed source and config files still use tabs, single quotes, trailingComma: none, flat ESLint config, ESM imports, PascalCase class names, and camelCase helpers exactly as already documented.
    • FINDINGS.md itself was stale because the initializer summary still referred to TRUEREF-0022 instead of the requested TRUEREF-0023 refresh.
  • Risks / follow-ups:
    • The write-worker protocol exists and is bundled, but the active indexing path is still centered on parse plus optional embed flow. Future documentation should keep distinguishing implemented infrastructure from the currently exercised path.
    • Cache validity should continue to be driven by deterministic hash evidence rather than document timestamps or trust text alone.

2026-03-27 — FEEDBACK-0001 initializer refresh audit

  • Task: Refresh only stale documentation after changes to retrieval, formatters, token budgeting, and parser behavior.
  • Files inspected:
    • docs/docs_cache_state.yaml
    • docs/ARCHITECTURE.md
    • docs/CODE_STYLE.md
    • docs/FINDINGS.md
    • package.json
    • src/routes/api/v1/context/+server.ts
    • src/lib/server/api/formatters.ts
    • src/lib/server/api/token-budget.ts
    • src/lib/server/search/query-preprocessor.ts
    • src/lib/server/search/search.service.ts
    • src/lib/server/search/hybrid.search.service.ts
    • src/lib/server/mappers/context-response.mapper.ts
    • src/lib/server/models/context-response.ts
    • src/lib/server/models/search-result.ts
    • src/lib/server/parser/index.ts
    • src/lib/server/parser/code.parser.ts
    • src/lib/server/parser/markdown.parser.ts
  • Findings:
    • The documentation cache was trusted, but the architecture summary no longer captured current retrieval behavior: query preprocessing now sanitizes punctuation-heavy input for FTS5, semantic mode can bypass FTS entirely, and auto or hybrid retrieval can fall back to vector search when keyword search returns no candidates.
    • Plain-text and JSON context formatting now carry repository and version metadata, and the text formatter emits an explicit no-results section instead of an empty body.
    • Token budgeting now skips individual over-budget snippets and continues evaluating lower-ranked candidates, which changes the response-selection behavior described at the architecture level.
    • Parser coverage now explicitly includes Markdown, code, config, HTML-like, and plain-text inputs, so the architecture summary needed to reflect that broader file-type handling.
    • The conventions documented in CODE_STYLE.md still match the current repository: strict TypeScript, tab indentation, ESM imports, Prettier and ESLint flat config, and pragmatic service-oriented server modules.
  • Risks / follow-ups:
    • Future cache invalidation should continue to distinguish between behavioral changes that affect architecture docs and localized implementation changes that do not affect the style guide.
    • If the public API contract becomes externally versioned, the new context metadata fields likely deserve a dedicated API document instead of only architecture-level coverage.

2026-03-27 — FEEDBACK-0001 planning research

  • Task: Plan the retrieval-fix iteration covering FTS query safety, hybrid fallback, empty-result behavior, result metadata, token budgeting, and parser chunking.
  • Files inspected:
    • package.json
    • src/routes/api/v1/context/+server.ts
    • src/lib/server/search/query-preprocessor.ts
    • src/lib/server/search/search.service.ts
    • src/lib/server/search/hybrid.search.service.ts
    • src/lib/server/search/vector.search.ts
    • src/lib/server/api/token-budget.ts
    • src/lib/server/api/formatters.ts
    • src/lib/server/mappers/context-response.mapper.ts
    • src/lib/server/models/context-response.ts
    • src/lib/server/models/search-result.ts
    • src/lib/server/parser/code.parser.ts
    • src/lib/server/search/search.service.test.ts
    • src/lib/server/search/hybrid.search.service.test.ts
    • src/lib/server/api/formatters.test.ts
    • src/lib/server/parser/code.parser.test.ts
    • src/routes/api/v1/api-contract.integration.test.ts
    • src/mcp/tools/query-docs.ts
    • src/mcp/client.ts
  • Findings:
    • better-sqlite3 ^12.6.2 backs the affected search path; the code already uses bound parameters for MATCH, so the practical fix belongs in query normalization and fallback handling rather than SQL string construction.
    • query-preprocessor.ts only strips parentheses and appends a trailing wildcard. Other code-like punctuation currently reaches the FTS execution path unsanitized.
    • search.service.ts sends the preprocessed text directly to snippets_fts MATCH ? and already returns [] for blank processed queries.
    • hybrid.search.service.ts always executes keyword search before semantic branching. In the current flow, an FTS parse failure can abort auto, hybrid, and semantic requests before vector retrieval runs.
    • vector.search.ts already preserves repositoryId, versionId, and profileId filtering and does not need architectural changes for this iteration.
    • token-budget.ts stops at the first over-budget snippet instead of skipping that item and continuing through later ranked results.
    • formatContextTxt([], []) returns an empty string, so /api/v1/context?type=txt can emit an empty 200 OK body today.
    • context-response.mapper.ts and context-response.ts expose snippet content and breadcrumb/page title but do not identify local TrueRef origin, repository source metadata, or normalized snippet origin labels.
    • code.parser.ts splits primarily at top-level declarations; class/object member functions remain in coarse chunks, which limits method-level recall for camelCase API queries.
    • Existing relevant automated coverage is concentrated in the search, formatter, and parser unit tests; /api/v1/context contract coverage currently omits the context endpoint entirely.
  • Risks / follow-ups:
    • Response-shape changes must be additive because src/mcp/client.ts, src/mcp/tools/query-docs.ts, and UI consumers expect the current top-level keys to remain present.
    • Parser improvements should stay inside parseCodeFile() and existing chunking helpers to avoid turning this fix iteration into a schema or pipeline redesign.

2026-03-27 — FEEDBACK-0001 SQLite FTS5 syntax research

  • Task: Verify the FTS5 query-grammar constraints that affect punctuation-heavy local search queries.
  • Files inspected:
    • package.json
    • src/lib/server/search/query-preprocessor.ts
    • src/lib/server/search/search.service.ts
    • src/lib/server/search/hybrid.search.service.ts
  • Findings:
    • better-sqlite3 is pinned at ^12.6.2 in package.json, and the application binds the MATCH string as a parameter instead of interpolating SQL directly.
    • The canonical SQLite FTS5 docs state that barewords may contain letters, digits, underscore, non-ASCII characters, and the substitute character; strings containing other punctuation must be quoted or they become syntax errors in MATCH expressions.
    • The same docs state that prefix search is expressed by placing * after the token or phrase, not inside quotes, which matches the current trailing-wildcard strategy in query-preprocessor.ts.
    • SQLite documents that FTS5 is stricter than FTS3/4 about unrecognized punctuation in query strings, which confirms that code-like user input should be normalized before it reaches snippets_fts MATCH ?.
    • Based on the current code path, the practical fix remains application-side sanitization and fallback behavior in query-preprocessor.ts and hybrid.search.service.ts, not SQL construction changes.
  • Risks / follow-ups:
    • Over-sanitizing punctuation-heavy inputs could erase useful identifiers, so the implementation should preserve searchable alphanumeric and underscore tokens while discarding grammar-breaking punctuation.
    • Prefix expansion should remain on the final searchable token only so the fix preserves current query-cost expectations and test semantics.

2026-03-27 — LINT-0001 planning research

  • Task: Plan the lint-fix iteration covering the reported ESLint and eslint-plugin-svelte violations across Svelte UI, SvelteKit routes, server modules, and Vitest suites.
  • Files inspected:
    • package.json
    • eslint.config.js
    • docs/FINDINGS.md
    • prompts/LINT-0001/prompt.yaml
    • prompts/LINT-0001/progress.yaml
    • src/lib/components/FolderPicker.svelte
    • src/lib/components/RepositoryCard.svelte
    • src/lib/components/search/SnippetCard.svelte
    • src/lib/server/crawler/local.crawler.test.ts
    • src/lib/server/embeddings/embedding.service.test.ts
    • src/lib/server/embeddings/local.provider.ts
    • src/lib/server/embeddings/provider.ts
    • src/lib/server/embeddings/registry.ts
    • src/lib/server/models/context-response.ts
    • src/lib/server/parser/code.parser.ts
    • src/lib/server/pipeline/indexing.pipeline.ts
    • src/lib/server/search/hybrid.search.service.test.ts
    • src/lib/server/search/query-preprocessor.ts
    • src/lib/server/services/repository.service.test.ts
    • src/lib/server/services/version.service.test.ts
    • src/lib/server/services/version.service.ts
    • src/routes/+layout.svelte
    • src/routes/+page.svelte
    • src/routes/api/v1/libs/search/+server.ts
    • src/routes/api/v1/settings/embedding/+server.ts
    • src/routes/repos/[id]/+page.svelte
    • src/routes/search/+page.svelte
    • src/routes/settings/+page.svelte
  • Findings:
    • The project lint stack is ESLint ^9.39.2 with typescript-eslint recommended rules and eslint-plugin-svelte recommended plus SvelteKit-aware rules, running over Svelte ^5.51.0 and SvelteKit ^2.50.2.
    • Context7 documentation for eslint-plugin-svelte confirms svelte/no-navigation-without-base flags root-relative <a href="/..."> links and goto('/...') calls in SvelteKit projects; compliant fixes must use $app/paths base-aware links or base-prefixed goto calls.
    • Context7 documentation for Svelte 5 confirms event handlers are regular element properties such as onclick, while side effects belong in $effect; repo memory also records that client-only fetch bootstrap should not be moved indiscriminately into $effect when onMount or load is the correct lifecycle boundary.
    • Concrete navigation violations already exist in src/routes/+layout.svelte, src/routes/repos/[id]/+page.svelte, src/routes/search/+page.svelte, and src/lib/components/RepositoryCard.svelte, each using hard-coded root-relative internal navigation.
    • Static diagnostics currently expose at least one direct TypeScript lint error in src/lib/server/embeddings/registry.ts, where _config is defined but never used.
    • src/routes/api/v1/libs/search/+server.ts imports json from @sveltejs/kit without using it, making that endpoint a concrete unused-import cleanup target.
    • src/lib/server/services/version.service.ts still uses CommonJS require(...) to reach git utilities from TypeScript, which is inconsistent with the repository's ESM style and is a likely lint target under the current ESLint stack.
    • The affected Svelte pages and settings UI already use Svelte 5 event-property syntax, so the lint work should preserve that syntax and focus on base-aware navigation, lifecycle correctness, and unused-symbol cleanup rather than regressing to legacy on: directives.
    • Existing automated coverage for the lint-touching backend areas already lives in src/lib/server/crawler/local.crawler.test.ts, src/lib/server/embeddings/embedding.service.test.ts, src/lib/server/search/hybrid.search.service.test.ts, src/lib/server/services/repository.service.test.ts, and src/lib/server/services/version.service.test.ts; route and component changes rely on build and lint validation rather than dedicated browser tests in this iteration.
  • Risks / follow-ups:
    • Base-aware navigation fixes must preserve internal app routing semantics and should not replace intentional external navigation, because SvelteKit goto(...) no longer accepts external URLs.
    • Settings and search page lifecycle changes must avoid reintroducing SSR-triggered fetches or self-triggered URL loops; client-only bootstrap logic should remain mounted once and URL-sync effects must stay idempotent.

2026-03-27 — ROUTING-0001 planning research

  • Task: Plan the repository-detail routing fix for slash-bearing repository IDs causing homepage SSR failures and invalid /repos/[id] navigation.
  • Files inspected:
    • package.json
    • src/lib/components/RepositoryCard.svelte
    • src/routes/+page.svelte
    • src/routes/+page.server.ts
    • src/routes/repos/[id]/+page.server.ts
    • src/routes/repos/[id]/+page.svelte
    • src/routes/api/v1/api-contract.integration.test.ts
    • src/lib/types.ts
  • Findings:
    • The app is on SvelteKit ^2.50.2 and uses $app/paths.resolve(...) for internal navigation, including resolveRoute('/repos/[id]', { id: repo.id }) in RepositoryCard.svelte.
    • SvelteKit’s [id] route is a single-segment dynamic parameter. Context7 routing docs show slash-containing values belong to rest parameters like [...param], so raw repository IDs containing / are invalid inputs for resolveRoute('/repos/[id]', ...).
    • The repository model intentionally stores slash-bearing IDs such as /facebook/react, and the existing API surface consistently treats those IDs as percent-encoded path segments. The integration contract already passes params.id = encodeURIComponent('/facebook/react') for /api/v1/libs/[id] handlers, which then call decodeURIComponent(params.id).
    • The homepage SSR failure is therefore rooted in UI link generation, not repository listing fetches: rendering RepositoryCard.svelte with a raw slash-bearing repo.id can throw before page load completes, which explains repeated 500 responses on /.
    • The repo detail page currently forwards params.id directly into encodeURIComponent(...) for downstream API requests. Once detail links are generated as encoded single segments, the page loader and client-side refresh/delete/reindex flows need one normalization step so API calls continue targeting the stored repository ID instead of a doubly encoded value.
    • No existing browser-facing test covers homepage card navigation or /repos/[id] loader behavior; the closest current evidence is the API contract test file, which already exercises encoded repository IDs on HTTP endpoints and provides reusable fixtures for slash-bearing IDs.
  • Risks / follow-ups:
    • The fix should preserve the existing /repos/[id] route shape instead of redesigning it to a rest route unless a broader navigation contract change is explicitly requested.
    • Any normalization helper introduced for the repo detail page should be reused consistently across server load and client event handlers to avoid mixed encoded and decoded repository IDs during navigation and fetches.

2026-04-01 — TRUEREF-0023 sqlite-vec replanning research

  • Task: Replan the rejected libSQL-native vector iteration around sqlite-vec using the current worktree and verified runtime constraints.
  • Files inspected:
    • package.json
    • docs/docs_cache_state.yaml
    • prompts/TRUEREF-0023/prompt.yaml
    • prompts/TRUEREF-0023/progress.yaml
    • prompts/TRUEREF-0023/iteration_0/review_report.yaml
    • prompts/TRUEREF-0023/iteration_0/plan.md
    • prompts/TRUEREF-0023/iteration_0/tasks.yaml
    • src/lib/server/db/client.ts
    • src/lib/server/db/index.ts
    • src/lib/server/db/schema.ts
    • src/lib/server/db/fts.sql
    • src/lib/server/db/vectors.sql
    • src/lib/server/db/schema.test.ts
    • src/lib/server/search/vector.search.ts
    • src/lib/server/search/hybrid.search.service.test.ts
    • src/lib/server/embeddings/embedding.service.ts
    • src/lib/server/embeddings/embedding.service.test.ts
    • src/lib/server/pipeline/job-queue.ts
    • src/lib/server/pipeline/progress-broadcaster.ts
    • src/lib/server/pipeline/progress-broadcaster.test.ts
    • src/lib/server/pipeline/worker-pool.ts
    • src/lib/server/pipeline/worker-entry.ts
    • src/lib/server/pipeline/embed-worker-entry.ts
    • src/lib/server/pipeline/worker-types.ts
    • src/lib/server/pipeline/startup.ts
    • src/lib/server/pipeline/indexing.pipeline.ts
    • src/lib/server/pipeline/indexing.pipeline.test.ts
    • src/routes/api/v1/jobs/+server.ts
    • src/routes/api/v1/jobs/stream/+server.ts
    • src/routes/api/v1/jobs/[id]/stream/+server.ts
    • src/routes/api/v1/sse-and-settings.integration.test.ts
    • src/routes/admin/jobs/+page.svelte
    • src/lib/components/IndexingProgress.svelte
    • src/lib/components/admin/JobStatusBadge.svelte
    • src/lib/components/admin/JobSkeleton.svelte
    • src/lib/components/admin/Toast.svelte
    • src/lib/components/admin/WorkerStatusPanel.svelte
    • scripts/build-workers.mjs
    • node_modules/libsql/types/index.d.ts
    • node_modules/libsql/index.js
  • Findings:
    • Iteration 0 already changed the workspace materially: direct DB imports were switched from better-sqlite3 to libsql, the extra WAL-related pragmas were added in the main DB clients and embed worker, composite indexes plus vec_embedding were added to the Drizzle schema and migration metadata, IndexingProgress.svelte now uses SSE, the admin jobs page was overhauled, and WorkerPool now serializes on (repositoryId, versionId) instead of repository only.
    • The rejected vector implementation is still invalid in the current tree. src/lib/server/db/vectors.sql contains the rejected libSQL-native assumptions, including a dangling USING libsql_vector_idx(...) clause with no valid CREATE INDEX statement, and src/lib/server/search/vector.search.ts still performs full-table JS cosine scoring over snippet_embeddings instead of true in-database KNN.
    • sqlite-vec is not currently present in package.json or the lockfile, and there is no existing sqliteVec.load(...), db.loadExtension(...), vec0, or extension bootstrap code anywhere under src/.
    • Context7 sqlite-vec docs confirm the supported Node integration path is import * as sqliteVec from 'sqlite-vec'; sqliteVec.load(db);, storing vectors in a vec0 virtual table and querying with WHERE embedding MATCH ? ORDER BY distance LIMIT ?. The docs also show vec0 metadata columns can be filtered directly, which fits the repositoryId, versionId, and profileId requirements.
    • Context7 better-sqlite3 v12.6.2 docs confirm db.loadExtension(path) exists. The installed libsql package in this workspace also exposes loadExtension(path): this in node_modules/libsql/types/index.d.ts and loadExtension(...args) in node_modules/libsql/index.js, so extension loading is not obviously blocked by the driver API surface alone.
    • The review report remains the only verified runtime evidence for the current libsql path: vector_from_float32(...) is unavailable and libsql_vector_idx DDL is rejected in this environment. That invalidates the original native-vector approach but does not by itself prove sqlite-vec extension loading succeeds through the current libsql package alias, so the replan must include explicit connection-bootstrap and test coverage for real extension loading on the main DB client and worker-owned connections.
    • Two iteration-0 deliverables referenced in the rejected plan do not exist in the current worktree: src/lib/server/pipeline/write-worker-entry.ts and src/routes/api/v1/workers/+server.ts. scripts/build-workers.mjs and the admin WorkerStatusPanel.svelte already reference those missing paths, so iteration 1 must either create them or revert those dangling references as part of a consistent plan.
    • The existing admin/SSE work is largely salvageable. src/routes/api/v1/jobs/stream/+server.ts, src/routes/api/v1/jobs/[id]/stream/+server.ts, src/lib/server/pipeline/progress-broadcaster.ts, src/lib/components/IndexingProgress.svelte, src/lib/components/admin/JobSkeleton.svelte, src/lib/components/admin/Toast.svelte, and src/lib/components/admin/WorkerStatusPanel.svelte provide a usable foundation, but src/routes/admin/jobs/+page.svelte still contains confirm(...) and the queue API still only supports exact repository_id = ? and single-status filtering.
    • The existing tests still encode the rejected pre-sqlite-vec model: embedding.service.test.ts, schema.test.ts, hybrid.search.service.test.ts, and indexing.pipeline.test.ts seed and assert against snippet_embeddings.embedding blobs only. The sqlite-vec replan therefore needs new DB bootstrap helpers, vec-table lifecycle assertions, and vector-search tests that validate actual vec0 writes and filtered KNN queries.
  • Risks / follow-ups:
    • The current worktree is dirty with iteration-0 partial changes and generated migration metadata, so iteration-1 tasks must explicitly distinguish keep/revise/revert work to avoid sibling tasks fighting over the same files.
    • Because the current libsql package appears to expose loadExtension, the replan should avoid assuming an immediate full revert to upstream better-sqlite3; instead it should sequence a driver/bootstrap compatibility decision around actual sqlite-vec extension loading behavior with testable acceptance criteria.

2026-04-01 — TRUEREF-0023 iteration-2 current-worktree verification

  • Task: Replan iteration 2 against the post-iteration-1 workspace state so the first validation unit no longer leaves a known vec_embedding mismatch behind.
  • Files inspected:
    • package.json
    • package-lock.json
    • scripts/build-workers.mjs
    • src/lib/server/db/client.ts
    • src/lib/server/db/index.ts
    • src/lib/server/db/schema.ts
    • src/lib/server/db/vectors.sql
    • src/lib/server/db/migrations/0006_yielding_centennial.sql
    • src/lib/server/db/schema.test.ts
    • src/lib/server/embeddings/embedding.service.ts
    • src/lib/server/embeddings/embedding.service.test.ts
    • src/lib/server/search/vector.search.ts
    • src/lib/server/search/hybrid.search.service.ts
    • src/lib/server/search/hybrid.search.service.test.ts
    • src/lib/server/pipeline/job-queue.ts
    • src/lib/server/pipeline/worker-pool.ts
    • src/lib/server/pipeline/worker-entry.ts
    • src/lib/server/pipeline/embed-worker-entry.ts
    • src/lib/server/pipeline/worker-types.ts
    • src/lib/server/pipeline/indexing.pipeline.ts
    • src/lib/server/pipeline/indexing.pipeline.test.ts
    • src/lib/server/pipeline/startup.ts
    • src/lib/server/pipeline/progress-broadcaster.ts
    • src/routes/api/v1/jobs/+server.ts
    • src/routes/api/v1/jobs/stream/+server.ts
    • src/routes/api/v1/sse-and-settings.integration.test.ts
    • src/routes/admin/jobs/+page.svelte
    • src/lib/components/IndexingProgress.svelte
    • src/lib/components/admin/JobStatusBadge.svelte
    • src/lib/components/admin/JobSkeleton.svelte
    • src/lib/components/admin/Toast.svelte
    • src/lib/components/admin/WorkerStatusPanel.svelte
    • prompts/TRUEREF-0023/iteration_1/plan.md
    • prompts/TRUEREF-0023/iteration_1/tasks.yaml
  • Findings:
    • Iteration 1 already completed the direct-driver reset in the working tree: package.json and package-lock.json now contain real better-sqlite3 plus sqlite-vec, and the current production/test files read in this pass import better-sqlite3, not libsql.
    • The remaining failing intermediate state is exactly the schema/write mismatch called out in the review report: src/lib/server/db/schema.ts and src/lib/server/db/migrations/0006_yielding_centennial.sql still declare vec_embedding, src/lib/server/db/index.ts still executes vectors.sql, and src/lib/server/embeddings/embedding.service.ts still inserts (embedding, vec_embedding) into snippet_embeddings.
    • src/lib/server/db/vectors.sql is still invalid startup SQL. It contains a dangling USING libsql_vector_idx(...) clause with no enclosing CREATE INDEX, so leaving it in the initialization path keeps the rejected libSQL-native design alive.
    • The first iteration-1 task boundary was therefore wrong for the current baseline: the package/import reset is already present, but it only becomes a valid foundation once the relational vec_embedding artifacts and EmbeddingService insert path are cleaned up in the same validation unit.
    • The current search path is still the pre-sqlite-vec implementation. src/lib/server/search/vector.search.ts reads every candidate embedding blob and scores in JavaScript; no vec0, sqliteVec.load(db), or sqlite-vec KNN query exists anywhere under src/ yet.
    • The write worker and worker-status backend are still missing in the live tree even though they are already referenced elsewhere: scripts/build-workers.mjs includes src/lib/server/pipeline/write-worker-entry.ts, src/lib/components/admin/WorkerStatusPanel.svelte fetches /api/v1/workers, and src/routes/api/v1/jobs/stream/+server.ts currently has no worker-status event source.
    • The admin jobs page remains incomplete but salvageable: src/routes/admin/jobs/+page.svelte still uses confirm(...) and alert(...), while JobSkeleton.svelte, Toast.svelte, WorkerStatusPanel.svelte, JobStatusBadge.svelte, and IndexingProgress.svelte already provide the intended UI foundation.
    • src/lib/server/pipeline/job-queue.ts still only supports exact repository_id = ? and single status = ? filtering, so API-side filter work remains a separate backend task and does not need to block the vector-storage implementation.
  • Risks / follow-ups:
    • Iteration 2 task decomposition must treat the current dirty code files from iterations 0 and 1 as the validation baseline, otherwise the executor will keep rediscovering pre-existing worktree drift instead of new task deltas.
    • The sqlite-vec bootstrap helper and the relational cleanup should be planned as one acceptance unit before any downstream vec0, worker-status, or admin-page tasks, because that is the smallest unit that removes the known broken intermediate state.
Reference in New Issue View Git Blame Copy Permalink