feat(TRUEREF-0023): add sqlite-vec search pipeline
This commit is contained in:
Giancarmine Salucci
@@ -1,171 +1,168 @@
|
||||
# Architecture
|
||||
|
||||
Last Updated: 2026-03-30T00:00:00.000Z
|
||||
Last Updated: 2026-04-01T12:05:23.000Z
|
||||
|
||||
## Overview
|
||||
|
||||
TrueRef is a TypeScript-first, self-hosted documentation retrieval platform built on SvelteKit. The repository contains a Node-targeted web application, a REST API, a Model Context Protocol server, and a multi-threaded server-side indexing pipeline backed by SQLite via better-sqlite3 and Drizzle ORM.
|
||||
TrueRef is a TypeScript-first, self-hosted documentation retrieval platform built on SvelteKit. The repository contains a Node-targeted web application, a REST API, a Model Context Protocol server, and a worker-threaded indexing pipeline backed by SQLite via better-sqlite3, Drizzle ORM, FTS5, and sqlite-vec.
|
||||
|
||||
- Primary language: TypeScript (141 files) with a small amount of JavaScript configuration (2 files)
|
||||
- Application type: Full-stack SvelteKit application with worker-threaded indexing and retrieval services
|
||||
- Primary language: TypeScript (147 `.ts` files) with a small amount of JavaScript configuration and build code (2 `.js` files), excluding generated output and dependencies
|
||||
- Application type: Full-stack SvelteKit application with server-side indexing, retrieval, and MCP integration
|
||||
- Runtime framework: SvelteKit with adapter-node
|
||||
- Storage: SQLite (WAL mode) with Drizzle-managed schema plus hand-written FTS5 setup
|
||||
- Concurrency: Node.js worker_threads for parse and embedding work
|
||||
- Testing: Vitest with separate client and server projects
|
||||
- Storage: SQLite in WAL mode with Drizzle-managed relational schema, FTS5 full-text indexes, and sqlite-vec virtual tables for vector lookup
|
||||
- Concurrency: Node.js `worker_threads` for parse, embed, and auxiliary write-worker infrastructure
|
||||
- Testing: Vitest for unit and integration coverage
|
||||
|
||||
## Project Structure
|
||||
|
||||
- src/routes: SvelteKit pages and HTTP endpoints, including the public UI and /api/v1 surface
|
||||
- src/lib/server: Backend implementation grouped by concern: api, config, crawler, db, embeddings, mappers, models, parser, pipeline, search, services, utils
|
||||
- src/mcp: Standalone MCP server entry point and tool handlers
|
||||
- static: Static assets such as robots.txt
|
||||
- docs/features: Feature-level implementation notes and product documentation
|
||||
- build: Generated SvelteKit output
|
||||
- `src/routes`: SvelteKit pages and HTTP endpoints, including the public UI and `/api/v1` surface
|
||||
- `src/lib/server`: Backend implementation grouped by concern: `api`, `config`, `crawler`, `db`, `embeddings`, `mappers`, `models`, `parser`, `pipeline`, `search`, `services`, `utils`
|
||||
- `src/mcp`: Standalone MCP server entry point, client, tests, and tool handlers
|
||||
- `scripts`: Build helpers, including worker bundling
|
||||
- `static`: Static assets such as `robots.txt`
|
||||
- `docs/features`: Feature-level implementation notes and product documentation
|
||||
- `build`: Generated SvelteKit output and bundled worker entrypoints
|
||||
|
||||
## Key Directories
|
||||
|
||||
### src/routes
|
||||
### `src/routes`
|
||||
|
||||
Contains the UI entry points and API routes. The API tree under src/routes/api/v1 is the public HTTP contract for repository management, indexing jobs, search/context retrieval, settings, filesystem browsing, JSON schema discovery, real-time SSE progress streaming, and job control (pause/resume/cancel).
|
||||
Contains the UI entry points and API routes. The API tree under `src/routes/api/v1` is the public HTTP contract for repository management, version discovery, indexing jobs, search/context retrieval, embedding settings, indexing settings, filesystem browsing, worker-status inspection, and SSE progress streaming.
|
||||
|
||||
### src/lib/server/db
|
||||
### `src/lib/server/db`
|
||||
|
||||
Owns SQLite schema definitions, migration bootstrapping, and FTS initialization. Database startup runs through initializeDatabase(), which executes Drizzle migrations and then applies FTS5 SQL that cannot be expressed directly in the ORM.
|
||||
Owns SQLite schema definitions, relational migrations, connection bootstrapping, and sqlite-vec loading. Database startup goes through `initializeDatabase()` and `getClient()`, both of which configure WAL-mode pragmas and ensure sqlite-vec is loaded on each connection before vector-backed queries run.
|
||||
|
||||
### src/lib/server/pipeline
|
||||
### `src/lib/server/search`
|
||||
|
||||
Coordinates crawl, parse, chunk, store, and optional embedding generation work using a worker thread pool. The pipeline module consists of:
|
||||
Implements keyword, vector, and hybrid retrieval. Keyword search uses SQLite FTS5 and BM25-style ranking. Vector search uses `SqliteVecStore` to maintain per-profile sqlite-vec `vec0` tables plus rowid mapping tables, and hybrid search blends FTS and vector candidates through reciprocal rank fusion.
|
||||
|
||||
- **WorkerPool** (`worker-pool.ts`): Manages a configurable number of Node.js `worker_threads` for parse jobs and an optional dedicated embed worker. Dispatches jobs round-robin to idle workers, enforces per-repository serialisation (one active job per repo), auto-respawns crashed workers, and supports runtime concurrency adjustment via `setMaxConcurrency()`. Falls back to main-thread execution when worker scripts are not found.
|
||||
- **Parse worker** (`worker-entry.ts`): Runs in a worker thread. Opens its own `better-sqlite3` connection (WAL mode, `busy_timeout = 5000`), constructs a local `IndexingPipeline` instance, and processes jobs by posting `progress`, `done`, or `failed` messages back to the parent.
|
||||
- **Embed worker** (`embed-worker-entry.ts`): Dedicated worker for embedding generation. Loads the embedding profile from the database, creates an `EmbeddingService`, and processes embed requests after the parse worker finishes a job.
|
||||
- **ProgressBroadcaster** (`progress-broadcaster.ts`): Server-side pub/sub for real-time SSE streaming. Supports per-job, per-repository, and global subscriptions. Caches the last event per job for reconnect support.
|
||||
- **Worker types** (`worker-types.ts`): Shared TypeScript discriminated union types for `ParseWorkerRequest`/`ParseWorkerResponse` and `EmbedWorkerRequest`/`EmbedWorkerResponse` message protocols.
|
||||
- **Startup** (`startup.ts`): Recovers stale jobs, constructs singleton `JobQueue`, `IndexingPipeline`, `WorkerPool`, and `ProgressBroadcaster` instances, reads concurrency settings from the database, and drains queued work after restart.
|
||||
- **JobQueue** (`job-queue.ts`): SQLite-backed queue that delegates to the `WorkerPool` when available, with pause/resume/cancel support.
|
||||
### `src/lib/server/pipeline`
|
||||
|
||||
### src/lib/server/search
|
||||
Coordinates crawl, diff, parse, store, embed, and job-state broadcasting. The pipeline module consists of:
|
||||
|
||||
Implements keyword, vector, and hybrid retrieval. The keyword path uses SQLite FTS5 and BM25; the hybrid path blends FTS and vector search with reciprocal rank fusion.
|
||||
- `IndexingPipeline`: orchestrates crawl, diff, parse, transactional replacement, optional embedding generation, and repository statistics updates
|
||||
- `WorkerPool`: manages parse workers, an optional embed worker, an optional write worker, per-repository-and-version serialization, worker respawn, and runtime concurrency changes
|
||||
- `worker-entry.ts`: parse worker that opens its own `better-sqlite3` connection, runs the indexing pipeline, and reports progress back to the parent
|
||||
- `embed-worker-entry.ts`: embedding worker that loads the active profile, creates an `EmbeddingService`, and generates vectors after parse completion
|
||||
- `write-worker-entry.ts`: batch-write worker with a `write`/`write_ack`/`write_error` message protocol for document and snippet persistence
|
||||
- `progress-broadcaster.ts`: server-side pub/sub for per-job, per-repository, global, and worker-status SSE streams
|
||||
- `startup.ts`: recovers stale jobs, constructs singleton queue/pipeline/pool/broadcaster instances, loads concurrency settings, and drains queued work after restart
|
||||
- `worker-types.ts`: shared TypeScript discriminated unions for parse, embed, and write worker protocols
|
||||
|
||||
### src/lib/server/crawler and src/lib/server/parser
|
||||
### `src/lib/server/crawler` and `src/lib/server/parser`
|
||||
|
||||
Convert GitHub repositories and local folders into normalized snippet records. Crawlers fetch repository contents, parsers split Markdown, code, config, HTML-like, and plain-text files into chunks, and downstream services persist searchable content.
|
||||
Convert GitHub repositories and local folders into normalized snippet records. Crawlers fetch repository contents and configuration, parsers split Markdown, code, config, HTML-like, and plain-text files into searchable snippet records, and downstream services persist searchable content and embeddings.
|
||||
|
||||
### src/mcp
|
||||
### `src/mcp`
|
||||
|
||||
Provides a thin compatibility layer over the HTTP API. The MCP server exposes resolve-library-id and query-docs over stdio or HTTP and forwards work to local tool handlers.
|
||||
Provides a thin compatibility layer over the HTTP API. The MCP server exposes `resolve-library-id` and `query-docs` over stdio or HTTP and forwards work to local handlers that reuse the application retrieval stack.
|
||||
|
||||
## Design Patterns
|
||||
|
||||
- The WorkerPool implements an **observer/callback pattern**: the pool owner provides `onProgress`, `onJobDone`, `onJobFailed`, `onEmbedDone`, and `onEmbedFailed` callbacks at construction time, and the pool invokes them when workers post messages.
|
||||
- ProgressBroadcaster implements a **pub/sub pattern** with three subscription tiers (per-job, per-repository, global) and last-event caching for SSE reconnect.
|
||||
- The implementation consistently uses **service classes** such as RepositoryService, SearchService, and HybridSearchService for business logic.
|
||||
- Mapping and entity layers separate raw database rows from domain objects through **mapper/entity pairs** such as RepositoryMapper and RepositoryEntity.
|
||||
- Pipeline startup uses **module-level singletons** for JobQueue, IndexingPipeline, WorkerPool, and ProgressBroadcaster lifecycle management, with accessor functions (getQueue, getPool, getBroadcaster) for route handlers.
|
||||
- Worker message protocols use **TypeScript discriminated unions** (`type` field) for type-safe worker ↔ parent communication.
|
||||
- **Service layer**: business logic lives in classes such as `RepositoryService`, `VersionService`, `SearchService`, `HybridSearchService`, and `EmbeddingService`
|
||||
- **Factory pattern**: embedding providers are created from persisted profile records through registry/factory helpers
|
||||
- **Mapper/entity separation**: mappers translate between raw database rows and domain entities such as `RepositoryEntity`, `RepositoryVersionEntity`, and `EmbeddingProfileEntity`
|
||||
- **Module-level singletons**: pipeline startup owns lifecycle for `JobQueue`, `IndexingPipeline`, `WorkerPool`, and `ProgressBroadcaster`, with accessor functions for route handlers
|
||||
- **Pub/sub**: `ProgressBroadcaster` maintains job, repository, global, and worker-status subscriptions for SSE delivery
|
||||
- **Discriminated unions**: worker message protocols use a `type` field for type-safe parent/worker communication
|
||||
|
||||
## Key Components
|
||||
|
||||
### SvelteKit server bootstrap
|
||||
|
||||
src/hooks.server.ts initializes the database, loads persisted embedding configuration, creates the optional EmbeddingService, reads indexing concurrency settings from the database, starts the indexing pipeline with WorkerPool and ProgressBroadcaster via `initializePipeline(db, embeddingService, { concurrency, dbPath })`, and applies CORS headers to all /api routes.
|
||||
`src/hooks.server.ts` initializes the relational database, opens the shared raw SQLite client, loads the default embedding profile, creates the optional `EmbeddingService`, reads indexing concurrency from the `settings` table, and initializes the queue/pipeline/worker infrastructure.
|
||||
|
||||
### Database layer
|
||||
|
||||
src/lib/server/db/schema.ts defines repositories, repository_versions, documents, snippets, embedding_profiles, snippet_embeddings, indexing_jobs, repository_configs, and settings. This schema models the indexed library catalog, retrieval corpus, embedding state, and job tracking.
|
||||
`src/lib/server/db/schema.ts` defines repositories, repository versions, documents, snippets, embedding profiles, relational embedding metadata, indexing jobs, repository configs, and generic settings. Relational embedding rows keep canonical model metadata and raw float buffers, while sqlite-vec virtual tables are managed separately per profile through `SqliteVecStore`.
|
||||
|
||||
### sqlite-vec integration
|
||||
|
||||
`src/lib/server/db/sqlite-vec.ts` centralizes sqlite-vec loading and deterministic per-profile table naming. `SqliteVecStore` creates `vec0` tables plus rowid mapping tables, backfills missing rows from `snippet_embeddings`, removes stale vector references, and executes nearest-neighbor queries constrained by repository, optional version, and profile.
|
||||
|
||||
### Retrieval API
|
||||
|
||||
src/routes/api/v1/context/+server.ts validates input, resolves repository and optional version IDs, chooses keyword, semantic, or hybrid retrieval, applies token budgeting that skips oversized snippets instead of stopping early, prepends repository rules, and formats JSON or text responses with repository and version metadata.
|
||||
`src/routes/api/v1/context/+server.ts` validates input, resolves repository and optional version scope, chooses keyword, semantic, or hybrid retrieval, applies token budgeting, and formats JSON or text responses. `/api/v1/libs/search` handles repository-level lookup, while MCP tool handlers expose the same retrieval behavior over stdio or HTTP transports.
|
||||
|
||||
### Search engine
|
||||
|
||||
src/lib/server/search/search.service.ts preprocesses raw user input into FTS5-safe MATCH expressions before keyword search and repository lookup. src/lib/server/search/hybrid.search.service.ts supports explicit keyword, semantic, and hybrid modes, falls back to vector retrieval when FTS yields no candidates and an embedding provider is configured, and uses reciprocal rank fusion for blended ranking.
|
||||
`SearchService` preprocesses raw user input into FTS5-safe expressions before keyword search. `HybridSearchService` supports explicit keyword, semantic, and hybrid modes, falls back to vector retrieval when keyword search yields no candidates and an embedding provider is configured, and uses reciprocal rank fusion to merge ranked lists. `VectorSearch` delegates KNN execution to `SqliteVecStore` instead of doing brute-force in-memory cosine scoring.
|
||||
|
||||
### Repository management
|
||||
### Repository and version management
|
||||
|
||||
src/lib/server/services/repository.service.ts provides CRUD and statistics for indexed repositories, including canonical ID generation for GitHub and local sources.
|
||||
`RepositoryService` and `VersionService` provide CRUD, indexing-status, cleanup, and statistics logic for indexed repositories and tagged versions, including sqlite-vec cleanup when repository-scoped or version-scoped content is removed.
|
||||
|
||||
### MCP surface
|
||||
### Worker-threaded indexing
|
||||
|
||||
src/mcp/index.ts creates the MCP server, registers the two supported tools, and exposes them over stdio or streamable HTTP.
|
||||
The active indexing path is parse-worker-first: queued jobs are dispatched to parse workers, progress is written to SQLite and broadcast over SSE, and successful parse completion can enqueue embedding work on the dedicated embed worker. The worker pool also exposes status snapshots through `/api/v1/workers`. Write-worker infrastructure exists in the current architecture and is bundled at build time, but parse/embed flow remains the primary live path described by `IndexingPipeline` and `WorkerPool`.
|
||||
|
||||
### Worker thread pool
|
||||
### SSE streaming and job control
|
||||
|
||||
src/lib/server/pipeline/worker-pool.ts manages a pool of Node.js worker threads. Parse workers run the full crawl → parse → store pipeline inside isolated threads with their own better-sqlite3 connections (WAL mode enables concurrent readers). An optional embed worker handles embedding generation in a separate thread. The pool enforces per-repository serialisation, auto-respawns crashed workers, and supports runtime concurrency changes persisted through the settings table.
|
||||
|
||||
### SSE streaming
|
||||
|
||||
src/lib/server/pipeline/progress-broadcaster.ts provides real-time Server-Sent Event streaming of indexing progress. Route handlers in src/routes/api/v1/jobs/stream and src/routes/api/v1/jobs/[id]/stream expose SSE endpoints. The broadcaster supports per-job, per-repository, and global subscriptions, with last-event caching for reconnect via the `Last-Event-ID` header.
|
||||
|
||||
### Job control
|
||||
|
||||
src/routes/api/v1/jobs/[id]/pause, resume, and cancel endpoints allow runtime control of indexing jobs. The JobQueue supports pause/resume/cancel state transitions persisted to SQLite.
|
||||
`progress-broadcaster.ts` provides real-time Server-Sent Event streaming of indexing progress. Route handlers under `/api/v1/jobs/stream` and `/api/v1/jobs/[id]/stream` expose SSE endpoints, and `/api/v1/workers` exposes worker-pool status. Job control endpoints support pause, resume, and cancel transitions backed by SQLite job state.
|
||||
|
||||
### Indexing settings
|
||||
|
||||
src/routes/api/v1/settings/indexing exposes GET and PUT for indexing concurrency. PUT validates and clamps the value to `max(cpus - 1, 1)`, persists it to the settings table, and live-updates the WorkerPool via `setMaxConcurrency()`.
|
||||
`/api/v1/settings/indexing` exposes GET and PUT for indexing concurrency. The value is persisted in the `settings` table and applied live to the `WorkerPool` through `setMaxConcurrency()`.
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Production
|
||||
|
||||
- @modelcontextprotocol/sdk: MCP server transport and protocol types
|
||||
- @xenova/transformers: local embedding support
|
||||
- better-sqlite3: synchronous SQLite driver
|
||||
- zod: runtime input validation for MCP tools and server helpers
|
||||
- `@modelcontextprotocol/sdk`: MCP server transport and protocol types
|
||||
- `@xenova/transformers`: local embedding support
|
||||
- `better-sqlite3`: synchronous SQLite driver used by the main app and workers
|
||||
- `sqlite-vec`: SQLite vector extension used for `vec0` storage and nearest-neighbor queries
|
||||
- `zod`: runtime validation for MCP tools and server helpers
|
||||
|
||||
### Development
|
||||
|
||||
- @sveltejs/kit and @sveltejs/adapter-node: application framework and Node deployment target
|
||||
- drizzle-kit and drizzle-orm: schema management and typed database access
|
||||
- esbuild: worker thread entry point bundling (build/workers/)
|
||||
- vite and @tailwindcss/vite: bundling and Tailwind integration
|
||||
- vitest and @vitest/browser-playwright: server and browser test execution
|
||||
- eslint, typescript-eslint, eslint-plugin-svelte, prettier, prettier-plugin-svelte, prettier-plugin-tailwindcss: linting and formatting
|
||||
- typescript and @types/node: type-checking and Node typings
|
||||
- `@sveltejs/kit` and `@sveltejs/adapter-node`: application framework and Node deployment target
|
||||
- `drizzle-kit` and `drizzle-orm`: schema management and typed database access
|
||||
- `esbuild`: worker entrypoint bundling into `build/workers`
|
||||
- `vite` and `@tailwindcss/vite`: application bundling and Tailwind integration
|
||||
- `vitest` and `@vitest/browser-playwright`: server and browser test execution
|
||||
- `eslint`, `typescript-eslint`, `eslint-plugin-svelte`, `prettier`, `prettier-plugin-svelte`, `prettier-plugin-tailwindcss`: linting and formatting
|
||||
- `typescript` and `@types/node`: type-checking and Node typings
|
||||
|
||||
## Module Organization
|
||||
|
||||
The backend is organized by responsibility rather than by route. HTTP handlers in src/routes/api/v1 are intentionally thin and delegate to library modules in src/lib/server. Within src/lib/server, concerns are separated into:
|
||||
The backend is organized by responsibility rather than by route. HTTP handlers under `src/routes/api/v1` are intentionally thin and delegate to modules in `src/lib/server`. Within `src/lib/server`, concerns are separated into:
|
||||
|
||||
- models and mappers for entity translation
|
||||
- services for repository/version operations
|
||||
- search for retrieval strategies
|
||||
- crawler and parser for indexing input transformation
|
||||
- pipeline for orchestration and job execution
|
||||
- embeddings for provider abstraction and embedding generation
|
||||
- api and utils for response formatting, validation, and shared helpers
|
||||
- `models` and `mappers` for entity translation
|
||||
- `services` for repository/version operations
|
||||
- `search` for keyword, vector, and hybrid retrieval strategies
|
||||
- `crawler` and `parser` for indexing input transformation
|
||||
- `pipeline` for orchestration, workers, and job execution
|
||||
- `embeddings` for provider abstraction and vector generation
|
||||
- `db`, `api`, `config`, and `utils` for persistence, response formatting, validation, and shared helpers
|
||||
|
||||
The frontend and backend share the same SvelteKit repository, but most non-UI behavior is implemented on the server side.
|
||||
The frontend and backend live in the same SvelteKit repository, but most non-UI behavior is implemented on the server side.
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Indexing flow
|
||||
|
||||
1. Server startup runs initializeDatabase() and initializePipeline() from src/hooks.server.ts, which creates the WorkerPool, ProgressBroadcaster, and JobQueue singletons.
|
||||
2. The pipeline recovers stale jobs (marks running → failed, indexing → error), reads concurrency settings, and resumes queued work.
|
||||
3. When a job is enqueued, the JobQueue delegates to the WorkerPool, which dispatches work to an idle parse worker thread.
|
||||
4. Each parse worker opens its own better-sqlite3 connection (WAL mode) and runs the full crawl → parse → store pipeline, posting progress messages back to the parent thread.
|
||||
5. The parent thread updates job progress in the database and broadcasts SSE events through the ProgressBroadcaster.
|
||||
6. On parse completion, if an embedding provider is configured, the WorkerPool enqueues an embed request to the dedicated embed worker, which generates vectors in its own thread.
|
||||
7. Job control endpoints allow pausing, resuming, or cancelling jobs at runtime.
|
||||
1. Server startup runs database initialization, opens the shared client, loads sqlite-vec, and initializes the pipeline singletons.
|
||||
2. Startup recovery marks interrupted jobs as failed, resets repositories stuck in `indexing`, reads persisted concurrency settings, and drains queued jobs.
|
||||
3. `JobQueue` dispatches eligible work to the `WorkerPool`, which serializes by `(repositoryId, versionId)` and posts jobs to idle parse workers.
|
||||
4. Each parse worker opens its own SQLite connection, crawls the source, computes differential work, parses files into snippets, and persists replacement data through the indexing pipeline.
|
||||
5. The parent thread updates job progress in SQLite and broadcasts SSE progress and worker-status events.
|
||||
6. If an embedding provider is configured, the completed parse job triggers embed work that stores canonical embedding blobs and synchronizes sqlite-vec profile tables for nearest-neighbor lookup.
|
||||
7. Repository/version statistics and job status are finalized in SQLite, and control endpoints can pause, resume, or cancel subsequent queued work.
|
||||
|
||||
### Retrieval flow
|
||||
|
||||
1. Clients call /api/v1/libs/search, /api/v1/context, or the MCP tools.
|
||||
2. Route handlers validate input and load the SQLite client.
|
||||
3. Keyword search uses FTS5 via SearchService; hybrid search optionally adds vector results via HybridSearchService.
|
||||
4. Query preprocessing normalizes punctuation-heavy or code-like input before FTS search, while semantic mode bypasses FTS and auto or hybrid mode can fall back to vector retrieval when keyword search produces no candidates.
|
||||
5. Token budgeting walks ranked snippets in order and skips individual over-budget snippets so later matches can still be returned.
|
||||
6. Formatters emit repository and version metadata in JSON responses and origin-aware or explicit no-result text output for plain-text responses.
|
||||
7. MCP handlers expose the same retrieval behavior over stdio or HTTP transports.
|
||||
1. Clients call `/api/v1/libs/search`, `/api/v1/context`, or the MCP tools.
|
||||
2. Route handlers validate input and use the shared SQLite client.
|
||||
3. Keyword search uses FTS5 through `SearchService`; semantic search uses sqlite-vec KNN through `VectorSearch`; hybrid search merges both paths with reciprocal rank fusion.
|
||||
4. Retrieval is scoped by repository and optional version, and semantic/hybrid paths can fall back when keyword search yields no usable candidates.
|
||||
5. Token budgeting selects ranked snippets for the response formatter, which emits repository-aware JSON or text payloads.
|
||||
|
||||
## Build System
|
||||
|
||||
- Build command: npm run build (runs `vite build` then `node scripts/build-workers.mjs`)
|
||||
- Worker bundling: scripts/build-workers.mjs uses esbuild to compile worker-entry.ts and embed-worker-entry.ts into build/workers/ as ESM bundles (.mjs), with $lib path aliases resolved and better-sqlite3/@xenova/transformers marked external
|
||||
- Test command: npm run test
|
||||
- Primary local run command from package.json: npm run dev
|
||||
- MCP entry points: npm run mcp:start and npm run mcp:http
|
||||
- Build command: `npm run build` (runs `vite build` then `node scripts/build-workers.mjs`)
|
||||
- Worker bundling: `scripts/build-workers.mjs` uses esbuild to compile `worker-entry.ts`, `embed-worker-entry.ts`, and `write-worker-entry.ts` into `build/workers/` as ESM bundles
|
||||
- Test command: `npm test`
|
||||
- Primary local run command: `npm run dev`
|
||||
- MCP entry points: `npm run mcp:start` and `npm run mcp:http`
|
||||
|
||||
171
docs/FINDINGS.md
171
docs/FINDINGS.md
@@ -1,29 +1,28 @@
|
||||
# Findings
|
||||
|
||||
Last Updated: 2026-03-30T00:00:00.000Z
|
||||
Last Updated: 2026-04-01T12:05:23.000Z
|
||||
|
||||
## Initializer Summary
|
||||
|
||||
- JIRA: TRUEREF-0022
|
||||
- JIRA: TRUEREF-0023
|
||||
- Refresh mode: REFRESH_IF_REQUIRED
|
||||
- Result: Refreshed ARCHITECTURE.md and FINDINGS.md. CODE_STYLE.md remained trusted — new worker thread code follows established conventions.
|
||||
- Result: Refreshed ARCHITECTURE.md and FINDINGS.md. CODE_STYLE.md remained trusted — sqlite-vec, worker-status, and write-worker additions follow the established conventions already documented.
|
||||
|
||||
## Research Performed
|
||||
|
||||
- Discovered 141 TypeScript/JavaScript source files (up from 110), with new pipeline worker, broadcaster, and SSE endpoint files.
|
||||
- Read worker-pool.ts, worker-entry.ts, embed-worker-entry.ts, worker-types.ts, progress-broadcaster.ts, startup.ts, job-queue.ts to understand the new worker thread architecture.
|
||||
- Read SSE endpoints (jobs/stream, jobs/[id]/stream) and job control endpoints (pause, resume, cancel).
|
||||
- Read indexing settings endpoint and hooks.server.ts to verify startup wiring changes.
|
||||
- Read build-workers.mjs and package.json to verify build system and dependency changes.
|
||||
- Compared trusted cache state with current codebase to identify ARCHITECTURE.md as stale.
|
||||
- Confirmed CODE_STYLE.md conventions still match the codebase — new code uses PascalCase classes, camelCase functions, tab indentation, ESM imports, and TypeScript discriminated unions consistent with existing style.
|
||||
- Counted 149 TypeScript/JavaScript source files in the repository-wide scan and verified the live, non-generated source mix as 147 `.ts` files and 2 `.js` files.
|
||||
- Read `package.json`, `.prettierrc`, and `eslint.config.js` to verify dependencies, formatting rules, and linting conventions.
|
||||
- Read `sqlite-vec.ts`, `sqlite-vec.store.ts`, `vector.search.ts`, `hybrid.search.service.ts`, `schema.ts`, `client.ts`, and startup wiring to verify the accepted sqlite-vec implementation and current retrieval architecture.
|
||||
- Read `worker-pool.ts`, `worker-types.ts`, `write-worker-entry.ts`, and `/api/v1/workers/+server.ts` to verify the current worker topology and status surface.
|
||||
- Compared `docs/docs_cache_state.yaml` against the live docs and codebase to identify stale cache evidence and architecture drift.
|
||||
- Confirmed `CODE_STYLE.md` still matches the codebase: tabs, single quotes, `trailingComma: none`, ESM imports with `node:` built-ins, flat ESLint config, and descriptive PascalCase/camelCase naming remain consistent.
|
||||
|
||||
## Open Questions For Planner
|
||||
|
||||
- Verify whether the retrieval response contract should document the new repository and version metadata fields formally in a public API reference beyond the architecture summary.
|
||||
- Verify whether parser chunking should evolve further from file-level and declaration-level boundaries to member-level semantic chunks for class-heavy codebases.
|
||||
- Verify whether the SSE streaming contract (event names, data shapes) should be documented in a dedicated API reference for external consumers.
|
||||
- Assess whether the WorkerPool fallback mode (main-thread execution when worker scripts are missing) needs explicit test coverage or should be removed in favour of a hard build requirement.
|
||||
- Verify whether the write-worker protocol should become part of the active indexing flow or remain documented as optional infrastructure only.
|
||||
- Verify whether worker-status and SSE event payloads should be documented in a dedicated API reference for external consumers.
|
||||
- Verify whether sqlite-vec operational details such as per-profile vec-table lifecycle and backfill behavior should move into a separate persistence document if the subsystem grows further.
|
||||
- Assess whether the WorkerPool fallback mode (main-thread execution when worker scripts are missing) still belongs in the runtime contract or should be removed in favour of a hard build requirement.
|
||||
|
||||
## Planner Notes Template
|
||||
|
||||
@@ -37,6 +36,41 @@ Add subsequent research below this section.
|
||||
- Findings:
|
||||
- Risks / follow-ups:
|
||||
|
||||
### 2026-04-01 — TRUEREF-0023 initializer refresh audit
|
||||
|
||||
- Task: Refresh only stale or invalid documentation after the accepted sqlite-vec implementation.
|
||||
- Files inspected:
|
||||
- `docs/docs_cache_state.yaml`
|
||||
- `docs/ARCHITECTURE.md`
|
||||
- `docs/CODE_STYLE.md`
|
||||
- `docs/FINDINGS.md`
|
||||
- `package.json`
|
||||
- `.prettierrc`
|
||||
- `eslint.config.js`
|
||||
- `src/hooks.server.ts`
|
||||
- `src/lib/server/db/client.ts`
|
||||
- `src/lib/server/db/schema.ts`
|
||||
- `src/lib/server/db/sqlite-vec.ts`
|
||||
- `src/lib/server/search/sqlite-vec.store.ts`
|
||||
- `src/lib/server/search/vector.search.ts`
|
||||
- `src/lib/server/search/hybrid.search.service.ts`
|
||||
- `src/lib/server/pipeline/startup.ts`
|
||||
- `src/lib/server/pipeline/worker-pool.ts`
|
||||
- `src/lib/server/pipeline/worker-types.ts`
|
||||
- `src/lib/server/pipeline/write-worker-entry.ts`
|
||||
- `src/routes/api/v1/workers/+server.ts`
|
||||
- `scripts/build-workers.mjs`
|
||||
- Findings:
|
||||
- The trusted cache metadata was no longer reliable as evidence for planning: `docs/docs_cache_state.yaml` still referenced 2026-03-27 hashes while `ARCHITECTURE.md` and `FINDINGS.md` had been edited later.
|
||||
- `ARCHITECTURE.md` was stale. It still described only parse and embed worker concurrency, omitted the `sqlite-vec` production dependency, and did not document the current per-profile vec-table storage layer, worker-status endpoint, or write-worker infrastructure.
|
||||
- The current retrieval stack uses sqlite-vec concretely: `loadSqliteVec()` bootstraps connections, `SqliteVecStore` manages vec0 tables plus rowid mapping tables, and `VectorSearch` delegates nearest-neighbor lookup to that store instead of brute-force scoring.
|
||||
- The worker architecture now includes parse, embed, and write worker protocols in `worker-types.ts`, build-time bundling for all three entries, and a `/api/v1/workers` route that returns `WorkerPool` status snapshots.
|
||||
- `CODE_STYLE.md` remained valid and did not require refresh. The observed source and config files still use tabs, single quotes, `trailingComma: none`, flat ESLint config, ESM imports, PascalCase class names, and camelCase helpers exactly as already documented.
|
||||
- `FINDINGS.md` itself was stale because the initializer summary still referred to `TRUEREF-0022` instead of the requested `TRUEREF-0023` refresh.
|
||||
- Risks / follow-ups:
|
||||
- The write-worker protocol exists and is bundled, but the active indexing path is still centered on parse plus optional embed flow. Future documentation should keep distinguishing implemented infrastructure from the currently exercised path.
|
||||
- Cache validity should continue to be driven by deterministic hash evidence rather than document timestamps or trust text alone.
|
||||
|
||||
### 2026-03-27 — FEEDBACK-0001 initializer refresh audit
|
||||
|
||||
- Task: Refresh only stale documentation after changes to retrieval, formatters, token budgeting, and parser behavior.
|
||||
@@ -192,3 +226,112 @@ Add subsequent research below this section.
|
||||
- Risks / follow-ups:
|
||||
- The fix should preserve the existing `/repos/[id]` route shape instead of redesigning it to a rest route unless a broader navigation contract change is explicitly requested.
|
||||
- Any normalization helper introduced for the repo detail page should be reused consistently across server load and client event handlers to avoid mixed encoded and decoded repository IDs during navigation and fetches.
|
||||
|
||||
### 2026-04-01 — TRUEREF-0023 sqlite-vec replanning research
|
||||
|
||||
- Task: Replan the rejected libSQL-native vector iteration around sqlite-vec using the current worktree and verified runtime constraints.
|
||||
- Files inspected:
|
||||
- `package.json`
|
||||
- `docs/docs_cache_state.yaml`
|
||||
- `prompts/TRUEREF-0023/prompt.yaml`
|
||||
- `prompts/TRUEREF-0023/progress.yaml`
|
||||
- `prompts/TRUEREF-0023/iteration_0/review_report.yaml`
|
||||
- `prompts/TRUEREF-0023/iteration_0/plan.md`
|
||||
- `prompts/TRUEREF-0023/iteration_0/tasks.yaml`
|
||||
- `src/lib/server/db/client.ts`
|
||||
- `src/lib/server/db/index.ts`
|
||||
- `src/lib/server/db/schema.ts`
|
||||
- `src/lib/server/db/fts.sql`
|
||||
- `src/lib/server/db/vectors.sql`
|
||||
- `src/lib/server/db/schema.test.ts`
|
||||
- `src/lib/server/search/vector.search.ts`
|
||||
- `src/lib/server/search/hybrid.search.service.test.ts`
|
||||
- `src/lib/server/embeddings/embedding.service.ts`
|
||||
- `src/lib/server/embeddings/embedding.service.test.ts`
|
||||
- `src/lib/server/pipeline/job-queue.ts`
|
||||
- `src/lib/server/pipeline/progress-broadcaster.ts`
|
||||
- `src/lib/server/pipeline/progress-broadcaster.test.ts`
|
||||
- `src/lib/server/pipeline/worker-pool.ts`
|
||||
- `src/lib/server/pipeline/worker-entry.ts`
|
||||
- `src/lib/server/pipeline/embed-worker-entry.ts`
|
||||
- `src/lib/server/pipeline/worker-types.ts`
|
||||
- `src/lib/server/pipeline/startup.ts`
|
||||
- `src/lib/server/pipeline/indexing.pipeline.ts`
|
||||
- `src/lib/server/pipeline/indexing.pipeline.test.ts`
|
||||
- `src/routes/api/v1/jobs/+server.ts`
|
||||
- `src/routes/api/v1/jobs/stream/+server.ts`
|
||||
- `src/routes/api/v1/jobs/[id]/stream/+server.ts`
|
||||
- `src/routes/api/v1/sse-and-settings.integration.test.ts`
|
||||
- `src/routes/admin/jobs/+page.svelte`
|
||||
- `src/lib/components/IndexingProgress.svelte`
|
||||
- `src/lib/components/admin/JobStatusBadge.svelte`
|
||||
- `src/lib/components/admin/JobSkeleton.svelte`
|
||||
- `src/lib/components/admin/Toast.svelte`
|
||||
- `src/lib/components/admin/WorkerStatusPanel.svelte`
|
||||
- `scripts/build-workers.mjs`
|
||||
- `node_modules/libsql/types/index.d.ts`
|
||||
- `node_modules/libsql/index.js`
|
||||
- Findings:
|
||||
- Iteration 0 already changed the workspace materially: direct DB imports were switched from `better-sqlite3` to `libsql`, the extra WAL-related pragmas were added in the main DB clients and embed worker, composite indexes plus `vec_embedding` were added to the Drizzle schema and migration metadata, `IndexingProgress.svelte` now uses SSE, the admin jobs page was overhauled, and `WorkerPool` now serializes on `(repositoryId, versionId)` instead of repository only.
|
||||
- The rejected vector implementation is still invalid in the current tree. `src/lib/server/db/vectors.sql` contains the rejected libSQL-native assumptions, including a dangling `USING libsql_vector_idx(...)` clause with no valid `CREATE INDEX` statement, and `src/lib/server/search/vector.search.ts` still performs full-table JS cosine scoring over `snippet_embeddings` instead of true in-database KNN.
|
||||
- `sqlite-vec` is not currently present in `package.json` or the lockfile, and there is no existing `sqliteVec.load(...)`, `db.loadExtension(...)`, `vec0`, or extension bootstrap code anywhere under `src/`.
|
||||
- Context7 sqlite-vec docs confirm the supported Node integration path is `import * as sqliteVec from 'sqlite-vec'; sqliteVec.load(db);`, storing vectors in a `vec0` virtual table and querying with `WHERE embedding MATCH ? ORDER BY distance LIMIT ?`. The docs also show vec0 metadata columns can be filtered directly, which fits the repositoryId, versionId, and profileId requirements.
|
||||
- Context7 `better-sqlite3` v12.6.2 docs confirm `db.loadExtension(path)` exists. The installed `libsql` package in this workspace also exposes `loadExtension(path): this` in `node_modules/libsql/types/index.d.ts` and `loadExtension(...args)` in `node_modules/libsql/index.js`, so extension loading is not obviously blocked by the driver API surface alone.
|
||||
- The review report remains the only verified runtime evidence for the current libsql path: `vector_from_float32(...)` is unavailable and `libsql_vector_idx` DDL is rejected in this environment. That invalidates the original native-vector approach but does not by itself prove sqlite-vec extension loading succeeds through the current `libsql` package alias, so the replan must include explicit connection-bootstrap and test coverage for real extension loading on the main DB client and worker-owned connections.
|
||||
- Two iteration-0 deliverables referenced in the rejected plan do not exist in the current worktree: `src/lib/server/pipeline/write-worker-entry.ts` and `src/routes/api/v1/workers/+server.ts`. `scripts/build-workers.mjs` and the admin `WorkerStatusPanel.svelte` already reference those missing paths, so iteration 1 must either create them or revert those dangling references as part of a consistent plan.
|
||||
- The existing admin/SSE work is largely salvageable. `src/routes/api/v1/jobs/stream/+server.ts`, `src/routes/api/v1/jobs/[id]/stream/+server.ts`, `src/lib/server/pipeline/progress-broadcaster.ts`, `src/lib/components/IndexingProgress.svelte`, `src/lib/components/admin/JobSkeleton.svelte`, `src/lib/components/admin/Toast.svelte`, and `src/lib/components/admin/WorkerStatusPanel.svelte` provide a usable foundation, but `src/routes/admin/jobs/+page.svelte` still contains `confirm(...)` and the queue API still only supports exact `repository_id = ?` and single-status filtering.
|
||||
- The existing tests still encode the rejected pre-sqlite-vec model: `embedding.service.test.ts`, `schema.test.ts`, `hybrid.search.service.test.ts`, and `indexing.pipeline.test.ts` seed and assert against `snippet_embeddings.embedding` blobs only. The sqlite-vec replan therefore needs new DB bootstrap helpers, vec-table lifecycle assertions, and vector-search tests that validate actual vec0 writes and filtered KNN queries.
|
||||
- Risks / follow-ups:
|
||||
- The current worktree is dirty with iteration-0 partial changes and generated migration metadata, so iteration-1 tasks must explicitly distinguish keep/revise/revert work to avoid sibling tasks fighting over the same files.
|
||||
- Because the current `libsql` package appears to expose `loadExtension`, the replan should avoid assuming an immediate full revert to upstream `better-sqlite3`; instead it should sequence a driver/bootstrap compatibility decision around actual sqlite-vec extension loading behavior with testable acceptance criteria.
|
||||
|
||||
### 2026-04-01 — TRUEREF-0023 iteration-2 current-worktree verification
|
||||
|
||||
- Task: Replan iteration 2 against the post-iteration-1 workspace state so the first validation unit no longer leaves a known vec_embedding mismatch behind.
|
||||
- Files inspected:
|
||||
- `package.json`
|
||||
- `package-lock.json`
|
||||
- `scripts/build-workers.mjs`
|
||||
- `src/lib/server/db/client.ts`
|
||||
- `src/lib/server/db/index.ts`
|
||||
- `src/lib/server/db/schema.ts`
|
||||
- `src/lib/server/db/vectors.sql`
|
||||
- `src/lib/server/db/migrations/0006_yielding_centennial.sql`
|
||||
- `src/lib/server/db/schema.test.ts`
|
||||
- `src/lib/server/embeddings/embedding.service.ts`
|
||||
- `src/lib/server/embeddings/embedding.service.test.ts`
|
||||
- `src/lib/server/search/vector.search.ts`
|
||||
- `src/lib/server/search/hybrid.search.service.ts`
|
||||
- `src/lib/server/search/hybrid.search.service.test.ts`
|
||||
- `src/lib/server/pipeline/job-queue.ts`
|
||||
- `src/lib/server/pipeline/worker-pool.ts`
|
||||
- `src/lib/server/pipeline/worker-entry.ts`
|
||||
- `src/lib/server/pipeline/embed-worker-entry.ts`
|
||||
- `src/lib/server/pipeline/worker-types.ts`
|
||||
- `src/lib/server/pipeline/indexing.pipeline.ts`
|
||||
- `src/lib/server/pipeline/indexing.pipeline.test.ts`
|
||||
- `src/lib/server/pipeline/startup.ts`
|
||||
- `src/lib/server/pipeline/progress-broadcaster.ts`
|
||||
- `src/routes/api/v1/jobs/+server.ts`
|
||||
- `src/routes/api/v1/jobs/stream/+server.ts`
|
||||
- `src/routes/api/v1/sse-and-settings.integration.test.ts`
|
||||
- `src/routes/admin/jobs/+page.svelte`
|
||||
- `src/lib/components/IndexingProgress.svelte`
|
||||
- `src/lib/components/admin/JobStatusBadge.svelte`
|
||||
- `src/lib/components/admin/JobSkeleton.svelte`
|
||||
- `src/lib/components/admin/Toast.svelte`
|
||||
- `src/lib/components/admin/WorkerStatusPanel.svelte`
|
||||
- `prompts/TRUEREF-0023/iteration_1/plan.md`
|
||||
- `prompts/TRUEREF-0023/iteration_1/tasks.yaml`
|
||||
- Findings:
|
||||
- Iteration 1 already completed the direct-driver reset in the working tree: `package.json` and `package-lock.json` now contain real `better-sqlite3` plus `sqlite-vec`, and the current production/test files read in this pass import `better-sqlite3`, not `libsql`.
|
||||
- The remaining failing intermediate state is exactly the schema/write mismatch called out in the review report: `src/lib/server/db/schema.ts` and `src/lib/server/db/migrations/0006_yielding_centennial.sql` still declare `vec_embedding`, `src/lib/server/db/index.ts` still executes `vectors.sql`, and `src/lib/server/embeddings/embedding.service.ts` still inserts `(embedding, vec_embedding)` into `snippet_embeddings`.
|
||||
- `src/lib/server/db/vectors.sql` is still invalid startup SQL. It contains a dangling `USING libsql_vector_idx(...)` clause with no enclosing `CREATE INDEX`, so leaving it in the initialization path keeps the rejected libSQL-native design alive.
|
||||
- The first iteration-1 task boundary was therefore wrong for the current baseline: the package/import reset is already present, but it only becomes a valid foundation once the relational `vec_embedding` artifacts and `EmbeddingService` insert path are cleaned up in the same validation unit.
|
||||
- The current search path is still the pre-sqlite-vec implementation. `src/lib/server/search/vector.search.ts` reads every candidate embedding blob and scores in JavaScript; no `vec0`, `sqliteVec.load(db)`, or sqlite-vec KNN query exists anywhere under `src/` yet.
|
||||
- The write worker and worker-status backend are still missing in the live tree even though they are already referenced elsewhere: `scripts/build-workers.mjs` includes `src/lib/server/pipeline/write-worker-entry.ts`, `src/lib/components/admin/WorkerStatusPanel.svelte` fetches `/api/v1/workers`, and `src/routes/api/v1/jobs/stream/+server.ts` currently has no worker-status event source.
|
||||
- The admin jobs page remains incomplete but salvageable: `src/routes/admin/jobs/+page.svelte` still uses `confirm(...)` and `alert(...)`, while `JobSkeleton.svelte`, `Toast.svelte`, `WorkerStatusPanel.svelte`, `JobStatusBadge.svelte`, and `IndexingProgress.svelte` already provide the intended UI foundation.
|
||||
- `src/lib/server/pipeline/job-queue.ts` still only supports exact `repository_id = ?` and single `status = ?` filtering, so API-side filter work remains a separate backend task and does not need to block the vector-storage implementation.
|
||||
- Risks / follow-ups:
|
||||
- Iteration 2 task decomposition must treat the current dirty code files from iterations 0 and 1 as the validation baseline, otherwise the executor will keep rediscovering pre-existing worktree drift instead of new task deltas.
|
||||
- The sqlite-vec bootstrap helper and the relational cleanup should be planned as one acceptance unit before any downstream vec0, worker-status, or admin-page tasks, because that is the smallest unit that removes the known broken intermediate state.
|
||||
|
||||
Reference in New Issue
Block a user