fix(FEEDBACK-0001): complete iteration 0 - harden context search

2026-03-27 01:25:46 +01:00
parent e7a2a83cdb
commit 16436bfab2
15 changed files with 1469 additions and 44 deletions
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -0,0 +1,141 @@
+# Architecture
+
+Last Updated: 2026-03-27T00:24:13.000Z
+
+## Overview
+
+TrueRef is a TypeScript-first, self-hosted documentation retrieval platform built on SvelteKit. The repository contains a Node-targeted web application, a REST API, a Model Context Protocol server, and a server-side indexing pipeline backed by SQLite via better-sqlite3 and Drizzle ORM.
+
+- Primary language: TypeScript (110 files) with a small amount of JavaScript configuration (2 files)
+- Application type: Full-stack SvelteKit application with server-side indexing and retrieval services
+- Runtime framework: SvelteKit with adapter-node
+- Storage: SQLite with Drizzle-managed schema plus hand-written FTS5 setup
+- Testing: Vitest with separate client and server projects
+
+## Project Structure
+
+- src/routes: SvelteKit pages and HTTP endpoints, including the public UI and /api/v1 surface
+- src/lib/server: Backend implementation grouped by concern: api, config, crawler, db, embeddings, mappers, models, parser, pipeline, search, services, utils
+- src/mcp: Standalone MCP server entry point and tool handlers
+- static: Static assets such as robots.txt
+- docs/features: Feature-level implementation notes and product documentation
+- build: Generated SvelteKit output
+
+## Key Directories
+
+### src/routes
+
+Contains the UI entry points and API routes. The API tree under src/routes/api/v1 is the public HTTP contract for repository management, indexing jobs, search/context retrieval, settings, filesystem browsing, and JSON schema discovery.
+
+### src/lib/server/db
+
+Owns SQLite schema definitions, migration bootstrapping, and FTS initialization. Database startup runs through initializeDatabase(), which executes Drizzle migrations and then applies FTS5 SQL that cannot be expressed directly in the ORM.
+
+### src/lib/server/pipeline
+
+Coordinates crawl, parse, chunk, store, and optional embedding generation work. Startup recovery marks stale jobs as failed, resets repositories stuck in indexing state, initializes singleton queue/pipeline instances, and drains queued work after restart.
+
+### src/lib/server/search
+
+Implements keyword, vector, and hybrid retrieval. The keyword path uses SQLite FTS5 and BM25; the hybrid path blends FTS and vector search with reciprocal rank fusion.
+
+### src/lib/server/crawler and src/lib/server/parser
+
+Convert GitHub repositories and local folders into normalized snippet records. Crawlers fetch repository contents, parsers split Markdown, code, config, HTML-like, and plain-text files into chunks, and downstream services persist searchable content.
+
+### src/mcp
+
+Provides a thin compatibility layer over the HTTP API. The MCP server exposes resolve-library-id and query-docs over stdio or HTTP and forwards work to local tool handlers.
+
+## Design Patterns
+
+- No explicit design patterns detected from semantic analysis.
+- The implementation does consistently use service classes such as RepositoryService, SearchService, and HybridSearchService for business logic.
+- Mapping and entity layers separate raw database rows from domain objects through mapper/entity pairs such as RepositoryMapper and RepositoryEntity.
+- Pipeline startup uses module-level singleton state for JobQueue and IndexingPipeline lifecycle management.
+
+## Key Components
+
+### SvelteKit server bootstrap
+
+src/hooks.server.ts initializes the database, loads persisted embedding configuration, creates the optional EmbeddingService, starts the indexing pipeline, and applies CORS headers to all /api routes.
+
+### Database layer
+
+src/lib/server/db/schema.ts defines repositories, repository_versions, documents, snippets, embedding_profiles, snippet_embeddings, indexing_jobs, repository_configs, and settings. This schema models the indexed library catalog, retrieval corpus, embedding state, and job tracking.
+
+### Retrieval API
+
+src/routes/api/v1/context/+server.ts validates input, resolves repository and optional version IDs, chooses keyword, semantic, or hybrid retrieval, applies token budgeting that skips oversized snippets instead of stopping early, prepends repository rules, and formats JSON or text responses with repository and version metadata.
+
+### Search engine
+
+src/lib/server/search/search.service.ts preprocesses raw user input into FTS5-safe MATCH expressions before keyword search and repository lookup. src/lib/server/search/hybrid.search.service.ts supports explicit keyword, semantic, and hybrid modes, falls back to vector retrieval when FTS yields no candidates and an embedding provider is configured, and uses reciprocal rank fusion for blended ranking.
+
+### Repository management
+
+src/lib/server/services/repository.service.ts provides CRUD and statistics for indexed repositories, including canonical ID generation for GitHub and local sources.
+
+### MCP surface
+
+src/mcp/index.ts creates the MCP server, registers the two supported tools, and exposes them over stdio or streamable HTTP.
+
+## Dependencies
+
+### Production
+
+- @modelcontextprotocol/sdk: MCP server transport and protocol types
+- @xenova/transformers: local embedding support
+- better-sqlite3: synchronous SQLite driver
+- zod: runtime input validation for MCP tools and server helpers
+
+### Development
+
+- @sveltejs/kit and @sveltejs/adapter-node: application framework and Node deployment target
+- drizzle-kit and drizzle-orm: schema management and typed database access
+- vite and @tailwindcss/vite: bundling and Tailwind integration
+- vitest and @vitest/browser-playwright: server and browser test execution
+- eslint, typescript-eslint, eslint-plugin-svelte, prettier, prettier-plugin-svelte, prettier-plugin-tailwindcss: linting and formatting
+- typescript and @types/node: type-checking and Node typings
+
+## Module Organization
+
+The backend is organized by responsibility rather than by route. HTTP handlers in src/routes/api/v1 are intentionally thin and delegate to library modules in src/lib/server. Within src/lib/server, concerns are separated into:
+
+- models and mappers for entity translation
+- services for repository/version operations
+- search for retrieval strategies
+- crawler and parser for indexing input transformation
+- pipeline for orchestration and job execution
+- embeddings for provider abstraction and embedding generation
+- api and utils for response formatting, validation, and shared helpers
+
+The frontend and backend share the same SvelteKit repository, but most non-UI behavior is implemented on the server side.
+
+## Data Flow
+
+### Indexing flow
+
+1. Server startup runs initializeDatabase() and initializePipeline() from src/hooks.server.ts.
+2. The pipeline recovers stale jobs, initializes crawler/parser infrastructure, and resumes queued work.
+3. Crawlers ingest GitHub or local repository contents.
+4. Parsers split files into document and snippet records with token counts and metadata.
+5. Database modules persist repositories, documents, snippets, versions, configs, and job state.
+6. If an embedding provider is configured, embedding services generate vectors for snippet search.
+
+### Retrieval flow
+
+1. Clients call /api/v1/libs/search, /api/v1/context, or the MCP tools.
+2. Route handlers validate input and load the SQLite client.
+3. Keyword search uses FTS5 via SearchService; hybrid search optionally adds vector results via HybridSearchService.
+4. Query preprocessing normalizes punctuation-heavy or code-like input before FTS search, while semantic mode bypasses FTS and auto or hybrid mode can fall back to vector retrieval when keyword search produces no candidates.
+5. Token budgeting walks ranked snippets in order and skips individual over-budget snippets so later matches can still be returned.
+6. Formatters emit repository and version metadata in JSON responses and origin-aware or explicit no-result text output for plain-text responses.
+7. MCP handlers expose the same retrieval behavior over stdio or HTTP transports.
+
+## Build System
+
+- Build command: npm run build
+- Test command: npm run test
+- Primary local run command from package.json: npm run dev
+- MCP entry points: npm run mcp:start and npm run mcp:http
--- a/docs/CODE_STYLE.md
+++ b/docs/CODE_STYLE.md
@@ -0,0 +1,156 @@
+# Code Style
+
+Last Updated: 2026-03-26T23:52:10.000Z
+
+## Language And Tooling
+
+- Language: TypeScript with strict mode enabled in tsconfig.json
+- Framework conventions: SvelteKit route files and server modules
+- Formatter: Prettier
+- Linter: ESLint flat config with JavaScript, TypeScript, and Svelte recommended presets
+- Test style: Vitest for server and browser-facing tests
+
+## Naming Conventions
+
+The codebase uses descriptive PascalCase for classes, interfaces, and domain model types.
+
+Examples:
+
+- RepositoryService
+- HybridSearchService
+- EmbeddingService
+- SearchResultMapper
+- RepositoryEntity
+- RawSnippetRow
+
+Functions, variables, and exported helpers use camelCase.
+
+Examples:
+
+- initializeDatabase
+- recoverStaleJobs
+- parseCodeFile
+- selectSnippetsWithinBudget
+- createMcpServer
+
+Database tables and persisted columns use snake_case, matching SQLite conventions.
+
+Examples from schema.ts:
+
+- repository_versions
+- total_snippets
+- source_url
+- indexed_at
+
+Constants use UPPER_SNAKE_CASE when they represent configuration or shared immutable values.
+
+Examples:
+
+- EMBEDDING_CONFIG_KEY
+- DEFAULT_TOKEN_BUDGET
+- BOUNDARY_PATTERNS
+- CORS_HEADERS
+
+## Indentation And Formatting
+
+- Tabs are the project-wide indentation style
+- Single quotes are preferred
+- Trailing commas are disabled
+- Print width is 100
+- Svelte files are formatted with the Svelte parser and Tailwind-aware class sorting
+
+These settings are enforced in .prettierrc.
+
+## Import Patterns
+
+The project favors ES module syntax everywhere.
+
+Observed patterns:
+
+- Node built-ins use node: specifiers, for example import { parseArgs } from 'node:util'
+- Internal modules often use the SvelteKit $lib alias
+- Many TypeScript server modules include .js on relative runtime imports to align with emitted ESM paths
+- Type-only imports are used where appropriate
+
+Examples:
+
+```ts
+import type { RequestHandler } from './$types';
+import { getClient } from '$lib/server/db/client';
+import { IndexingPipeline } from './indexing.pipeline.js';
+import { parseArgs } from 'node:util';
+```
+
+## Comments And Docstrings
+
+The codebase uses structured block comments more often than inline comments. Files commonly start with a short header describing the feature or subsystem, and larger modules use banner separators to divide helper sections, public types, and exported behavior.
+
+Representative style:
+
+```ts
+/**
+ * SearchService — FTS5-backed full-text search over snippets and repositories.
+ *
+ * Implements keyword search using SQLite's built-in BM25 ranking.
+ */
+```
+
+Inline comments are used sparingly and usually explain behavior that is runtime-specific, such as fallback logic, recovery semantics, or SQL limitations.
+
+## Code Examples
+
+Representative function declaration style:
+
+```ts
+export function recoverStaleJobs(db: Database.Database): void {
+	db.prepare(
+		`UPDATE indexing_jobs
+	 SET status      = 'failed',
+		 error       = 'Server restarted while job was running',
+		 completed_at = unixepoch()
+	 WHERE status = 'running'`
+	).run();
+}
+```
+
+Representative interface style:
+
+```ts
+export interface AddRepositoryInput {
+	source: 'github' | 'local';
+	sourceUrl: string;
+	title?: string;
+	description?: string;
+	branch?: string;
+	githubToken?: string;
+}
+```
+
+Representative object-literal and configuration style:
+
+```ts
+export default defineConfig({
+	plugins: [tailwindcss(), sveltekit()],
+	test: {
+		expect: { requireAssertions: true }
+	}
+});
+```
+
+## Linting Configuration
+
+eslint.config.js defines a flat-config stack with:
+
+- @eslint/js recommended rules
+- typescript-eslint recommended rules
+- eslint-plugin-svelte recommended rules
+- prettier compatibility presets
+- shared browser and node globals
+- no-undef explicitly disabled for TypeScript files
+- Svelte parserOptions configured with projectService and svelteConfig
+
+This indicates the project expects type-aware linting for Svelte and TypeScript rather than relying on formatting-only enforcement.
+
+## Style Summary
+
+The dominant style is pragmatic server-side TypeScript: explicit types, small interfaces for inputs/options, banner-commented modules, thin route handlers, and class-based services around synchronous SQLite access. Naming is consistent, import usage is modern ESM, and formatting is standardized through Prettier rather than ad hoc conventions.
--- a/docs/FINDINGS.md
+++ b/docs/FINDINGS.md
@@ -0,0 +1,121 @@
+# Findings
+
+Last Updated: 2026-03-27T00:24:13.000Z
+
+## Initializer Summary
+
+- JIRA: FEEDBACK-0001
+- Refresh mode: REFRESH_IF_REQUIRED
+- Result: refreshed affected documentation only. ARCHITECTURE.md and FINDINGS.md were updated from current repository analysis; CODE_STYLE.md remained trusted and unchanged because the documented conventions still match the codebase.
+
+## Research Performed
+
+- Discovered source-language distribution, dependency manifest, import patterns, and project structure.
+- Read the retrieval, formatter, token-budget, parser, mapper, and response-model modules affected by the latest implementation changes.
+- Compared the trusted cache state with current behavior to identify which documentation files were actually stale.
+- Confirmed package scripts for build and test.
+- Confirmed Linux-native md5sum availability for documentation trust metadata.
+
+## Open Questions For Planner
+
+- Verify whether the retrieval response contract should document the new repository and version metadata fields formally in a public API reference beyond the architecture summary.
+- Verify whether parser chunking should evolve further from file-level and declaration-level boundaries to member-level semantic chunks for class-heavy codebases.
+
+## Planner Notes Template
+
+Add subsequent research below this section.
+
+### Entry Template
+
+- Date:
+- Task:
+- Files inspected:
+- Findings:
+- Risks / follow-ups:
+
+### 2026-03-27 — FEEDBACK-0001 initializer refresh audit
+
+- Task: Refresh only stale documentation after changes to retrieval, formatters, token budgeting, and parser behavior.
+- Files inspected:
+	- `docs/docs_cache_state.yaml`
+	- `docs/ARCHITECTURE.md`
+	- `docs/CODE_STYLE.md`
+	- `docs/FINDINGS.md`
+	- `package.json`
+	- `src/routes/api/v1/context/+server.ts`
+	- `src/lib/server/api/formatters.ts`
+	- `src/lib/server/api/token-budget.ts`
+	- `src/lib/server/search/query-preprocessor.ts`
+	- `src/lib/server/search/search.service.ts`
+	- `src/lib/server/search/hybrid.search.service.ts`
+	- `src/lib/server/mappers/context-response.mapper.ts`
+	- `src/lib/server/models/context-response.ts`
+	- `src/lib/server/models/search-result.ts`
+	- `src/lib/server/parser/index.ts`
+	- `src/lib/server/parser/code.parser.ts`
+	- `src/lib/server/parser/markdown.parser.ts`
+- Findings:
+	- The documentation cache was trusted, but the architecture summary no longer captured current retrieval behavior: query preprocessing now sanitizes punctuation-heavy input for FTS5, semantic mode can bypass FTS entirely, and auto or hybrid retrieval can fall back to vector search when keyword search returns no candidates.
+	- Plain-text and JSON context formatting now carry repository and version metadata, and the text formatter emits an explicit no-results section instead of an empty body.
+	- Token budgeting now skips individual over-budget snippets and continues evaluating lower-ranked candidates, which changes the response-selection behavior described at the architecture level.
+	- Parser coverage now explicitly includes Markdown, code, config, HTML-like, and plain-text inputs, so the architecture summary needed to reflect that broader file-type handling.
+	- The conventions documented in CODE_STYLE.md still match the current repository: strict TypeScript, tab indentation, ESM imports, Prettier and ESLint flat config, and pragmatic service-oriented server modules.
+- Risks / follow-ups:
+	- Future cache invalidation should continue to distinguish between behavioral changes that affect architecture docs and localized implementation changes that do not affect the style guide.
+	- If the public API contract becomes externally versioned, the new context metadata fields likely deserve a dedicated API document instead of only architecture-level coverage.
+
+### 2026-03-27 — FEEDBACK-0001 planning research
+
+- Task: Plan the retrieval-fix iteration covering FTS query safety, hybrid fallback, empty-result behavior, result metadata, token budgeting, and parser chunking.
+- Files inspected:
+	- `package.json`
+	- `src/routes/api/v1/context/+server.ts`
+	- `src/lib/server/search/query-preprocessor.ts`
+	- `src/lib/server/search/search.service.ts`
+	- `src/lib/server/search/hybrid.search.service.ts`
+	- `src/lib/server/search/vector.search.ts`
+	- `src/lib/server/api/token-budget.ts`
+	- `src/lib/server/api/formatters.ts`
+	- `src/lib/server/mappers/context-response.mapper.ts`
+	- `src/lib/server/models/context-response.ts`
+	- `src/lib/server/models/search-result.ts`
+	- `src/lib/server/parser/code.parser.ts`
+	- `src/lib/server/search/search.service.test.ts`
+	- `src/lib/server/search/hybrid.search.service.test.ts`
+	- `src/lib/server/api/formatters.test.ts`
+	- `src/lib/server/parser/code.parser.test.ts`
+	- `src/routes/api/v1/api-contract.integration.test.ts`
+	- `src/mcp/tools/query-docs.ts`
+	- `src/mcp/client.ts`
+- Findings:
+	- `better-sqlite3` `^12.6.2` backs the affected search path; the code already uses bound parameters for `MATCH`, so the practical fix belongs in query normalization and fallback handling rather than SQL string construction.
+	- `query-preprocessor.ts` only strips parentheses and appends a trailing wildcard. Other code-like punctuation currently reaches the FTS execution path unsanitized.
+	- `search.service.ts` sends the preprocessed text directly to `snippets_fts MATCH ?` and already returns `[]` for blank processed queries.
+	- `hybrid.search.service.ts` always executes keyword search before semantic branching. In the current flow, an FTS parse failure can abort `auto`, `hybrid`, and `semantic` requests before vector retrieval runs.
+	- `vector.search.ts` already preserves `repositoryId`, `versionId`, and `profileId` filtering and does not need architectural changes for this iteration.
+	- `token-budget.ts` stops at the first over-budget snippet instead of skipping that item and continuing through later ranked results.
+	- `formatContextTxt([], [])` returns an empty string, so `/api/v1/context?type=txt` can emit an empty `200 OK` body today.
+	- `context-response.mapper.ts` and `context-response.ts` expose snippet content and breadcrumb/page title but do not identify local TrueRef origin, repository source metadata, or normalized snippet origin labels.
+	- `code.parser.ts` splits primarily at top-level declarations; class/object member functions remain in coarse chunks, which limits method-level recall for camelCase API queries.
+	- Existing relevant automated coverage is concentrated in the search, formatter, and parser unit tests; `/api/v1/context` contract coverage currently omits the context endpoint entirely.
+- Risks / follow-ups:
+	- Response-shape changes must be additive because `src/mcp/client.ts`, `src/mcp/tools/query-docs.ts`, and UI consumers expect the current top-level keys to remain present.
+	- Parser improvements should stay inside `parseCodeFile()` and existing chunking helpers to avoid turning this fix iteration into a schema or pipeline redesign.
+
+### 2026-03-27 — FEEDBACK-0001 SQLite FTS5 syntax research
+
+- Task: Verify the FTS5 query-grammar constraints that affect punctuation-heavy local search queries.
+- Files inspected:
+	- `package.json`
+	- `src/lib/server/search/query-preprocessor.ts`
+	- `src/lib/server/search/search.service.ts`
+	- `src/lib/server/search/hybrid.search.service.ts`
+- Findings:
+	- `better-sqlite3` is pinned at `^12.6.2` in `package.json`, and the application binds the `MATCH` string as a parameter instead of interpolating SQL directly.
+	- The canonical SQLite FTS5 docs state that barewords may contain letters, digits, underscore, non-ASCII characters, and the substitute character; strings containing other punctuation must be quoted or they become syntax errors in `MATCH` expressions.
+	- The same docs state that prefix search is expressed by placing `*` after the token or phrase, not inside quotes, which matches the current trailing-wildcard strategy in `query-preprocessor.ts`.
+	- SQLite documents that FTS5 is stricter than FTS3/4 about unrecognized punctuation in query strings, which confirms that code-like user input should be normalized before it reaches `snippets_fts MATCH ?`.
+	- Based on the current code path, the practical fix remains application-side sanitization and fallback behavior in `query-preprocessor.ts` and `hybrid.search.service.ts`, not SQL construction changes.
+- Risks / follow-ups:
+	- Over-sanitizing punctuation-heavy inputs could erase useful identifiers, so the implementation should preserve searchable alphanumeric and underscore tokens while discarding grammar-breaking punctuation.
+	- Prefix expansion should remain on the final searchable token only so the fix preserves current query-cost expectations and test semantics.