trueref/docs/FINDINGS.md

Findings

Last Updated: 2026-03-27T00:24:13.000Z

Initializer Summary

• JIRA: FEEDBACK-0001
• Refresh mode: REFRESH_IF_REQUIRED
• Result: refreshed affected documentation only. ARCHITECTURE.md and FINDINGS.md were updated from current repository analysis; CODE_STYLE.md remained trusted and unchanged because the documented conventions still match the codebase.

Research Performed

• Discovered source-language distribution, dependency manifest, import patterns, and project structure.
• Read the retrieval, formatter, token-budget, parser, mapper, and response-model modules affected by the latest implementation changes.
• Compared the trusted cache state with current behavior to identify which documentation files were actually stale.
• Confirmed package scripts for build and test.
• Confirmed Linux-native md5sum availability for documentation trust metadata.

Open Questions For Planner

• Verify whether the retrieval response contract should document the new repository and version metadata fields formally in a public API reference beyond the architecture summary.
• Verify whether parser chunking should evolve further from file-level and declaration-level boundaries to member-level semantic chunks for class-heavy codebases.

Planner Notes Template

Add subsequent research below this section.

Entry Template

• Date:
• Task:
• Files inspected:
• Findings:
• Risks / follow-ups:

2026-03-27 — FEEDBACK-0001 initializer refresh audit

• Task: Refresh only stale documentation after changes to retrieval, formatters, token budgeting, and parser behavior.
• Files inspected:
  • docs/docs_cache_state.yaml
  • docs/ARCHITECTURE.md
  • docs/CODE_STYLE.md
  • docs/FINDINGS.md
  • package.json
  • src/routes/api/v1/context/+server.ts
  • src/lib/server/api/formatters.ts
  • src/lib/server/api/token-budget.ts
  • src/lib/server/search/query-preprocessor.ts
  • src/lib/server/search/search.service.ts
  • src/lib/server/search/hybrid.search.service.ts
  • src/lib/server/mappers/context-response.mapper.ts
  • src/lib/server/models/context-response.ts
  • src/lib/server/models/search-result.ts
  • src/lib/server/parser/index.ts
  • src/lib/server/parser/code.parser.ts
  • src/lib/server/parser/markdown.parser.ts
• Findings:
  • The documentation cache was trusted, but the architecture summary no longer captured current retrieval behavior: query preprocessing now sanitizes punctuation-heavy input for FTS5, semantic mode can bypass FTS entirely, and auto or hybrid retrieval can fall back to vector search when keyword search returns no candidates.
  • Plain-text and JSON context formatting now carry repository and version metadata, and the text formatter emits an explicit no-results section instead of an empty body.
  • Token budgeting now skips individual over-budget snippets and continues evaluating lower-ranked candidates, which changes the response-selection behavior described at the architecture level.
  • Parser coverage now explicitly includes Markdown, code, config, HTML-like, and plain-text inputs, so the architecture summary needed to reflect that broader file-type handling.
  • The conventions documented in CODE_STYLE.md still match the current repository: strict TypeScript, tab indentation, ESM imports, Prettier and ESLint flat config, and pragmatic service-oriented server modules.
• Risks / follow-ups:
  • Future cache invalidation should continue to distinguish between behavioral changes that affect architecture docs and localized implementation changes that do not affect the style guide.
  • If the public API contract becomes externally versioned, the new context metadata fields likely deserve a dedicated API document instead of only architecture-level coverage.

2026-03-27 — FEEDBACK-0001 planning research

• Task: Plan the retrieval-fix iteration covering FTS query safety, hybrid fallback, empty-result behavior, result metadata, token budgeting, and parser chunking.
• Files inspected:
  • package.json
  • src/routes/api/v1/context/+server.ts
  • src/lib/server/search/query-preprocessor.ts
  • src/lib/server/search/search.service.ts
  • src/lib/server/search/hybrid.search.service.ts
  • src/lib/server/search/vector.search.ts
  • src/lib/server/api/token-budget.ts
  • src/lib/server/api/formatters.ts
  • src/lib/server/mappers/context-response.mapper.ts
  • src/lib/server/models/context-response.ts
  • src/lib/server/models/search-result.ts
  • src/lib/server/parser/code.parser.ts
  • src/lib/server/search/search.service.test.ts
  • src/lib/server/search/hybrid.search.service.test.ts
  • src/lib/server/api/formatters.test.ts
  • src/lib/server/parser/code.parser.test.ts
  • src/routes/api/v1/api-contract.integration.test.ts
  • src/mcp/tools/query-docs.ts
  • src/mcp/client.ts
• Findings:
  • better-sqlite3 ^12.6.2 backs the affected search path; the code already uses bound parameters for MATCH, so the practical fix belongs in query normalization and fallback handling rather than SQL string construction.
  • query-preprocessor.ts only strips parentheses and appends a trailing wildcard. Other code-like punctuation currently reaches the FTS execution path unsanitized.
  • search.service.ts sends the preprocessed text directly to snippets_fts MATCH ? and already returns [] for blank processed queries.
  • hybrid.search.service.ts always executes keyword search before semantic branching. In the current flow, an FTS parse failure can abort auto, hybrid, and semantic requests before vector retrieval runs.
  • vector.search.ts already preserves repositoryId, versionId, and profileId filtering and does not need architectural changes for this iteration.
  • token-budget.ts stops at the first over-budget snippet instead of skipping that item and continuing through later ranked results.
  • formatContextTxt([], []) returns an empty string, so /api/v1/context?type=txt can emit an empty 200 OK body today.
  • context-response.mapper.ts and context-response.ts expose snippet content and breadcrumb/page title but do not identify local TrueRef origin, repository source metadata, or normalized snippet origin labels.
  • code.parser.ts splits primarily at top-level declarations; class/object member functions remain in coarse chunks, which limits method-level recall for camelCase API queries.
  • Existing relevant automated coverage is concentrated in the search, formatter, and parser unit tests; /api/v1/context contract coverage currently omits the context endpoint entirely.
• Risks / follow-ups:
  • Response-shape changes must be additive because src/mcp/client.ts, src/mcp/tools/query-docs.ts, and UI consumers expect the current top-level keys to remain present.
  • Parser improvements should stay inside parseCodeFile() and existing chunking helpers to avoid turning this fix iteration into a schema or pipeline redesign.

2026-03-27 — FEEDBACK-0001 SQLite FTS5 syntax research

• Task: Verify the FTS5 query-grammar constraints that affect punctuation-heavy local search queries.
• Files inspected:
  • package.json
  • src/lib/server/search/query-preprocessor.ts
  • src/lib/server/search/search.service.ts
  • src/lib/server/search/hybrid.search.service.ts
• Findings:
  • better-sqlite3 is pinned at ^12.6.2 in package.json, and the application binds the MATCH string as a parameter instead of interpolating SQL directly.
  • The canonical SQLite FTS5 docs state that barewords may contain letters, digits, underscore, non-ASCII characters, and the substitute character; strings containing other punctuation must be quoted or they become syntax errors in MATCH expressions.
  • The same docs state that prefix search is expressed by placing * after the token or phrase, not inside quotes, which matches the current trailing-wildcard strategy in query-preprocessor.ts.
  • SQLite documents that FTS5 is stricter than FTS3/4 about unrecognized punctuation in query strings, which confirms that code-like user input should be normalized before it reaches snippets_fts MATCH ?.
  • Based on the current code path, the practical fix remains application-side sanitization and fallback behavior in query-preprocessor.ts and hybrid.search.service.ts, not SQL construction changes.
• Risks / follow-ups:
  • Over-sanitizing punctuation-heavy inputs could erase useful identifiers, so the implementation should preserve searchable alphanumeric and underscore tokens while discarding grammar-breaking punctuation.
  • Prefix expansion should remain on the final searchable token only so the fix preserves current query-cost expectations and test semantics.