trueref/docs/features/TRUEREF-0021.md

TRUEREF-0021 — Differential Tag Indexing

Priority: P1 Status: Implemented Depends On: TRUEREF-0014, TRUEREF-0017, TRUEREF-0019 Blocks: —

---

Problem Statement

Repositories with many version tags (e.g. hundreds or thousands, as seen in projects like RWC UXFramework) make full re-indexing prohibitively expensive. Between consecutive semver tags the overwhelming majority of files are unchanged — often only dependency manifests (package.json, *.lock) differ. Indexing the complete file tree for every tag wastes compute time, GitHub API quota, and embedding credits.

---

Solution

Differential tag indexing detects when an already-indexed ancestor version exists for a given target tag, determines exactly which files changed, and:

1. Clones unchanged document rows, snippet rows, and embedding rows from the ancestor version into the target version in a single SQLite transaction (cloneFromAncestor).
2. Crawls only the changed (added / modified) files, parses and embeds them normally.
3. Skips deleted files (not cloned, not crawled).
4. Falls back silently to a full crawl when no indexed ancestor can be found or any step fails.

---

Algorithm

Stage 0 — Differential Plan (buildDifferentialPlan)

Executed in IndexingPipeline.run() before the crawl, when the job has a versionId:

1. Ancestor selection (findBestAncestorVersion in tag-order.ts): Loads all indexed versions for the repository, parses their tags as semver, and returns the closest predecessor to the target tag. Falls back to creation-timestamp ordering for non-semver tags.

2. Changed-file detection: For GitHub repositories, calls the GitHub Compare API (fetchGitHubChangedFiles in github-compare.ts). For local repositories, uses git diff --name-status via getChangedFilesBetweenRefs in git.ts (implemented with execFileSync — not execSync — to prevent shell-injection attacks on branch/tag names containing shell metacharacters).

3. Path partitioning: The changed-file list is split into changedPaths (added + modified

   • renamed-destination) and deletedPaths. unchangedPaths is derived as ancestorFilePaths − changedPaths − deletedPaths.

4. Guard: Returns null when no indexed ancestor exists, when the ancestor has no indexed documents, or when all files changed (nothing to clone).

Stage 0.5 — Clone Unchanged Files (cloneFromAncestor)

When buildDifferentialPlan returns a non-null plan with unchangedPaths.size > 0:

• Fetches ancestor documents rows for the unchanged paths using a parameterised IN (?, ?, …) query (no string interpolation of path values → no SQL injection).
• Inserts new documents rows for each, with new UUIDs and version_id = targetVersionId.
• Fetches ancestor snippets rows for those document IDs; inserts clones with new IDs.
• Fetches ancestor snippet_embeddings rows; inserts clones pointing to the new snippet IDs.
• The entire operation runs inside a single this.db.transaction(…)() call for atomicity.

Stage 1 — Partial Crawl

IndexingPipeline.crawl() accepts an optional third argument allowedPaths?: Set<string>. When provided (set to differentialPlan.changedPaths), the crawl result is filtered so only matching files are returned. This minimises GitHub API requests and local I/O.

---

API Surface Changes

Symbol	Location	Change
buildDifferentialPlan	pipeline/differential-strategy.ts	New — async function
DifferentialPlan	pipeline/differential-strategy.ts	New — interface
findBestAncestorVersion	utils/tag-order.ts	New — pure function
fetchGitHubChangedFiles	crawler/github-compare.ts	New — async function
getChangedFilesBetweenRefs	utils/git.ts	New — sync function (uses execFileSync)
ChangedFile	crawler/types.ts	New — interface
CrawlOptions.allowedPaths	crawler/types.ts	New — optional field
IndexingPipeline.crawl()	pipeline/indexing.pipeline.ts	Modified — added allowedPaths param
IndexingPipeline.cloneFromAncestor()	pipeline/indexing.pipeline.ts	New — private method
IndexingPipeline.run()	pipeline/indexing.pipeline.ts	Modified — Stage 0 added

---

Correctness Properties

• Atomicity: cloneFromAncestor wraps all inserts in one SQLite transaction; a failure leaves the target version with no partially-cloned data.
• Idempotency (fallback): If the clone or plan step fails for any reason, the pipeline catches the error, logs a warning, and continues with a full crawl. No data loss occurs.
• No shell injection: getChangedFilesBetweenRefs uses execFileSync with an argument array rather than execSync with a template-literal string.
• No SQL injection: Path values are never interpolated into SQL strings; only ? placeholders are used.

---

Fallback Conditions

The differential plan returns null (triggering a full crawl) when:

• No versions for this repository have state = 'indexed'.
• The best ancestor has no indexed documents.
• All files changed between ancestor and target (unchangedPaths.size === 0).
• The GitHub Compare API call or git diff call throws an error.
• Any unexpected exception inside buildDifferentialPlan.