# TRUEREF-0017 — Incremental Re-indexing (Checksum Diff)

**Priority:** P1
**Status:** Pending
**Depends On:** TRUEREF-0009
**Blocks:** —

---

## Overview

Optimize re-indexing by skipping files that haven't changed since the last indexing run. Uses file checksums (SHA-256) to detect changes. Only modified, added, or deleted files trigger parser/embedding work. This dramatically reduces re-indexing time for large repositories.

---

## Acceptance Criteria

- [ ] Checksum comparison before parsing each file
- [ ] Unchanged files reuse existing `Document` and `Snippet` records (no re-parse, no re-embed)
- [ ] New files: full parse + embed
- [ ] Modified files: delete old snippets, parse new ones, re-embed
- [ ] Deleted files (present in DB but not in new crawl): delete documents and snippets
- [ ] Job progress reflects total files (including skipped), not just processed
- [ ] Statistics updated correctly after incremental run
- [ ] Integration test covering unchanged, modified, added, and deleted files

---

## Diff Algorithm

```typescript
interface FileDiff {
  added: CrawledFile[];      // new files not in DB
  modified: CrawledFile[];   // files with changed checksum
  deleted: string[];         // file paths in DB but not in crawl
  unchanged: string[];       // file paths with matching checksum
}

function computeDiff(
  crawledFiles: CrawledFile[],
  existingDocs: Document[]   // documents currently in DB for this repo
): FileDiff {
  const existingMap = new Map(existingDocs.map(d => [d.filePath, d]));
  const crawledMap = new Map(crawledFiles.map(f => [f.path, f]));

  const added: CrawledFile[] = [];
  const modified: CrawledFile[] = [];
  const unchanged: string[] = [];

  for (const file of crawledFiles) {
    const existing = existingMap.get(file.path);
    if (!existing) {
      added.push(file);
    } else if (existing.checksum !== file.sha) {
      modified.push(file);
    } else {
      unchanged.push(file.path);
    }
  }

  const deleted = existingDocs
    .filter(doc => !crawledMap.has(doc.filePath))
    .map(doc => doc.filePath);

  return { added, modified, deleted, unchanged };
}
```

---

## Integration with IndexingPipeline

```typescript
// In IndexingPipeline.run(), after crawling:

const existingDocs = this.getExistingDocuments(repo.id, job.versionId);
const diff = computeDiff(crawledResult.files, existingDocs);

// Log diff summary
this.updateJob(job.id, {
  totalFiles: crawledResult.files.length,
});

// Process only changed/new files
const filesToProcess = [...diff.added, ...diff.modified];
const newSnippets: NewSnippet[] = [];
const newDocuments: NewDocument[] = [];
const docIdsToDelete: string[] = [];

// Map modified files to their existing document IDs for deletion
for (const file of diff.modified) {
  const existing = existingDocs.find(d => d.filePath === file.path);
  if (existing) docIdsToDelete.push(existing.id);
}

// Map deleted file paths to document IDs
for (const filePath of diff.deleted) {
  const existing = existingDocs.find(d => d.filePath === filePath);
  if (existing) docIdsToDelete.push(existing.id);
}

// Parse new/modified files
for (const [i, file] of filesToProcess.entries()) {
  const docId = crypto.randomUUID();
  newDocuments.push({ id: docId, ...buildDocument(file, repo.id, job.versionId) });
  newSnippets.push(...parseFile(file, { repositoryId: repo.id, documentId: docId }));

  // Count ALL files (including skipped) in progress
  const totalProcessed = diff.unchanged.length + i + 1;
  const progress = Math.round((totalProcessed / crawledResult.files.length) * 80);
  this.updateJob(job.id, {
    processedFiles: totalProcessed,
    progress,
  });
}

// Atomic replacement of only changed documents
this.replaceSnippets(repo.id, docIdsToDelete, newDocuments, newSnippets);
```

---

## Performance Impact

For a typical repository with 1,000 files where 50 changed:
- **Without incremental**: 1,000 files parsed + 1,000 embed batches
- **With incremental**: 50 files parsed + 50 embed batches
- Estimated speedup: ~20x for re-indexing

---

## Files to Modify

- `src/lib/server/pipeline/indexing.pipeline.ts` — add diff computation
- `src/lib/server/pipeline/diff.ts` — `computeDiff` function (new file)
- `src/lib/server/pipeline/diff.test.ts` — unit tests
