366 lines
16 KiB
Markdown
366 lines
16 KiB
Markdown
# TrueRef — Product Requirements Document
|
|
|
|
**Version:** 1.0
|
|
**Date:** 2026-03-22
|
|
**Status:** Draft
|
|
|
|
---
|
|
|
|
## 1. Executive Summary
|
|
|
|
TrueRef is a self-hosted, open-source documentation intelligence platform — a full-stack clone of context7 that lets teams index, search, and query their own private or public code repositories. Unlike context7 (which has a private indexing backend), TrueRef ships the complete stack: crawler, parser, indexer, semantic search engine, REST API, MCP server, and a management web UI.
|
|
|
|
The core use case is enabling AI coding assistants (Claude Code, Cursor, Zed, etc.) to retrieve accurate, up-to-date, version-aware documentation from repositories the user controls — without sending code to third-party services.
|
|
|
|
---
|
|
|
|
## 2. Problem Statement
|
|
|
|
### 2.1 Context7's Limitations
|
|
- The indexing and crawling backend is entirely private and closed-source.
|
|
- Only public libraries already in the context7.com catalog are available.
|
|
- Private, internal, or niche repositories cannot be added.
|
|
- Data sovereignty: all queries go to context7.com servers.
|
|
- No way to self-host for air-gapped or compliance-constrained environments.
|
|
|
|
### 2.2 The Gap
|
|
Teams with internal SDKs, private libraries, proprietary documentation, or a need for data sovereignty have no tooling that provides context7-equivalent LLM documentation retrieval.
|
|
|
|
---
|
|
|
|
## 3. Goals & Non-Goals
|
|
|
|
### Goals
|
|
- Replicate all context7 capabilities: library search, documentation retrieval, MCP tools (`resolve-library-id`, `query-docs`).
|
|
- Support both GitHub-hosted and local filesystem repositories.
|
|
- Provide a full indexing pipeline: crawl → parse → chunk → embed → store → query.
|
|
- Expose a REST API compatible with context7's `/api/v2/*` surface.
|
|
- Ship an MCP server implementing `resolve-library-id` and `query-docs`.
|
|
- Provide a web UI for repository management and search exploration.
|
|
- Support `trueref.json` config files in repos (analogous to `context7.json`).
|
|
- Support versioned documentation via git tags/branches.
|
|
- Self-hostable with minimal dependencies (SQLite-first, no external vector DB required).
|
|
|
|
### Non-Goals (v1)
|
|
- Authentication & authorization (deferred to a future version).
|
|
- Skill generation (context7 CLI skill feature).
|
|
- Multi-tenant SaaS mode.
|
|
- Binary pre-built releases / Docker image (infrastructure, not product).
|
|
- Paid API tier / rate limiting.
|
|
- Support for non-git version control systems.
|
|
|
|
---
|
|
|
|
## 4. Users & Personas
|
|
|
|
### Primary: The Developer / Tech Lead
|
|
Configures TrueRef, adds repositories, integrates the MCP server with their AI coding assistant. Technical, comfortable with CLI and config files.
|
|
|
|
### Secondary: The AI Coding Assistant
|
|
The "user" at query time. Calls `resolve-library-id` and `query-docs` via MCP to retrieve documentation snippets for code generation.
|
|
|
|
---
|
|
|
|
## 5. Architecture Overview
|
|
|
|
```
|
|
┌────────────────────────────────────────────────────────────────────┐
|
|
│ TrueRef Platform │
|
|
│ │
|
|
│ ┌──────────────────┐ ┌──────────────────┐ ┌───────────────┐ │
|
|
│ │ Web UI │ │ REST API │ │ MCP Server │ │
|
|
│ │ (SvelteKit) │ │ /api/v1/* │ │ (stdio/HTTP) │ │
|
|
│ └────────┬─────────┘ └────────┬─────────┘ └───────┬───────┘ │
|
|
│ │ │ │ │
|
|
│ └──────────────────────┼─────────────────────┘ │
|
|
│ │ │
|
|
│ ┌─────────────▼──────────────┐ │
|
|
│ │ Service Layer │ │
|
|
│ │ LibraryService │ │
|
|
│ │ SearchService │ │
|
|
│ │ IndexingService │ │
|
|
│ └─────────────┬──────────────┘ │
|
|
│ │ │
|
|
│ ┌──────────────────────┼───────────────────┐ │
|
|
│ │ │ │ │
|
|
│ ┌────────▼────────┐ ┌─────────▼──────┐ ┌────────▼──────────┐ │
|
|
│ │ Indexing │ │ SQLite DB │ │ Vector/FTS Index │ │
|
|
│ │ Pipeline │ │ (drizzle-orm) │ │ (SQLite FTS5 + │ │
|
|
│ │ Crawler │ │ │ │ embeddings) │ │
|
|
│ │ Parser │ │ │ │ │ │
|
|
│ │ Chunker │ │ │ │ │ │
|
|
│ └────────┬────────┘ └───────────────┘ └────────────────────┘ │
|
|
│ │ │
|
|
│ ┌───────▼──────────────────────┐ │
|
|
│ │ Repository Sources │ │
|
|
│ │ - GitHub API │ │
|
|
│ │ - Local filesystem │ │
|
|
│ └──────────────────────────────┘ │
|
|
└────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Technology Stack
|
|
| Layer | Technology |
|
|
|-------|-----------|
|
|
| Framework | SvelteKit (Node adapter) |
|
|
| Language | TypeScript |
|
|
| Database | SQLite via better-sqlite3 + drizzle-orm |
|
|
| Full-Text Search | SQLite FTS5 |
|
|
| Vector Search | SQLite `sqlite-vec` extension (cosine similarity) |
|
|
| Embeddings | Pluggable: local (transformers.js / ONNX) or OpenAI-compatible API |
|
|
| MCP Protocol | `@modelcontextprotocol/sdk` |
|
|
| HTTP | SvelteKit API routes + optional standalone MCP HTTP server |
|
|
| CSS | TailwindCSS v4 |
|
|
| Testing | Vitest |
|
|
| Linting | ESLint + Prettier |
|
|
|
|
---
|
|
|
|
## 6. Data Model
|
|
|
|
### 6.1 Repositories
|
|
A `Repository` is the top-level entity. It maps to a GitHub repo or local directory.
|
|
|
|
```
|
|
Repository {
|
|
id TEXT PRIMARY KEY -- e.g. "/facebook/react"
|
|
title TEXT NOT NULL -- display name
|
|
description TEXT
|
|
source TEXT NOT NULL -- "github" | "local"
|
|
sourceUrl TEXT -- GitHub URL or local path
|
|
branch TEXT -- default branch
|
|
state TEXT NOT NULL -- "pending" | "indexing" | "indexed" | "error"
|
|
totalSnippets INTEGER DEFAULT 0
|
|
totalTokens INTEGER DEFAULT 0
|
|
trustScore REAL DEFAULT 0
|
|
stars INTEGER
|
|
lastIndexedAt DATETIME
|
|
createdAt DATETIME
|
|
updatedAt DATETIME
|
|
}
|
|
```
|
|
|
|
### 6.2 Repository Versions
|
|
```
|
|
RepositoryVersion {
|
|
id TEXT PRIMARY KEY
|
|
repositoryId TEXT FK → Repository
|
|
tag TEXT NOT NULL -- git tag or branch name
|
|
title TEXT
|
|
state TEXT -- "pending" | "indexed" | "error"
|
|
indexedAt DATETIME
|
|
}
|
|
```
|
|
|
|
### 6.3 Documents (parsed files)
|
|
```
|
|
Document {
|
|
id TEXT PRIMARY KEY
|
|
repositoryId TEXT FK → Repository
|
|
versionId TEXT FK → RepositoryVersion (nullable = default branch)
|
|
filePath TEXT NOT NULL
|
|
title TEXT
|
|
content TEXT NOT NULL -- raw markdown/code
|
|
language TEXT -- programming language if code file
|
|
tokenCount INTEGER
|
|
checksum TEXT -- SHA-256 for change detection
|
|
indexedAt DATETIME
|
|
}
|
|
```
|
|
|
|
### 6.4 Snippets (indexed chunks)
|
|
```
|
|
Snippet {
|
|
id TEXT PRIMARY KEY
|
|
documentId TEXT FK → Document
|
|
repositoryId TEXT FK → Repository
|
|
type TEXT NOT NULL -- "code" | "info"
|
|
title TEXT
|
|
content TEXT NOT NULL -- the actual searchable text/code
|
|
language TEXT
|
|
breadcrumb TEXT -- heading hierarchy path
|
|
tokenCount INTEGER
|
|
embedding BLOB -- float32[] stored as blob
|
|
createdAt DATETIME
|
|
}
|
|
```
|
|
|
|
### 6.5 Indexing Jobs
|
|
```
|
|
IndexingJob {
|
|
id TEXT PRIMARY KEY
|
|
repositoryId TEXT FK → Repository
|
|
versionId TEXT
|
|
status TEXT -- "queued" | "running" | "done" | "failed"
|
|
progress INTEGER DEFAULT 0 -- 0-100
|
|
totalFiles INTEGER
|
|
processedFiles INTEGER
|
|
error TEXT
|
|
startedAt DATETIME
|
|
completedAt DATETIME
|
|
createdAt DATETIME
|
|
}
|
|
```
|
|
|
|
### 6.6 Repository Configuration (`trueref.json`)
|
|
```
|
|
RepositoryConfig {
|
|
repositoryId TEXT FK → Repository
|
|
projectTitle TEXT
|
|
description TEXT
|
|
folders TEXT[] -- include paths
|
|
excludeFolders TEXT[]
|
|
excludeFiles TEXT[]
|
|
rules TEXT[] -- best practices for LLMs
|
|
previousVersions { tag, title }[]
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 7. Core Features
|
|
|
|
### F1: Repository Management
|
|
Add, remove, update, and list repositories. Support GitHub (public/private via token) and local filesystem sources. Trigger indexing on demand or on schedule.
|
|
|
|
### F2: GitHub Crawler
|
|
Fetch repository file trees via GitHub Trees API. Download file contents. Respect `trueref.json` include/exclude rules. Support rate limiting and incremental re-indexing (checksum-based).
|
|
|
|
### F3: Local Filesystem Crawler
|
|
Walk directory trees. Apply include/exclude rules from `trueref.json`. Watch for file changes (optional).
|
|
|
|
### F4: Document Parser & Chunker
|
|
- Parse Markdown files into sections (heading-based splitting).
|
|
- Extract code blocks from Markdown.
|
|
- Parse standalone code files into function/class-level chunks.
|
|
- Calculate token counts.
|
|
- Produce structured `Snippet` records (type: "code" or "info").
|
|
|
|
### F5: Embedding & Vector Storage
|
|
- Generate embeddings for each snippet using a pluggable embeddings backend.
|
|
- Store embeddings as binary blobs in SQLite (sqlite-vec).
|
|
- Support fallback to FTS5-only search when no embedding provider is configured.
|
|
|
|
### F6: Semantic Search Engine
|
|
- Hybrid search: vector similarity + FTS5 keyword matching (BM25) with reciprocal rank fusion.
|
|
- Query-time retrieval: given `libraryId + query`, return ranked snippets.
|
|
- Library search: given `libraryName + query`, return matching repositories.
|
|
|
|
### F7: REST API (`/api/v1/*`)
|
|
- `GET /api/v1/libs/search?query=&libraryName=` — search libraries (context7-compatible)
|
|
- `GET /api/v1/context?query=&libraryId=&type=json|txt` — fetch documentation
|
|
- `GET /api/v1/libs` — list all indexed libraries
|
|
- `POST /api/v1/libs` — add a new repository
|
|
- `DELETE /api/v1/libs/:id` — remove a repository
|
|
- `POST /api/v1/libs/:id/index` — trigger re-indexing
|
|
- `GET /api/v1/jobs/:id` — get indexing job status
|
|
|
|
### F8: MCP Server
|
|
- Tool: `resolve-library-id` — search for libraries by name
|
|
- Tool: `query-docs` — fetch documentation by libraryId + query
|
|
- Transport: stdio (primary), HTTP (optional)
|
|
- Compatible with Claude Code, Cursor, and other MCP-aware tools
|
|
|
|
### F9: Web UI — Repository Dashboard
|
|
- List all repositories with status, snippet count, last indexed date
|
|
- Add/remove repositories (GitHub URL or local path)
|
|
- Trigger re-indexing
|
|
- View indexing job progress
|
|
- View repository config (`trueref.json`)
|
|
|
|
### F10: Web UI — Search Explorer
|
|
- Interactive search interface (resolve library → query docs)
|
|
- Preview snippets with syntax highlighting
|
|
- View raw document content
|
|
|
|
### F11: `trueref.json` Config Support
|
|
- Parse `trueref.json` from repo root (or `context7.json` for compatibility)
|
|
- Apply `folders`, `excludeFolders`, `excludeFiles` during crawling
|
|
- Inject `rules` into LLM context alongside snippets
|
|
- Support `previousVersions` for versioned documentation
|
|
|
|
### F12: Indexing Pipeline & Job Queue
|
|
- SQLite-backed job queue (no external message broker required)
|
|
- Sequential processing with progress tracking
|
|
- Error recovery and retry logic
|
|
- Incremental re-indexing using file checksums
|
|
|
|
### F13: Version Support
|
|
- Index specific git tags/branches per repository
|
|
- Serve version-specific context when libraryId includes version (`/owner/repo/v1.2.3`)
|
|
- UI for managing available versions
|
|
|
|
---
|
|
|
|
## 8. API Compatibility with context7
|
|
|
|
TrueRef's REST API mirrors context7's `/api/v2/*` interface to allow drop-in compatibility:
|
|
|
|
| context7 Endpoint | TrueRef Endpoint | Notes |
|
|
|-------------------|-----------------|-------|
|
|
| `GET /api/v2/libs/search` | `GET /api/v1/libs/search` | Same query params |
|
|
| `GET /api/v2/context` | `GET /api/v1/context` | Same query params, same response shape |
|
|
|
|
The MCP tool names and input schemas are identical:
|
|
- `resolve-library-id` with `libraryName` + `query`
|
|
- `query-docs` with `libraryId` + `query`
|
|
|
|
Library IDs follow the same convention: `/owner/repo` or `/owner/repo/version`.
|
|
|
|
---
|
|
|
|
## 9. Non-Functional Requirements
|
|
|
|
### Performance
|
|
- Library search: < 200ms p99
|
|
- Documentation retrieval: < 500ms p99 for 20 snippets
|
|
- Indexing throughput: > 1,000 files/minute (GitHub API rate-limited)
|
|
|
|
### Reliability
|
|
- Failed indexing jobs must not corrupt existing indexed data
|
|
- Atomic snippet replacement during re-indexing
|
|
|
|
### Portability
|
|
- Single SQLite file for all data
|
|
- Runs on Linux, macOS, Windows (Node.js 20+)
|
|
- No required external services beyond optional embedding API
|
|
|
|
### Scalability (v1 constraints)
|
|
- Designed for single-node deployment
|
|
- SQLite suitable for up to ~500 repositories, ~500k snippets
|
|
|
|
---
|
|
|
|
## 10. Milestones & Feature Order
|
|
|
|
| ID | Feature | Priority | Depends On |
|
|
|----|---------|----------|-----------|
|
|
| TRUEREF-0001 | Database schema & core data models | P0 | — |
|
|
| TRUEREF-0002 | Repository management service & REST API | P0 | TRUEREF-0001 |
|
|
| TRUEREF-0003 | GitHub repository crawler | P0 | TRUEREF-0001 |
|
|
| TRUEREF-0004 | Local filesystem crawler | P1 | TRUEREF-0001 |
|
|
| TRUEREF-0005 | Document parser & chunker | P0 | TRUEREF-0001 |
|
|
| TRUEREF-0006 | SQLite FTS5 full-text search | P0 | TRUEREF-0005 |
|
|
| TRUEREF-0007 | Embedding generation & vector storage | P1 | TRUEREF-0005 |
|
|
| TRUEREF-0008 | Hybrid semantic search engine | P1 | TRUEREF-0006, TRUEREF-0007 |
|
|
| TRUEREF-0009 | Indexing pipeline & job queue | P0 | TRUEREF-0003, TRUEREF-0005 |
|
|
| TRUEREF-0010 | REST API (search + context endpoints) | P0 | TRUEREF-0006, TRUEREF-0009 |
|
|
| TRUEREF-0011 | MCP server (stdio transport) | P0 | TRUEREF-0010 |
|
|
| TRUEREF-0012 | MCP server (HTTP transport) | P1 | TRUEREF-0011 |
|
|
| TRUEREF-0013 | `trueref.json` config file support | P0 | TRUEREF-0003 |
|
|
| TRUEREF-0014 | Repository version management | P1 | TRUEREF-0003 |
|
|
| TRUEREF-0015 | Web UI — repository dashboard | P1 | TRUEREF-0002, TRUEREF-0009 |
|
|
| TRUEREF-0016 | Web UI — search explorer | P2 | TRUEREF-0010, TRUEREF-0015 |
|
|
| TRUEREF-0017 | Incremental re-indexing (checksum diff) | P1 | TRUEREF-0009 |
|
|
| TRUEREF-0018 | Embedding provider configuration UI | P2 | TRUEREF-0007, TRUEREF-0015 |
|
|
|
|
---
|
|
|
|
## 11. Open Questions
|
|
|
|
1. **Embedding provider default**: Should we bundle a local model (transformers.js) or require external API configuration? Local model provides zero-config experience but adds ~100MB to install size.
|
|
2. **Streaming responses**: Should `query-docs` support streaming for long responses?
|
|
3. **GitHub private repos**: Should we store the GitHub token in the DB or require it per-request?
|
|
4. **context7.json backward compatibility**: Should we auto-detect and use `context7.json` if no `trueref.json` is present?
|
|
5. **Re-indexing strategy**: Automatic periodic re-indexing (cron) vs. manual-only?
|