chore(FEEDBACK-0001): linting

This commit is contained in:
Giancarmine Salucci
2026-03-27 02:23:01 +01:00
parent 16436bfab2
commit 5a3c27224d
102 changed files with 5108 additions and 4976 deletions

View File

@@ -602,6 +602,7 @@ alwaysApply: true
---
When answering questions about indexed libraries, always use the TrueRef MCP tools:
1. Call `resolve-library-id` with the library name and the user's question to get the library ID.
2. Call `query-docs` with the library ID and question to retrieve relevant documentation.
3. Use the returned documentation to answer accurately.
@@ -639,7 +640,7 @@ docker compose up --build
This builds the image and starts two services:
| Service | Default port | Purpose |
|---------|-------------|---------|
| ------- | ------------ | ----------------------------- |
| `web` | `3000` | SvelteKit web UI and REST API |
| `mcp` | `3001` | MCP HTTP server |
@@ -687,10 +688,10 @@ services:
- ${USERPROFILE:-$HOME}/.gitconfig:/root/.gitconfig:ro
- ${CORP_CA_CERT}:/certs/corp-ca.crt:ro
environment:
BITBUCKET_HOST: "${BITBUCKET_HOST}"
GITLAB_HOST: "${GITLAB_HOST}"
GIT_TOKEN_BITBUCKET: "${GIT_TOKEN_BITBUCKET}"
GIT_TOKEN_GITLAB: "${GIT_TOKEN_GITLAB}"
BITBUCKET_HOST: '${BITBUCKET_HOST}'
GITLAB_HOST: '${GITLAB_HOST}'
GIT_TOKEN_BITBUCKET: '${GIT_TOKEN_BITBUCKET}'
GIT_TOKEN_GITLAB: '${GIT_TOKEN_GITLAB}'
```
5. **Start the services**:
@@ -708,6 +709,7 @@ The Docker entrypoint script (`docker-entrypoint.sh`) runs these steps in order:
3. **Configure git credentials**: Sets up per-host credential helpers that provide the correct username and token for each remote.
This setup works for:
- HTTPS cloning with personal access tokens
- SSH cloning with mounted SSH keys
- On-premise servers with custom CA certificates
@@ -718,6 +720,7 @@ This setup works for:
For long-lived deployments, SSH authentication is recommended:
1. Generate an SSH key pair if you don't have one:
```sh
ssh-keygen -t ed25519 -C "trueref@your-company.com"
```
@@ -727,6 +730,7 @@ For long-lived deployments, SSH authentication is recommended:
- GitLab: User Settings → SSH Keys
3. Ensure your `~/.ssh/config` has the correct host entries:
```
Host bitbucket.corp.example.com
IdentityFile ~/.ssh/id_ed25519
@@ -738,7 +742,7 @@ For long-lived deployments, SSH authentication is recommended:
### Environment variables
| Variable | Default | Description |
|----------|---------|-------------|
| ----------------- | ----------------------- | -------------------------------------------------- |
| `DATABASE_URL` | `/data/trueref.db` | Path to the SQLite database inside the container |
| `PORT` | `3000` | Port the web app listens on |
| `HOST` | `0.0.0.0` | Bind address for the web app |
@@ -807,7 +811,7 @@ Verify the connection inside Claude Code:
### Health checks
| Endpoint | Expected response |
|----------|------------------|
| ----------------------------------- | ------------------------------- |
| `http://localhost:3000/api/v1/libs` | JSON array of indexed libraries |
| `http://localhost:3001/ping` | `{"ok":true}` |

View File

@@ -2,7 +2,7 @@ services:
web:
build: .
ports:
- "3000:3000"
- '3000:3000'
volumes:
- trueref-data:/data
# Corporate deployment support (TRUEREF-0019)
@@ -24,10 +24,10 @@ services:
build: .
command: mcp
ports:
- "3001:3001"
- '3001:3001'
environment:
TRUEREF_API_URL: http://web:3000
MCP_PORT: "3001"
MCP_PORT: '3001'
depends_on:
- web
restart: unless-stopped

View File

@@ -17,6 +17,7 @@ The core use case is enabling AI coding assistants (Claude Code, Cursor, Zed, et
## 2. Problem Statement
### 2.1 Context7's Limitations
- The indexing and crawling backend is entirely private and closed-source.
- Only public libraries already in the context7.com catalog are available.
- Private, internal, or niche repositories cannot be added.
@@ -24,6 +25,7 @@ The core use case is enabling AI coding assistants (Claude Code, Cursor, Zed, et
- No way to self-host for air-gapped or compliance-constrained environments.
### 2.2 The Gap
Teams with internal SDKs, private libraries, proprietary documentation, or a need for data sovereignty have no tooling that provides context7-equivalent LLM documentation retrieval.
---
@@ -31,6 +33,7 @@ Teams with internal SDKs, private libraries, proprietary documentation, or a nee
## 3. Goals & Non-Goals
### Goals
- Replicate all context7 capabilities: library search, documentation retrieval, MCP tools (`resolve-library-id`, `query-docs`).
- Support both GitHub-hosted and local filesystem repositories.
- Provide a full indexing pipeline: crawl → parse → chunk → embed → store → query.
@@ -42,6 +45,7 @@ Teams with internal SDKs, private libraries, proprietary documentation, or a nee
- Self-hostable with minimal dependencies (SQLite-first, no external vector DB required).
### Non-Goals (v1)
- Authentication & authorization (deferred to a future version).
- Skill generation (context7 CLI skill feature).
- Multi-tenant SaaS mode.
@@ -54,9 +58,11 @@ Teams with internal SDKs, private libraries, proprietary documentation, or a nee
## 4. Users & Personas
### Primary: The Developer / Tech Lead
Configures TrueRef, adds repositories, integrates the MCP server with their AI coding assistant. Technical, comfortable with CLI and config files.
### Secondary: The AI Coding Assistant
The "user" at query time. Calls `resolve-library-id` and `query-docs` via MCP to retrieve documentation snippets for code generation.
---
@@ -100,8 +106,9 @@ The "user" at query time. Calls `resolve-library-id` and `query-docs` via MCP to
```
### Technology Stack
| Layer | Technology |
|-------|-----------|
| ---------------- | ------------------------------------------------------------------ |
| Framework | SvelteKit (Node adapter) |
| Language | TypeScript |
| Database | SQLite via better-sqlite3 + drizzle-orm |
@@ -119,6 +126,7 @@ The "user" at query time. Calls `resolve-library-id` and `query-docs` via MCP to
## 6. Data Model
### 6.1 Repositories
A `Repository` is the top-level entity. It maps to a GitHub repo or local directory.
```
@@ -141,6 +149,7 @@ Repository {
```
### 6.2 Repository Versions
```
RepositoryVersion {
id TEXT PRIMARY KEY
@@ -153,6 +162,7 @@ RepositoryVersion {
```
### 6.3 Documents (parsed files)
```
Document {
id TEXT PRIMARY KEY
@@ -169,6 +179,7 @@ Document {
```
### 6.4 Snippets (indexed chunks)
```
Snippet {
id TEXT PRIMARY KEY
@@ -186,6 +197,7 @@ Snippet {
```
### 6.5 Indexing Jobs
```
IndexingJob {
id TEXT PRIMARY KEY
@@ -203,6 +215,7 @@ IndexingJob {
```
### 6.6 Repository Configuration (`trueref.json`)
```
RepositoryConfig {
repositoryId TEXT FK → Repository
@@ -221,15 +234,19 @@ RepositoryConfig {
## 7. Core Features
### F1: Repository Management
Add, remove, update, and list repositories. Support GitHub (public/private via token) and local filesystem sources. Trigger indexing on demand or on schedule.
### F2: GitHub Crawler
Fetch repository file trees via GitHub Trees API. Download file contents. Respect `trueref.json` include/exclude rules. Support rate limiting and incremental re-indexing (checksum-based).
### F3: Local Filesystem Crawler
Walk directory trees. Apply include/exclude rules from `trueref.json`. Watch for file changes (optional).
### F4: Document Parser & Chunker
- Parse Markdown files into sections (heading-based splitting).
- Extract code blocks from Markdown.
- Parse standalone code files into function/class-level chunks.
@@ -237,16 +254,19 @@ Walk directory trees. Apply include/exclude rules from `trueref.json`. Watch for
- Produce structured `Snippet` records (type: "code" or "info").
### F5: Embedding & Vector Storage
- Generate embeddings for each snippet using a pluggable embeddings backend.
- Store embeddings as binary blobs in SQLite (sqlite-vec).
- Support fallback to FTS5-only search when no embedding provider is configured.
### F6: Semantic Search Engine
- Hybrid search: vector similarity + FTS5 keyword matching (BM25) with reciprocal rank fusion.
- Query-time retrieval: given `libraryId + query`, return ranked snippets.
- Library search: given `libraryName + query`, return matching repositories.
### F7: REST API (`/api/v1/*`)
- `GET /api/v1/libs/search?query=&libraryName=` — search libraries (context7-compatible)
- `GET /api/v1/context?query=&libraryId=&type=json|txt` — fetch documentation
- `GET /api/v1/libs` — list all indexed libraries
@@ -256,12 +276,14 @@ Walk directory trees. Apply include/exclude rules from `trueref.json`. Watch for
- `GET /api/v1/jobs/:id` — get indexing job status
### F8: MCP Server
- Tool: `resolve-library-id` — search for libraries by name
- Tool: `query-docs` — fetch documentation by libraryId + query
- Transport: stdio (primary), HTTP (optional)
- Compatible with Claude Code, Cursor, and other MCP-aware tools
### F9: Web UI — Repository Dashboard
- List all repositories with status, snippet count, last indexed date
- Add/remove repositories (GitHub URL or local path)
- Trigger re-indexing
@@ -269,23 +291,27 @@ Walk directory trees. Apply include/exclude rules from `trueref.json`. Watch for
- View repository config (`trueref.json`)
### F10: Web UI — Search Explorer
- Interactive search interface (resolve library → query docs)
- Preview snippets with syntax highlighting
- View raw document content
### F11: `trueref.json` Config Support
- Parse `trueref.json` from repo root (or `context7.json` for compatibility)
- Apply `folders`, `excludeFolders`, `excludeFiles` during crawling
- Inject `rules` into LLM context alongside snippets
- Support `previousVersions` for versioned documentation
### F12: Indexing Pipeline & Job Queue
- SQLite-backed job queue (no external message broker required)
- Sequential processing with progress tracking
- Error recovery and retry logic
- Incremental re-indexing using file checksums
### F13: Version Support
- Index specific git tags/branches per repository
- Serve version-specific context when libraryId includes version (`/owner/repo/v1.2.3`)
- UI for managing available versions
@@ -297,11 +323,12 @@ Walk directory trees. Apply include/exclude rules from `trueref.json`. Watch for
TrueRef's REST API mirrors context7's `/api/v2/*` interface to allow drop-in compatibility:
| context7 Endpoint | TrueRef Endpoint | Notes |
|-------------------|-----------------|-------|
| ------------------------- | ------------------------- | -------------------------------------- |
| `GET /api/v2/libs/search` | `GET /api/v1/libs/search` | Same query params |
| `GET /api/v2/context` | `GET /api/v1/context` | Same query params, same response shape |
The MCP tool names and input schemas are identical:
- `resolve-library-id` with `libraryName` + `query`
- `query-docs` with `libraryId` + `query`
@@ -312,20 +339,24 @@ Library IDs follow the same convention: `/owner/repo` or `/owner/repo/version`.
## 9. Non-Functional Requirements
### Performance
- Library search: < 200ms p99
- Documentation retrieval: < 500ms p99 for 20 snippets
- Indexing throughput: > 1,000 files/minute (GitHub API rate-limited)
### Reliability
- Failed indexing jobs must not corrupt existing indexed data
- Atomic snippet replacement during re-indexing
### Portability
- Single SQLite file for all data
- Runs on Linux, macOS, Windows (Node.js 20+)
- No required external services beyond optional embedding API
### Scalability (v1 constraints)
- Designed for single-node deployment
- SQLite suitable for up to ~500 repositories, ~500k snippets
@@ -334,7 +365,7 @@ Library IDs follow the same convention: `/owner/repo` or `/owner/repo/version`.
## 10. Milestones & Feature Order
| ID | Feature | Priority | Depends On |
|----|---------|----------|-----------|
| ------------ | ---------------------------------------- | -------- | -------------------------- |
| TRUEREF-0001 | Database schema & core data models | P0 | — |
| TRUEREF-0002 | Repository management service & REST API | P0 | TRUEREF-0001 |
| TRUEREF-0003 | GitHub repository crawler | P0 | TRUEREF-0001 |

View File

@@ -39,7 +39,9 @@ export const repositories = sqliteTable('repositories', {
branch: text('branch').default('main'),
state: text('state', {
enum: ['pending', 'indexing', 'indexed', 'error']
}).notNull().default('pending'),
})
.notNull()
.default('pending'),
totalSnippets: integer('total_snippets').default(0),
totalTokens: integer('total_tokens').default(0),
trustScore: real('trust_score').default(0), // 0.010.0
@@ -48,7 +50,7 @@ export const repositories = sqliteTable('repositories', {
githubToken: text('github_token'), // encrypted PAT for private repos
lastIndexedAt: integer('last_indexed_at', { mode: 'timestamp' }),
createdAt: integer('created_at', { mode: 'timestamp' }).notNull(),
updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull(),
updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull()
});
```
@@ -59,16 +61,19 @@ Tracks indexed git tags/branches beyond the default branch.
```typescript
export const repositoryVersions = sqliteTable('repository_versions', {
id: text('id').primaryKey(), // e.g. "/facebook/react/v18.3.0"
repositoryId: text('repository_id').notNull()
repositoryId: text('repository_id')
.notNull()
.references(() => repositories.id, { onDelete: 'cascade' }),
tag: text('tag').notNull(), // git tag or branch name
title: text('title'),
state: text('state', {
enum: ['pending', 'indexing', 'indexed', 'error']
}).notNull().default('pending'),
})
.notNull()
.default('pending'),
totalSnippets: integer('total_snippets').default(0),
indexedAt: integer('indexed_at', { mode: 'timestamp' }),
createdAt: integer('created_at', { mode: 'timestamp' }).notNull(),
createdAt: integer('created_at', { mode: 'timestamp' }).notNull()
});
```
@@ -79,16 +84,16 @@ A parsed source file within a repository.
```typescript
export const documents = sqliteTable('documents', {
id: text('id').primaryKey(), // UUID
repositoryId: text('repository_id').notNull()
repositoryId: text('repository_id')
.notNull()
.references(() => repositories.id, { onDelete: 'cascade' }),
versionId: text('version_id')
.references(() => repositoryVersions.id, { onDelete: 'cascade' }),
versionId: text('version_id').references(() => repositoryVersions.id, { onDelete: 'cascade' }),
filePath: text('file_path').notNull(), // relative path within repo
title: text('title'),
language: text('language'), // e.g. "typescript", "markdown"
tokenCount: integer('token_count').default(0),
checksum: text('checksum').notNull(), // SHA-256 of file content
indexedAt: integer('indexed_at', { mode: 'timestamp' }).notNull(),
indexedAt: integer('indexed_at', { mode: 'timestamp' }).notNull()
});
```
@@ -99,19 +104,20 @@ An indexed chunk of content, the atomic unit of search.
```typescript
export const snippets = sqliteTable('snippets', {
id: text('id').primaryKey(), // UUID
documentId: text('document_id').notNull()
documentId: text('document_id')
.notNull()
.references(() => documents.id, { onDelete: 'cascade' }),
repositoryId: text('repository_id').notNull()
repositoryId: text('repository_id')
.notNull()
.references(() => repositories.id, { onDelete: 'cascade' }),
versionId: text('version_id')
.references(() => repositoryVersions.id, { onDelete: 'cascade' }),
versionId: text('version_id').references(() => repositoryVersions.id, { onDelete: 'cascade' }),
type: text('type', { enum: ['code', 'info'] }).notNull(),
title: text('title'),
content: text('content').notNull(), // searchable text / code
language: text('language'),
breadcrumb: text('breadcrumb'), // e.g. "Installation > Getting Started"
tokenCount: integer('token_count').default(0),
createdAt: integer('created_at', { mode: 'timestamp' }).notNull(),
createdAt: integer('created_at', { mode: 'timestamp' }).notNull()
});
```
@@ -121,12 +127,13 @@ Stores vector embeddings separately to keep snippets table lean.
```typescript
export const snippetEmbeddings = sqliteTable('snippet_embeddings', {
snippetId: text('snippet_id').primaryKey()
snippetId: text('snippet_id')
.primaryKey()
.references(() => snippets.id, { onDelete: 'cascade' }),
model: text('model').notNull(), // embedding model identifier
dimensions: integer('dimensions').notNull(),
embedding: blob('embedding').notNull(), // Float32Array as binary blob
createdAt: integer('created_at', { mode: 'timestamp' }).notNull(),
createdAt: integer('created_at', { mode: 'timestamp' }).notNull()
});
```
@@ -137,19 +144,22 @@ Tracks asynchronous indexing operations.
```typescript
export const indexingJobs = sqliteTable('indexing_jobs', {
id: text('id').primaryKey(), // UUID
repositoryId: text('repository_id').notNull()
repositoryId: text('repository_id')
.notNull()
.references(() => repositories.id, { onDelete: 'cascade' }),
versionId: text('version_id'),
status: text('status', {
enum: ['queued', 'running', 'done', 'failed']
}).notNull().default('queued'),
})
.notNull()
.default('queued'),
progress: integer('progress').default(0), // 0100
totalFiles: integer('total_files').default(0),
processedFiles: integer('processed_files').default(0),
error: text('error'),
startedAt: integer('started_at', { mode: 'timestamp' }),
completedAt: integer('completed_at', { mode: 'timestamp' }),
createdAt: integer('created_at', { mode: 'timestamp' }).notNull(),
createdAt: integer('created_at', { mode: 'timestamp' }).notNull()
});
```
@@ -159,7 +169,8 @@ Stores parsed `trueref.json` / `context7.json` configuration.
```typescript
export const repositoryConfigs = sqliteTable('repository_configs', {
repositoryId: text('repository_id').primaryKey()
repositoryId: text('repository_id')
.primaryKey()
.references(() => repositories.id, { onDelete: 'cascade' }),
projectTitle: text('project_title'),
description: text('description'),
@@ -167,9 +178,10 @@ export const repositoryConfigs = sqliteTable('repository_configs', {
excludeFolders: text('exclude_folders', { mode: 'json' }).$type<string[]>(),
excludeFiles: text('exclude_files', { mode: 'json' }).$type<string[]>(),
rules: text('rules', { mode: 'json' }).$type<string[]>(),
previousVersions: text('previous_versions', { mode: 'json' })
.$type<{ tag: string; title: string }[]>(),
updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull(),
previousVersions: text('previous_versions', { mode: 'json' }).$type<
{ tag: string; title: string }[]
>(),
updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull()
});
```
@@ -181,7 +193,7 @@ Key-value store for global application settings.
export const settings = sqliteTable('settings', {
key: text('key').primaryKey(),
value: text('value', { mode: 'json' }),
updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull(),
updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull()
});
```

View File

@@ -31,10 +31,12 @@ Implement the core `RepositoryService` that handles CRUD operations for reposito
## Repository ID Generation
GitHub repositories:
- Input URL: `https://github.com/facebook/react` or `github.com/facebook/react`
- Generated ID: `/facebook/react`
Local repositories:
- Input path: `/home/user/projects/my-sdk`
- Generated ID: `/local/my-sdk` (basename of path, slugified)
- Collision resolution: append `-2`, `-3`, etc.
@@ -71,22 +73,22 @@ export class RepositoryService {
state?: Repository['state'];
limit?: number;
offset?: number;
}): Promise<Repository[]>
}): Promise<Repository[]>;
async get(id: string): Promise<Repository | null>
async get(id: string): Promise<Repository | null>;
async add(input: AddRepositoryInput): Promise<Repository>
async add(input: AddRepositoryInput): Promise<Repository>;
async update(id: string, input: UpdateRepositoryInput): Promise<Repository>
async update(id: string, input: UpdateRepositoryInput): Promise<Repository>;
async remove(id: string): Promise<void>
async remove(id: string): Promise<void>;
async getStats(id: string): Promise<{
totalSnippets: number;
totalTokens: number;
totalDocuments: number;
lastIndexedAt: Date | null;
}>
}>;
}
```
@@ -97,11 +99,13 @@ export class RepositoryService {
### `GET /api/v1/libs`
Query parameters:
- `state` (optional): filter by state (`pending`, `indexed`, `error`, etc.)
- `limit` (optional, default 50): max results
- `offset` (optional, default 0): pagination offset
Response `200`:
```json
{
"libraries": [
@@ -128,6 +132,7 @@ Response `200`:
### `POST /api/v1/libs`
Request body:
```json
{
"source": "github",
@@ -139,6 +144,7 @@ Request body:
```
Response `201`:
```json
{
"library": { ...Repository },
@@ -149,6 +155,7 @@ Response `201`:
`autoIndex: true` (default) immediately queues an indexing job.
Response `409` if repository already exists:
```json
{ "error": "Repository /facebook/react already exists" }
```
@@ -176,11 +183,13 @@ Response `404`: not found.
Triggers a new indexing job. If a job is already running for this repo, returns the existing job.
Request body (optional):
```json
{ "version": "v18.3.0" }
```
Response `202`:
```json
{
"job": {
@@ -198,6 +207,7 @@ Response `202`:
## Error Response Shape
All error responses follow:
```json
{
"error": "Human-readable message",
@@ -207,6 +217,7 @@ All error responses follow:
```
Error codes:
- `NOT_FOUND`
- `ALREADY_EXISTS`
- `INVALID_INPUT`

View File

@@ -38,16 +38,45 @@ The crawler only downloads files with these extensions:
```typescript
const INDEXABLE_EXTENSIONS = new Set([
// Documentation
'.md', '.mdx', '.txt', '.rst',
'.md',
'.mdx',
'.txt',
'.rst',
// Code
'.ts', '.tsx', '.js', '.jsx',
'.py', '.rb', '.go', '.rs', '.java', '.cs', '.cpp', '.c', '.h',
'.swift', '.kt', '.php', '.scala', '.clj', '.ex', '.exs',
'.sh', '.bash', '.zsh', '.fish',
'.ts',
'.tsx',
'.js',
'.jsx',
'.py',
'.rb',
'.go',
'.rs',
'.java',
'.cs',
'.cpp',
'.c',
'.h',
'.swift',
'.kt',
'.php',
'.scala',
'.clj',
'.ex',
'.exs',
'.sh',
'.bash',
'.zsh',
'.fish',
// Config / data
'.json', '.yaml', '.yml', '.toml',
'.json',
'.yaml',
'.yml',
'.toml',
// Web
'.html', '.css', '.svelte', '.vue',
'.html',
'.css',
'.svelte',
'.vue'
]);
const MAX_FILE_SIZE_BYTES = 500_000; // 500 KB — skip large generated files
@@ -89,12 +118,14 @@ export interface CrawlOptions {
## GitHub API Usage
### Step 1: Get default branch (if ref not specified)
```
GET https://api.github.com/repos/{owner}/{repo}
→ { default_branch: "main", stargazers_count: 12345 }
```
### Step 2: Fetch file tree (recursive)
```
GET https://api.github.com/repos/{owner}/{repo}/git/trees/{ref}?recursive=1
→ {
@@ -109,12 +140,14 @@ GET https://api.github.com/repos/{owner}/{repo}/git/trees/{ref}?recursive=1
If `truncated: true`, the tree has >100k items. Use `--depth` pagination or filter top-level directories first.
### Step 3: Download file contents (parallel)
```
GET https://api.github.com/repos/{owner}/{repo}/contents/{path}?ref={ref}
→ { content: "<base64>", encoding: "base64", size: 1234, sha: "abc123" }
```
Alternative for large repos: use raw content URL:
```
GET https://raw.githubusercontent.com/{owner}/{repo}/{ref}/{path}
```
@@ -124,11 +157,7 @@ GET https://raw.githubusercontent.com/{owner}/{repo}/{ref}/{path}
## Filtering Logic
```typescript
function shouldIndexFile(
filePath: string,
fileSize: number,
config?: RepoConfig
): boolean {
function shouldIndexFile(filePath: string, fileSize: number, config?: RepoConfig): boolean {
const ext = path.extname(filePath).toLowerCase();
const base = path.basename(filePath);
@@ -153,14 +182,17 @@ function shouldIndexFile(
if (isInIgnoredDirectory(filePath)) return false;
// 7. Apply config excludeFolders (regex or prefix match)
if (config?.excludeFolders?.some(folder =>
filePath.startsWith(folder) || new RegExp(folder).test(filePath)
)) return false;
if (
config?.excludeFolders?.some(
(folder) => filePath.startsWith(folder) || new RegExp(folder).test(filePath)
)
)
return false;
// 8. Apply config folders allowlist (if specified, only index those paths)
if (config?.folders?.length) {
const inAllowedFolder = config.folders.some(folder =>
filePath.startsWith(folder) || new RegExp(folder).test(filePath)
const inAllowedFolder = config.folders.some(
(folder) => filePath.startsWith(folder) || new RegExp(folder).test(filePath)
);
if (!inAllowedFolder) return false;
}
@@ -201,7 +233,7 @@ Requests are made with a concurrency limit of 10 parallel downloads using a sema
## Error Handling
| Scenario | Behavior |
|----------|---------|
| ------------------------- | --------------------------------------------------------------------------- |
| 404 Not Found | Throw `RepositoryNotFoundError` |
| 401 Unauthorized | Throw `AuthenticationError` (invalid or missing token) |
| 403 Forbidden | If `X-RateLimit-Remaining: 0`, wait and retry; else throw `PermissionError` |

View File

@@ -58,18 +58,14 @@ export class LocalCrawler {
const allFiles = await this.walkDirectory(options.rootPath, '', gitignore);
// 3. Look for trueref.json / context7.json first
const configFile = allFiles.find(f =>
f === 'trueref.json' || f === 'context7.json'
);
const configFile = allFiles.find((f) => f === 'trueref.json' || f === 'context7.json');
let config = options.config;
if (configFile && !config) {
config = await this.parseConfigFile(
path.join(options.rootPath, configFile)
);
config = await this.parseConfigFile(path.join(options.rootPath, configFile));
}
// 4. Filter files
const filteredFiles = allFiles.filter(relPath => {
const filteredFiles = allFiles.filter((relPath) => {
const stat = statCache.get(relPath);
return shouldIndexFile(relPath, stat.size, config);
});
@@ -85,7 +81,7 @@ export class LocalCrawler {
content,
size: Buffer.byteLength(content, 'utf-8'),
sha,
language: detectLanguage(relPath),
language: detectLanguage(relPath)
});
options.onProgress?.(i + 1, filteredFiles.length);
}
@@ -95,11 +91,15 @@ export class LocalCrawler {
totalFiles: filteredFiles.length,
skippedFiles: allFiles.length - filteredFiles.length,
branch: 'local',
commitSha: computeSHA256(crawledFiles.map(f => f.sha).join('')),
commitSha: computeSHA256(crawledFiles.map((f) => f.sha).join(''))
};
}
private async walkDirectory(dir: string, rel = '', gitignore?: GitignoreFilter): Promise<string[]> {
private async walkDirectory(
dir: string,
rel = '',
gitignore?: GitignoreFilter
): Promise<string[]> {
const entries = await fs.readdir(dir, { withFileTypes: true });
const files: string[] = [];
for (const entry of entries) {
@@ -109,9 +109,7 @@ export class LocalCrawler {
if (shouldPruneDirectory(relPath) || gitignore?.isIgnored(relPath, true)) {
continue;
}
files.push(...await this.walkDirectory(
path.join(dir, entry.name), relPath, gitignore
));
files.push(...(await this.walkDirectory(path.join(dir, entry.name), relPath, gitignore)));
} else {
if (gitignore?.isIgnored(relPath, false)) continue;
files.push(relPath);

View File

@@ -31,7 +31,7 @@ Implement the document parsing and chunking pipeline that transforms raw file co
## Supported File Types
| Extension | Parser Strategy |
|-----------|----------------|
| --------------------------------- | ------------------------------------------------------- |
| `.md`, `.mdx` | Heading-based section splitting + code block extraction |
| `.txt`, `.rst` | Paragraph-based splitting |
| `.ts`, `.tsx`, `.js`, `.jsx` | AST-free: function/class boundary detection via regex |
@@ -96,7 +96,7 @@ function parseMarkdown(content: string, filePath: string): Snippet[] {
title,
content: chunk,
breadcrumb,
tokenCount: estimateTokens(chunk),
tokenCount: estimateTokens(chunk)
});
}
}
@@ -110,7 +110,7 @@ function parseMarkdown(content: string, filePath: string): Snippet[] {
content: block.code,
language: block.language || detectLanguage('.' + block.language),
breadcrumb,
tokenCount: estimateTokens(block.code),
tokenCount: estimateTokens(block.code)
});
}
}
@@ -139,14 +139,10 @@ const BOUNDARY_PATTERNS: Record<string, RegExp> = {
python: /^(async\s+)?(def|class)\s+\w+/m,
go: /^(func|type|var|const)\s+\w+/m,
rust: /^(pub\s+)?(fn|impl|struct|enum|trait)\s+\w+/m,
java: /^(public|private|protected|static).*?(class|interface|enum|void|\w+)\s+\w+\s*[({]/m,
java: /^(public|private|protected|static).*?(class|interface|enum|void|\w+)\s+\w+\s*[({]/m
};
function parseCodeFile(
content: string,
filePath: string,
language: string
): Snippet[] {
function parseCodeFile(content: string, filePath: string, language: string): Snippet[] {
const pattern = BOUNDARY_PATTERNS[language];
const breadcrumb = filePath;
const title = path.basename(filePath);
@@ -158,17 +154,19 @@ function parseCodeFile(
const chunks = splitAtBoundaries(content, pattern);
return chunks
.filter(chunk => chunk.trim().length >= 20)
.flatMap(chunk => {
.filter((chunk) => chunk.trim().length >= 20)
.flatMap((chunk) => {
if (estimateTokens(chunk) <= MAX_TOKENS) {
return [{
return [
{
type: 'code' as const,
title,
content: chunk,
language,
breadcrumb,
tokenCount: estimateTokens(chunk),
}];
tokenCount: estimateTokens(chunk)
}
];
}
return slidingWindowChunks(chunk, filePath, language);
});
@@ -188,11 +186,7 @@ const MIN_CONTENT_LENGTH = 20; // characters
### Sliding Window Chunker
```typescript
function chunkText(
text: string,
maxTokens: number,
overlapTokens: number
): string[] {
function chunkText(text: string, maxTokens: number, overlapTokens: number): string[] {
const words = text.split(/\s+/);
const wordsPerToken = 0.75; // ~0.75 words per token
const maxWords = Math.floor(maxTokens * wordsPerToken);
@@ -218,29 +212,37 @@ function chunkText(
```typescript
const LANGUAGE_MAP: Record<string, string> = {
'.ts': 'typescript', '.tsx': 'typescript',
'.js': 'javascript', '.jsx': 'javascript',
'.ts': 'typescript',
'.tsx': 'typescript',
'.js': 'javascript',
'.jsx': 'javascript',
'.py': 'python',
'.rb': 'ruby',
'.go': 'go',
'.rs': 'rust',
'.java': 'java',
'.cs': 'csharp',
'.cpp': 'cpp', '.c': 'c', '.h': 'c',
'.cpp': 'cpp',
'.c': 'c',
'.h': 'c',
'.swift': 'swift',
'.kt': 'kotlin',
'.php': 'php',
'.scala': 'scala',
'.sh': 'bash', '.bash': 'bash', '.zsh': 'bash',
'.md': 'markdown', '.mdx': 'markdown',
'.sh': 'bash',
'.bash': 'bash',
'.zsh': 'bash',
'.md': 'markdown',
'.mdx': 'markdown',
'.json': 'json',
'.yaml': 'yaml', '.yml': 'yaml',
'.yaml': 'yaml',
'.yml': 'yaml',
'.toml': 'toml',
'.html': 'html',
'.css': 'css',
'.svelte': 'svelte',
'.vue': 'vue',
'.sql': 'sql',
'.sql': 'sql'
};
function detectLanguage(filePath: string): string {
@@ -260,12 +262,12 @@ export interface ParseOptions {
versionId?: string;
}
export function parseFile(
file: CrawledFile,
options: ParseOptions
): NewSnippet[] {
export function parseFile(file: CrawledFile, options: ParseOptions): NewSnippet[] {
const language = detectLanguage(file.path);
let rawSnippets: Omit<NewSnippet, 'id' | 'repositoryId' | 'documentId' | 'versionId' | 'createdAt'>[];
let rawSnippets: Omit<
NewSnippet,
'id' | 'repositoryId' | 'documentId' | 'versionId' | 'createdAt'
>[];
if (language === 'markdown') {
rawSnippets = parseMarkdown(file.content, file.path);
@@ -273,13 +275,13 @@ export function parseFile(
rawSnippets = parseCodeFile(file.content, file.path, language);
}
return rawSnippets.map(s => ({
return rawSnippets.map((s) => ({
...s,
id: crypto.randomUUID(),
repositoryId: options.repositoryId,
documentId: options.documentId,
versionId: options.versionId ?? null,
createdAt: new Date(),
createdAt: new Date()
}));
}
```

View File

@@ -61,14 +61,9 @@ export interface LibrarySearchResult {
export class SearchService {
constructor(private db: BetterSQLite3.Database) {}
searchSnippets(
query: string,
options: SnippetSearchOptions
): SnippetSearchResult[]
searchSnippets(query: string, options: SnippetSearchOptions): SnippetSearchResult[];
searchRepositories(
options: LibrarySearchOptions
): LibrarySearchResult[]
searchRepositories(options: LibrarySearchOptions): LibrarySearchResult[];
}
```
@@ -174,52 +169,61 @@ searchRepositories(options: LibrarySearchOptions): LibrarySearchResult[] {
The search results must be formatted for the REST API and MCP tool responses:
### Library search response (for `resolve-library-id`):
```typescript
function formatLibraryResults(results: LibrarySearchResult[]): string {
if (results.length === 0) {
return 'No libraries found matching your search.';
}
return results.map((r, i) => {
return results
.map((r, i) => {
const repo = r.repository;
const versions = r.versions.map(v => v.tag).join(', ') || 'default branch';
const versions = r.versions.map((v) => v.tag).join(', ') || 'default branch';
return [
`${i + 1}. ${repo.title}`,
` Library ID: ${repo.id}`,
` Description: ${repo.description ?? 'No description'}`,
` Snippets: ${repo.totalSnippets} | Trust Score: ${repo.trustScore.toFixed(1)}/10`,
` Available Versions: ${versions}`,
` Available Versions: ${versions}`
].join('\n');
}).join('\n\n');
})
.join('\n\n');
}
```
### Snippet search response (for `query-docs`):
```typescript
function formatSnippetResults(
results: SnippetSearchResult[],
rules?: string[]
): string {
function formatSnippetResults(results: SnippetSearchResult[], rules?: string[]): string {
const parts: string[] = [];
// Prepend repository rules if present
if (rules?.length) {
parts.push('## Library Rules\n' + rules.map(r => `- ${r}`).join('\n'));
parts.push('## Library Rules\n' + rules.map((r) => `- ${r}`).join('\n'));
}
for (const { snippet } of results) {
if (snippet.type === 'code') {
parts.push([
parts.push(
[
snippet.title ? `### ${snippet.title}` : '',
snippet.breadcrumb ? `*${snippet.breadcrumb}*` : '',
`\`\`\`${snippet.language ?? ''}\n${snippet.content}\n\`\`\``,
].filter(Boolean).join('\n'));
`\`\`\`${snippet.language ?? ''}\n${snippet.content}\n\`\`\``
]
.filter(Boolean)
.join('\n')
);
} else {
parts.push([
parts.push(
[
snippet.title ? `### ${snippet.title}` : '',
snippet.breadcrumb ? `*${snippet.breadcrumb}*` : '',
snippet.content,
].filter(Boolean).join('\n'));
snippet.content
]
.filter(Boolean)
.join('\n')
);
}
}

View File

@@ -74,14 +74,14 @@ export class OpenAIEmbeddingProvider implements EmbeddingProvider {
const response = await fetch(`${this.config.baseUrl}/embeddings`, {
method: 'POST',
headers: {
'Authorization': `Bearer ${this.config.apiKey}`,
'Content-Type': 'application/json',
Authorization: `Bearer ${this.config.apiKey}`,
'Content-Type': 'application/json'
},
body: JSON.stringify({
model: this.config.model,
input: batch,
dimensions: this.config.dimensions,
}),
dimensions: this.config.dimensions
})
});
if (!response.ok) {
@@ -93,7 +93,7 @@ export class OpenAIEmbeddingProvider implements EmbeddingProvider {
allEmbeddings.push({
values: new Float32Array(item.embedding),
dimensions: item.embedding.length,
model: this.config.model,
model: this.config.model
});
}
}
@@ -126,12 +126,12 @@ export class LocalEmbeddingProvider implements EmbeddingProvider {
for (const text of texts) {
const output = await (this.pipeline as Function)(text, {
pooling: 'mean',
normalize: true,
normalize: true
});
results.push({
values: new Float32Array(output.data),
dimensions: this.dimensions,
model: this.model,
model: this.model
});
}
return results;
@@ -163,12 +163,14 @@ export class EmbeddingService {
snippetIds: string[],
onProgress?: (done: number, total: number) => void
): Promise<void> {
const snippets = this.db.prepare(
const snippets = this.db
.prepare(
`SELECT id, content, type FROM snippets WHERE id IN (${snippetIds.map(() => '?').join(',')})`
).all(...snippetIds) as Snippet[];
)
.all(...snippetIds) as Snippet[];
// Prepare text for embedding: combine title + content
const texts = snippets.map(s =>
const texts = snippets.map((s) =>
[s.title, s.breadcrumb, s.content].filter(Boolean).join('\n').slice(0, 2048)
);
@@ -227,6 +229,7 @@ export interface EmbeddingConfig {
### API Endpoints
`GET /api/v1/settings/embedding`
```json
{
"provider": "openai",
@@ -251,11 +254,7 @@ Embeddings are stored as raw `Float32Array` binary blobs:
const buffer = Buffer.from(float32Array.buffer);
// Retrieve
const float32Array = new Float32Array(
buffer.buffer,
buffer.byteOffset,
buffer.byteLength / 4
);
const float32Array = new Float32Array(buffer.buffer, buffer.byteOffset, buffer.byteLength / 4);
```
---

View File

@@ -137,13 +137,10 @@ export class HybridSearchService {
constructor(
private db: BetterSQLite3.Database,
private searchService: SearchService,
private embeddingProvider: EmbeddingProvider | null,
private embeddingProvider: EmbeddingProvider | null
) {}
async search(
query: string,
options: HybridSearchOptions
): Promise<SnippetSearchResult[]> {
async search(query: string, options: HybridSearchOptions): Promise<SnippetSearchResult[]> {
const limit = options.limit ?? 20;
const alpha = options.alpha ?? 0.5;
@@ -152,7 +149,7 @@ export class HybridSearchService {
repositoryId: options.repositoryId,
versionId: options.versionId,
type: options.type,
limit: limit * 3, // get more candidates for fusion
limit: limit * 3 // get more candidates for fusion
});
// If no embedding provider or alpha = 0, return FTS5 results directly
@@ -171,18 +168,18 @@ export class HybridSearchService {
// Normalize result lists for RRF
const ftsRanked = ftsResults.map((r, i) => ({
id: r.snippet.id,
score: i,
score: i
}));
const vecRanked = vectorResults.map((r, i) => ({
id: r.snippetId,
score: i,
score: i
}));
// Apply RRF
const fused = reciprocalRankFusion(ftsRanked, vecRanked);
// Fetch full snippet data for top results
const topIds = fused.slice(0, limit).map(r => r.id);
const topIds = fused.slice(0, limit).map((r) => r.id);
return this.fetchSnippetsByIds(topIds, options.repositoryId);
}
}

View File

@@ -72,12 +72,16 @@ export class JobQueue {
error: null,
startedAt: null,
completedAt: null,
createdAt: new Date(),
createdAt: new Date()
};
this.db.prepare(`
this.db
.prepare(
`
INSERT INTO indexing_jobs VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`).run(...Object.values(job));
`
)
.run(...Object.values(job));
// Kick off processing if not already running
if (!this.isRunning) {
@@ -90,12 +94,16 @@ export class JobQueue {
private async processNext(): Promise<void> {
if (this.isRunning) return;
const job = this.db.prepare(`
const job = this.db
.prepare(
`
SELECT * FROM indexing_jobs
WHERE status = 'queued'
ORDER BY created_at ASC
LIMIT 1
`).get() as IndexingJob | undefined;
`
)
.get() as IndexingJob | undefined;
if (!job) return;
@@ -105,17 +113,17 @@ export class JobQueue {
} finally {
this.isRunning = false;
// Check for next queued job
const nextJob = this.db.prepare(
`SELECT id FROM indexing_jobs WHERE status = 'queued' LIMIT 1`
).get();
const nextJob = this.db
.prepare(`SELECT id FROM indexing_jobs WHERE status = 'queued' LIMIT 1`)
.get();
if (nextJob) setImmediate(() => this.processNext());
}
}
getJob(id: string): IndexingJob | null {
return this.db.prepare(
`SELECT * FROM indexing_jobs WHERE id = ?`
).get(id) as IndexingJob | null;
return this.db
.prepare(`SELECT * FROM indexing_jobs WHERE id = ?`)
.get(id) as IndexingJob | null;
}
listJobs(repositoryId?: string, limit = 20): IndexingJob[] {
@@ -140,7 +148,7 @@ export class IndexingPipeline {
private db: BetterSQLite3.Database,
private githubCrawler: GitHubCrawler,
private localCrawler: LocalCrawler,
private embeddingService: EmbeddingService | null,
private embeddingService: EmbeddingService | null
) {}
async run(job: IndexingJob): Promise<void> {
@@ -157,8 +165,11 @@ export class IndexingPipeline {
const crawlResult = await this.crawl(repo, job);
// Step 2: Parse and diff
const { newSnippets, changedDocIds, newDocuments } =
await this.parseAndDiff(crawlResult, repo, job);
const { newSnippets, changedDocIds, newDocuments } = await this.parseAndDiff(
crawlResult,
repo,
job
);
// Step 3: Atomic replacement
this.replaceSnippets(repo.id, changedDocIds, newDocuments, newSnippets);
@@ -166,7 +177,7 @@ export class IndexingPipeline {
// Step 4: Embeddings (async, non-blocking for job completion)
if (this.embeddingService && newSnippets.length > 0) {
await this.embeddingService.embedSnippets(
newSnippets.map(s => s.id),
newSnippets.map((s) => s.id),
(done, total) => {
// Update job progress for embedding phase
}
@@ -180,20 +191,19 @@ export class IndexingPipeline {
totalSnippets: stats.totalSnippets,
totalTokens: stats.totalTokens,
trustScore: computeTrustScore({ ...repo, ...stats }),
lastIndexedAt: new Date(),
lastIndexedAt: new Date()
});
this.updateJob(job.id, {
status: 'done',
progress: 100,
completedAt: new Date(),
completedAt: new Date()
});
} catch (error) {
this.updateJob(job.id, {
status: 'failed',
error: (error as Error).message,
completedAt: new Date(),
completedAt: new Date()
});
this.updateRepo(job.repositoryId, { state: 'error' });
throw error;
@@ -210,9 +220,9 @@ export class IndexingPipeline {
this.db.transaction(() => {
if (changedDocIds.length > 0) {
// Cascade deletes snippets via FK constraint
this.db.prepare(
`DELETE FROM documents WHERE id IN (${changedDocIds.map(() => '?').join(',')})`
).run(...changedDocIds);
this.db
.prepare(`DELETE FROM documents WHERE id IN (${changedDocIds.map(() => '?').join(',')})`)
.run(...changedDocIds);
}
for (const doc of newDocuments) {
@@ -248,9 +258,7 @@ function calculateProgress(
// Crawl+parse = 80%, embeddings = 20%
const parseProgress = (processedFiles / totalFiles) * 80;
const embedProgress = embeddingsTotal > 0
? (embeddingsDone / embeddingsTotal) * 20
: 0;
const embedProgress = embeddingsTotal > 0 ? (embeddingsDone / embeddingsTotal) * 20 : 0;
return Math.round(parseProgress + embedProgress);
}
@@ -263,6 +271,7 @@ function calculateProgress(
### `GET /api/v1/jobs/:id`
Response `200`:
```json
{
"job": {
@@ -285,6 +294,7 @@ Response `200`:
Query params: `repositoryId` (optional), `status` (optional), `limit` (default 20).
Response `200`:
```json
{
"jobs": [...],
@@ -300,20 +310,24 @@ On application start, mark any jobs in `running` state as `failed` (they were in
```typescript
function recoverStaleJobs(db: BetterSQLite3.Database): void {
db.prepare(`
db.prepare(
`
UPDATE indexing_jobs
SET status = 'failed',
error = 'Server restarted while job was running',
completed_at = unixepoch()
WHERE status = 'running'
`).run();
`
).run();
// Also reset any repositories stuck in 'indexing' state
db.prepare(`
db.prepare(
`
UPDATE repositories
SET state = 'error'
WHERE state = 'indexing'
`).run();
`
).run();
}
```

View File

@@ -33,7 +33,7 @@ Implement the public-facing REST API endpoints that replicate context7's `/api/v
### Query Parameters
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| ------------- | ------- | -------- | ------------------------------------- |
| `libraryName` | string | Yes | Library name to search for |
| `query` | string | No | User's question for relevance ranking |
| `limit` | integer | No | Max results (default: 10, max: 50) |
@@ -67,7 +67,7 @@ Note: `state: "finalized"` maps from TrueRef's `state: "indexed"` for compatibil
### State Mapping
| TrueRef state | context7 state |
|---------------|---------------|
| ------------- | -------------- |
| `pending` | `initial` |
| `indexing` | `initial` |
| `indexed` | `finalized` |
@@ -82,7 +82,7 @@ Note: `state: "finalized"` maps from TrueRef's `state: "indexed"` for compatibil
### Query Parameters
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| ----------- | ------- | -------- | --------------------------------------------------------------- |
| `libraryId` | string | Yes | Library ID, e.g. `/facebook/react` or `/facebook/react/v18.3.0` |
| `query` | string | Yes | Specific question about the library |
| `type` | string | No | `json` (default) or `txt` (plain text for LLM injection) |
@@ -125,7 +125,7 @@ Note: `state: "finalized"` maps from TrueRef's `state: "indexed"` for compatibil
Plain text formatted for direct LLM context injection:
```
````
## Library Rules
- Always use functional components
- Use hooks for state management
@@ -139,15 +139,17 @@ Plain text formatted for direct LLM context injection:
function MyComponent() {
return <div>Hello</div>;
}
```
````
---
### React components let you split the UI...
*Core Concepts > Components*
_Core Concepts > Components_
React components let you split the UI into independent, reusable pieces...
```
````
---
@@ -167,7 +169,7 @@ function parseLibraryId(libraryId: string): {
version: match[3],
};
}
```
````
---
@@ -176,10 +178,7 @@ function parseLibraryId(libraryId: string): {
The `tokens` parameter limits the total response size. Snippets are added greedily until the budget is exhausted:
```typescript
function selectSnippetsWithinBudget(
snippets: Snippet[],
maxTokens: number
): Snippet[] {
function selectSnippetsWithinBudget(snippets: Snippet[], maxTokens: number): Snippet[] {
const selected: Snippet[] = [];
let usedTokens = 0;
@@ -215,6 +214,7 @@ Default token budget: 10,000 tokens (~7,500 words) — enough for ~20 medium sni
## CORS Configuration
All API routes include:
```
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PATCH, DELETE, OPTIONS

View File

@@ -46,10 +46,7 @@ Implement a Model Context Protocol (MCP) server that exposes `resolve-library-id
import { Server } from '@modelcontextprotocol/sdk/server/index.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import {
CallToolRequestSchema,
ListToolsRequestSchema,
} from '@modelcontextprotocol/sdk/types.js';
import { CallToolRequestSchema, ListToolsRequestSchema } from '@modelcontextprotocol/sdk/types.js';
import { z } from 'zod';
const API_BASE = process.env.TRUEREF_API_URL ?? 'http://localhost:5173';
@@ -57,33 +54,29 @@ const API_BASE = process.env.TRUEREF_API_URL ?? 'http://localhost:5173';
const server = new Server(
{
name: 'io.github.trueref/trueref',
version: '1.0.0',
version: '1.0.0'
},
{
capabilities: { tools: {} },
capabilities: { tools: {} }
}
);
// Tool schemas — identical to context7 for drop-in compatibility
const ResolveLibraryIdSchema = z.object({
libraryName: z.string().describe(
'Library name to search for and resolve to a TrueRef library ID'
),
query: z.string().describe(
"The user's question or context to help rank results"
),
libraryName: z
.string()
.describe('Library name to search for and resolve to a TrueRef library ID'),
query: z.string().describe("The user's question or context to help rank results")
});
const QueryDocsSchema = z.object({
libraryId: z.string().describe(
'The TrueRef library ID obtained from resolve-library-id, e.g. /facebook/react'
),
query: z.string().describe(
'Specific question about the library to retrieve relevant documentation'
),
tokens: z.number().optional().describe(
'Maximum token budget for the response (default: 10000)'
),
libraryId: z
.string()
.describe('The TrueRef library ID obtained from resolve-library-id, e.g. /facebook/react'),
query: z
.string()
.describe('Specific question about the library to retrieve relevant documentation'),
tokens: z.number().optional().describe('Maximum token budget for the response (default: 10000)')
});
server.setRequestHandler(ListToolsRequestSchema, async () => ({
@@ -94,22 +87,22 @@ server.setRequestHandler(ListToolsRequestSchema, async () => ({
'Searches TrueRef to find a library matching the given name.',
'Returns a list of matching libraries with their IDs.',
'ALWAYS call this tool before query-docs to get the correct library ID.',
'Call at most 3 times per user question.',
'Call at most 3 times per user question.'
].join(' '),
inputSchema: {
type: 'object',
properties: {
libraryName: {
type: 'string',
description: 'Library name to search for',
description: 'Library name to search for'
},
query: {
type: 'string',
description: "User's question for relevance ranking",
},
},
required: ['libraryName', 'query'],
description: "User's question for relevance ranking"
}
},
required: ['libraryName', 'query']
}
},
{
name: 'query-docs',
@@ -117,28 +110,28 @@ server.setRequestHandler(ListToolsRequestSchema, async () => ({
'Fetches documentation and code examples from TrueRef for a specific library.',
'Requires a library ID obtained from resolve-library-id.',
'Returns relevant snippets formatted for LLM consumption.',
'Call at most 3 times per user question.',
'Call at most 3 times per user question.'
].join(' '),
inputSchema: {
type: 'object',
properties: {
libraryId: {
type: 'string',
description: 'TrueRef library ID, e.g. /facebook/react',
description: 'TrueRef library ID, e.g. /facebook/react'
},
query: {
type: 'string',
description: 'Specific question about the library',
description: 'Specific question about the library'
},
tokens: {
type: 'number',
description: 'Max token budget (default: 10000)',
description: 'Max token budget (default: 10000)'
}
},
},
required: ['libraryId', 'query'],
},
},
],
required: ['libraryId', 'query']
}
}
]
}));
server.setRequestHandler(CallToolRequestSchema, async (request) => {
@@ -155,17 +148,19 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
const response = await fetch(url.toString());
if (!response.ok) {
return {
content: [{
content: [
{
type: 'text',
text: `Error searching libraries: ${response.status} ${response.statusText}`,
}],
isError: true,
text: `Error searching libraries: ${response.status} ${response.statusText}`
}
],
isError: true
};
}
const text = await response.text();
return {
content: [{ type: 'text', text }],
content: [{ type: 'text', text }]
};
}
@@ -183,40 +178,46 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
const status = response.status;
if (status === 404) {
return {
content: [{
content: [
{
type: 'text',
text: `Library "${libraryId}" not found. Please run resolve-library-id first.`,
}],
isError: true,
text: `Library "${libraryId}" not found. Please run resolve-library-id first.`
}
],
isError: true
};
}
if (status === 503) {
return {
content: [{
content: [
{
type: 'text',
text: `Library "${libraryId}" is currently being indexed. Please try again in a moment.`,
}],
isError: true,
text: `Library "${libraryId}" is currently being indexed. Please try again in a moment.`
}
],
isError: true
};
}
return {
content: [{
content: [
{
type: 'text',
text: `Error fetching documentation: ${response.status} ${response.statusText}`,
}],
isError: true,
text: `Error fetching documentation: ${response.status} ${response.statusText}`
}
],
isError: true
};
}
const text = await response.text();
return {
content: [{ type: 'text', text }],
content: [{ type: 'text', text }]
};
}
return {
content: [{ type: 'text', text: `Unknown tool: ${name}` }],
isError: true,
isError: true
};
});
@@ -245,6 +246,7 @@ main().catch((err) => {
```
Or with `tsx` for TypeScript-direct execution:
```json
{
"scripts": {
@@ -274,6 +276,7 @@ Users add to `.mcp.json`:
```
Or with tsx for development:
```json
{
"mcpServers": {
@@ -295,13 +298,15 @@ Or with tsx for development:
The MCP server should include a `resources` list item (optional) or the library responses themselves prepend rules. Additionally, users should add a Claude rule file:
```markdown
<!-- .claude/rules/trueref.md -->
---
## <!-- .claude/rules/trueref.md -->
description: Use TrueRef to retrieve documentation for indexed libraries
alwaysApply: true
---
When answering questions about indexed libraries, always use the TrueRef MCP tools:
1. Call `resolve-library-id` with the library name and the user's question to get the library ID
2. Call `query-docs` with the library ID and question to retrieve relevant documentation
3. Use the returned documentation to answer the question accurately

View File

@@ -52,8 +52,8 @@ import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js'
const { values: args } = parseArgs({
options: {
transport: { type: 'string', default: 'stdio' },
port: { type: 'string', default: process.env.PORT ?? '3001' },
},
port: { type: 'string', default: process.env.PORT ?? '3001' }
}
});
async function startHttp(server: Server, port: number): Promise<void> {
@@ -81,7 +81,7 @@ async function startHttp(server: Server, port: number): Promise<void> {
}
const transport = new StreamableHTTPServerTransport({
sessionIdGenerator: () => crypto.randomUUID(),
sessionIdGenerator: () => crypto.randomUUID()
});
await server.connect(transport);

View File

@@ -95,7 +95,7 @@ const CONFIG_CONSTRAINTS = {
excludeFiles: { maxItems: 100, maxLength: 200 },
rules: { maxItems: 20, minLength: 5, maxLength: 500 },
previousVersions: { maxItems: 50 },
versionTag: { pattern: /^v?\d+\.\d+(\.\d+)?(-.*)?$/ },
versionTag: { pattern: /^v?\d+\.\d+(\.\d+)?(-.*)?$/ }
};
```
@@ -172,7 +172,7 @@ export function parseConfigFile(content: string, filename: string): ParsedConfig
if (Array.isArray(config.rules)) {
validated.rules = (config.rules as unknown[])
.filter((r): r is string => typeof r === 'string' && r.length >= 5)
.map(r => r.slice(0, 500))
.map((r) => r.slice(0, 500))
.slice(0, 20);
}
}
@@ -181,8 +181,10 @@ export function parseConfigFile(content: string, filename: string): ParsedConfig
if (config.previousVersions !== undefined) {
if (Array.isArray(config.previousVersions)) {
validated.previousVersions = (config.previousVersions as unknown[])
.filter((v): v is { tag: string; title: string } =>
typeof v === 'object' && v !== null &&
.filter(
(v): v is { tag: string; title: string } =>
typeof v === 'object' &&
v !== null &&
typeof (v as Record<string, unknown>).tag === 'string' &&
typeof (v as Record<string, unknown>).title === 'string'
)
@@ -193,7 +195,7 @@ export function parseConfigFile(content: string, filename: string): ParsedConfig
return {
config: validated,
source: filename.startsWith('trueref') ? 'trueref.json' : 'context7.json',
warnings,
warnings
};
}
```
@@ -219,17 +221,11 @@ When `query-docs` returns results, `rules` from `repository_configs` are prepend
```typescript
// In formatters.ts
function buildContextResponse(
snippets: Snippet[],
config: RepositoryConfig | null
): string {
function buildContextResponse(snippets: Snippet[], config: RepositoryConfig | null): string {
const parts: string[] = [];
if (config?.rules?.length) {
parts.push(
'## Library Best Practices\n' +
config.rules.map(r => `- ${r}`).join('\n')
);
parts.push('## Library Best Practices\n' + config.rules.map((r) => `- ${r}`).join('\n'));
}
// ... append snippet content

View File

@@ -45,6 +45,7 @@ Examples:
### `GET /api/v1/libs/:id/versions`
Response `200`:
```json
{
"versions": [
@@ -64,6 +65,7 @@ Response `200`:
### `POST /api/v1/libs/:id/versions`
Request body:
```json
{
"tag": "v18.3.0",
@@ -73,6 +75,7 @@ Request body:
```
Response `201`:
```json
{
"version": { ...RepositoryVersion },
@@ -101,15 +104,14 @@ async function listGitHubTags(
token?: string
): Promise<Array<{ name: string; commit: { sha: string } }>> {
const headers: Record<string, string> = {
'Accept': 'application/vnd.github.v3+json',
'User-Agent': 'TrueRef/1.0',
Accept: 'application/vnd.github.v3+json',
'User-Agent': 'TrueRef/1.0'
};
if (token) headers['Authorization'] = `Bearer ${token}`;
const response = await fetch(
`https://api.github.com/repos/${owner}/${repo}/tags?per_page=100`,
{ headers }
);
const response = await fetch(`https://api.github.com/repos/${owner}/${repo}/tags?per_page=100`, {
headers
});
if (!response.ok) throw new GitHubApiError(response.status);
return response.json();
@@ -135,14 +137,12 @@ function resolveSearchTarget(libraryId: string): {
}
// Look up versionId from tag
const versionRecord = db.prepare(
`SELECT id FROM repository_versions WHERE repository_id = ? AND tag = ?`
).get(repositoryId, version) as { id: string } | undefined;
const versionRecord = db
.prepare(`SELECT id FROM repository_versions WHERE repository_id = ? AND tag = ?`)
.get(repositoryId, version) as { id: string } | undefined;
if (!versionRecord) {
throw new NotFoundError(
`Version "${version}" not found for library "${repositoryId}"`
);
throw new NotFoundError(`Version "${version}" not found for library "${repositoryId}"`);
}
return { repositoryId, versionId: versionRecord.id };
@@ -159,18 +159,18 @@ Snippets with `version_id IS NULL` belong to the default branch; snippets with a
export class VersionService {
constructor(private db: BetterSQLite3.Database) {}
list(repositoryId: string): RepositoryVersion[]
list(repositoryId: string): RepositoryVersion[];
add(repositoryId: string, tag: string, title?: string): RepositoryVersion
add(repositoryId: string, tag: string, title?: string): RepositoryVersion;
remove(repositoryId: string, tag: string): void
remove(repositoryId: string, tag: string): void;
getByTag(repositoryId: string, tag: string): RepositoryVersion | null
getByTag(repositoryId: string, tag: string): RepositoryVersion | null;
registerFromConfig(
repositoryId: string,
previousVersions: { tag: string; title: string }[]
): RepositoryVersion[]
): RepositoryVersion[];
}
```

View File

@@ -61,14 +61,14 @@ Implement the main web interface for managing repositories. Built with SvelteKit
pending: 'bg-gray-100 text-gray-600',
indexing: 'bg-blue-100 text-blue-700',
indexed: 'bg-green-100 text-green-700',
error: 'bg-red-100 text-red-700',
error: 'bg-red-100 text-red-700'
};
const stateLabels = {
pending: 'Pending',
indexing: 'Indexing...',
indexed: 'Indexed',
error: 'Error',
error: 'Error'
};
</script>
@@ -150,7 +150,7 @@ Implement the main web interface for managing repositories. Built with SvelteKit
const res = await fetch('/api/v1/libs', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ source, sourceUrl, githubToken: githubToken || undefined }),
body: JSON.stringify({ source, sourceUrl, githubToken: githubToken || undefined })
});
if (!res.ok) {
const data = await res.json();
@@ -172,13 +172,17 @@ Implement the main web interface for managing repositories. Built with SvelteKit
<div class="mb-4 flex gap-2">
<button
class="flex-1 rounded-lg py-2 text-sm {source === 'github' ? 'bg-blue-600 text-white' : 'border border-gray-200 text-gray-700'}"
onclick={() => source = 'github'}
>GitHub</button>
class="flex-1 rounded-lg py-2 text-sm {source === 'github'
? 'bg-blue-600 text-white'
: 'border border-gray-200 text-gray-700'}"
onclick={() => (source = 'github')}>GitHub</button
>
<button
class="flex-1 rounded-lg py-2 text-sm {source === 'local' ? 'bg-blue-600 text-white' : 'border border-gray-200 text-gray-700'}"
onclick={() => source = 'local'}
>Local Path</button>
class="flex-1 rounded-lg py-2 text-sm {source === 'local'
? 'bg-blue-600 text-white'
: 'border border-gray-200 text-gray-700'}"
onclick={() => (source = 'local')}>Local Path</button
>
</div>
<label class="block">
@@ -197,7 +201,9 @@ Implement the main web interface for managing repositories. Built with SvelteKit
{#if source === 'github'}
<label class="mt-3 block">
<span class="text-sm font-medium text-gray-700">GitHub Token (optional, for private repos)</span>
<span class="text-sm font-medium text-gray-700"
>GitHub Token (optional, for private repos)</span
>
<input
type="password"
bind:value={githubToken}

View File

@@ -58,14 +58,20 @@ An interactive search interface within the web UI that lets users test the docum
<!-- src/lib/components/search/LibraryResult.svelte -->
<script lang="ts">
let { result, onSelect } = $props<{
result: { id: string; title: string; description: string; totalSnippets: number; trustScore: number };
result: {
id: string;
title: string;
description: string;
totalSnippets: number;
trustScore: number;
};
onSelect: (id: string) => void;
}>();
</script>
<button
onclick={() => onSelect(result.id)}
class="w-full rounded-xl border border-gray-200 bg-white p-4 text-left shadow-sm hover:border-blue-300 hover:shadow-md transition-all"
class="w-full rounded-xl border border-gray-200 bg-white p-4 text-left shadow-sm transition-all hover:border-blue-300 hover:shadow-md"
>
<div class="flex items-center justify-between">
<span class="font-semibold text-gray-900">{result.title}</span>
@@ -73,7 +79,7 @@ An interactive search interface within the web UI that lets users test the docum
</div>
<p class="font-mono text-xs text-gray-400">{result.id}</p>
{#if result.description}
<p class="mt-1.5 text-sm text-gray-600 line-clamp-2">{result.description}</p>
<p class="mt-1.5 line-clamp-2 text-sm text-gray-600">{result.description}</p>
{/if}
<p class="mt-2 text-xs text-gray-400">{result.totalSnippets.toLocaleString()} snippets</p>
</button>
@@ -91,7 +97,7 @@ An interactive search interface within the web UI that lets users test the docum
let { snippet } = $props<{ snippet: Snippet }>();
</script>
<div class="rounded-xl border border-gray-200 bg-white overflow-hidden">
<div class="overflow-hidden rounded-xl border border-gray-200 bg-white">
<div class="flex items-center justify-between border-b border-gray-100 px-4 py-2.5">
<div class="flex items-center gap-2">
{#if snippet.type === 'code'}
@@ -112,7 +118,9 @@ An interactive search interface within the web UI that lets users test the docum
<div class="p-4">
{#if snippet.type === 'code'}
<pre class="overflow-x-auto rounded bg-gray-950 p-4 text-sm text-gray-100"><code>{snippet.content}</code></pre>
<pre class="overflow-x-auto rounded bg-gray-950 p-4 text-sm text-gray-100"><code
>{snippet.content}</code
></pre>
{:else}
<div class="prose prose-sm max-w-none text-gray-700">{snippet.content}</div>
{/if}
@@ -162,7 +170,7 @@ An interactive search interface within the web UI that lets users test the docum
// Update URL
goto(`/search?lib=${encodeURIComponent(selectedLibraryId)}&q=${encodeURIComponent(query)}`, {
replaceState: true,
keepFocus: true,
keepFocus: true
});
}
</script>

View File

@@ -40,8 +40,8 @@ function computeDiff(
crawledFiles: CrawledFile[],
existingDocs: Document[] // documents currently in DB for this repo
): FileDiff {
const existingMap = new Map(existingDocs.map(d => [d.filePath, d]));
const crawledMap = new Map(crawledFiles.map(f => [f.path, f]));
const existingMap = new Map(existingDocs.map((d) => [d.filePath, d]));
const crawledMap = new Map(crawledFiles.map((f) => [f.path, f]));
const added: CrawledFile[] = [];
const modified: CrawledFile[] = [];
@@ -59,8 +59,8 @@ function computeDiff(
}
const deleted = existingDocs
.filter(doc => !crawledMap.has(doc.filePath))
.map(doc => doc.filePath);
.filter((doc) => !crawledMap.has(doc.filePath))
.map((doc) => doc.filePath);
return { added, modified, deleted, unchanged };
}
@@ -78,7 +78,7 @@ const diff = computeDiff(crawledResult.files, existingDocs);
// Log diff summary
this.updateJob(job.id, {
totalFiles: crawledResult.files.length,
totalFiles: crawledResult.files.length
});
// Process only changed/new files
@@ -89,13 +89,13 @@ const docIdsToDelete: string[] = [];
// Map modified files to their existing document IDs for deletion
for (const file of diff.modified) {
const existing = existingDocs.find(d => d.filePath === file.path);
const existing = existingDocs.find((d) => d.filePath === file.path);
if (existing) docIdsToDelete.push(existing.id);
}
// Map deleted file paths to document IDs
for (const filePath of diff.deleted) {
const existing = existingDocs.find(d => d.filePath === filePath);
const existing = existingDocs.find((d) => d.filePath === filePath);
if (existing) docIdsToDelete.push(existing.id);
}
@@ -110,7 +110,7 @@ for (const [i, file] of filesToProcess.entries()) {
const progress = Math.round((totalProcessed / crawledResult.files.length) * 80);
this.updateJob(job.id, {
processedFiles: totalProcessed,
progress,
progress
});
}
@@ -123,6 +123,7 @@ this.replaceSnippets(repo.id, docIdsToDelete, newDocuments, newSnippets);
## Performance Impact
For a typical repository with 1,000 files where 50 changed:
- **Without incremental**: 1,000 files parsed + 1,000 embed batches
- **With incremental**: 50 files parsed + 50 embed batches
- Estimated speedup: ~20x for re-indexing

View File

@@ -36,20 +36,20 @@ const PROVIDER_PRESETS = [
name: 'OpenAI',
baseUrl: 'https://api.openai.com/v1',
model: 'text-embedding-3-small',
dimensions: 1536,
dimensions: 1536
},
{
name: 'Ollama (local)',
baseUrl: 'http://localhost:11434/v1',
model: 'nomic-embed-text',
dimensions: 768,
dimensions: 768
},
{
name: 'Azure OpenAI',
baseUrl: 'https://{resource}.openai.azure.com/openai/deployments/{deployment}/v1',
model: 'text-embedding-3-small',
dimensions: 1536,
},
dimensions: 1536
}
];
```
@@ -76,7 +76,7 @@ const PROVIDER_PRESETS = [
const res = await fetch('/api/v1/settings/embedding/test', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ provider, openai: { baseUrl, apiKey, model, dimensions } }),
body: JSON.stringify({ provider, openai: { baseUrl, apiKey, model, dimensions } })
});
if (res.ok) {
testStatus = 'ok';
@@ -96,7 +96,7 @@ const PROVIDER_PRESETS = [
await fetch('/api/v1/settings/embedding', {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ provider, openai: { baseUrl, apiKey, model, dimensions } }),
body: JSON.stringify({ provider, openai: { baseUrl, apiKey, model, dimensions } })
});
saving = false;
}
@@ -114,7 +114,7 @@ const PROVIDER_PRESETS = [
<div class="mb-4 flex gap-2">
{#each ['none', 'openai', 'local'] as p}
<button
onclick={() => provider = p}
onclick={() => (provider = p)}
class="rounded-lg px-4 py-2 text-sm {provider === p
? 'bg-blue-600 text-white'
: 'border border-gray-200 text-gray-700 hover:bg-gray-50'}"
@@ -125,7 +125,7 @@ const PROVIDER_PRESETS = [
</div>
{#if provider === 'none'}
<div class="rounded-lg bg-amber-50 border border-amber-200 p-3 text-sm text-amber-700">
<div class="rounded-lg border border-amber-200 bg-amber-50 p-3 text-sm text-amber-700">
Search will use keyword matching only. Results may be less relevant for complex questions.
</div>
{/if}
@@ -133,10 +133,14 @@ const PROVIDER_PRESETS = [
{#if provider === 'openai'}
<div class="space-y-3">
<!-- Preset buttons -->
<div class="flex gap-2 flex-wrap">
<div class="flex flex-wrap gap-2">
{#each PROVIDER_PRESETS as preset}
<button
onclick={() => { baseUrl = preset.baseUrl; model = preset.model; dimensions = preset.dimensions; }}
onclick={() => {
baseUrl = preset.baseUrl;
model = preset.model;
dimensions = preset.dimensions;
}}
class="rounded border border-gray-200 px-2.5 py-1 text-xs text-gray-600 hover:bg-gray-50"
>
{preset.name}
@@ -146,26 +150,46 @@ const PROVIDER_PRESETS = [
<label class="block">
<span class="text-sm font-medium">Base URL</span>
<input type="text" bind:value={baseUrl} class="mt-1 w-full rounded-lg border px-3 py-2 text-sm" />
<input
type="text"
bind:value={baseUrl}
class="mt-1 w-full rounded-lg border px-3 py-2 text-sm"
/>
</label>
<label class="block">
<span class="text-sm font-medium">API Key</span>
<input type="password" bind:value={apiKey} class="mt-1 w-full rounded-lg border px-3 py-2 text-sm" placeholder="sk-..." />
<input
type="password"
bind:value={apiKey}
class="mt-1 w-full rounded-lg border px-3 py-2 text-sm"
placeholder="sk-..."
/>
</label>
<label class="block">
<span class="text-sm font-medium">Model</span>
<input type="text" bind:value={model} class="mt-1 w-full rounded-lg border px-3 py-2 text-sm" />
<input
type="text"
bind:value={model}
class="mt-1 w-full rounded-lg border px-3 py-2 text-sm"
/>
</label>
<label class="block">
<span class="text-sm font-medium">Dimensions (optional override)</span>
<input type="number" bind:value={dimensions} class="mt-1 w-full rounded-lg border px-3 py-2 text-sm" />
<input
type="number"
bind:value={dimensions}
class="mt-1 w-full rounded-lg border px-3 py-2 text-sm"
/>
</label>
<div class="flex items-center gap-3">
<button onclick={testConnection} class="rounded-lg border border-gray-300 px-3 py-1.5 text-sm">
<button
onclick={testConnection}
class="rounded-lg border border-gray-300 px-3 py-1.5 text-sm"
>
{testStatus === 'testing' ? 'Testing...' : 'Test Connection'}
</button>
{#if testStatus === 'ok'}

View File

@@ -80,6 +80,7 @@ git -C /path/to/repo archive <commit-hash> | tar -x -C /tmp/trueref-idx/<repo>-<
```
Advantages over `git checkout` or worktrees:
- Working directory is completely untouched
- No `.git` directory in the output (cleaner for parsing)
- Temp directory deleted after indexing with no git state to clean up
@@ -115,7 +116,7 @@ Allow commit hashes to be pinned explicitly per version, overriding tag resoluti
### Edge Cases
| Case | Handling |
|------|----------|
| ---------------------------- | ---------------------------------------------------------------------------------- |
| Annotated tags | `rev-parse <tag>^{commit}` peels to commit automatically |
| Mutable tags (e.g. `latest`) | Re-resolve on re-index; warn in UI if hash has changed |
| Branch as version | `rev-parse origin/<branch>^{commit}` gives tip; re-resolves on re-index |
@@ -170,7 +171,7 @@ fi
Username conventions by server type:
| Server | HTTPS username | Password |
|--------|---------------|----------|
| ------------------------------ | --------------------- | --------------------- |
| Bitbucket Server / Data Center | `x-token-auth` | HTTP access token |
| Bitbucket Cloud | account username | App password |
| GitLab (self-hosted or cloud) | `oauth2` | Personal access token |
@@ -220,7 +221,7 @@ services:
web:
build: .
ports:
- "3000:3000"
- '3000:3000'
volumes:
- trueref-data:/data
- ${USERPROFILE}/.ssh:/root/.ssh:ro
@@ -228,20 +229,20 @@ services:
- ${CORP_CA_CERT}:/certs/corp-ca.crt:ro
environment:
DATABASE_URL: /data/trueref.db
GIT_TOKEN_BITBUCKET: "${BITBUCKET_TOKEN}"
GIT_TOKEN_GITLAB: "${GITLAB_TOKEN}"
BITBUCKET_HOST: "${BITBUCKET_HOST}"
GITLAB_HOST: "${GITLAB_HOST}"
GIT_TOKEN_BITBUCKET: '${BITBUCKET_TOKEN}'
GIT_TOKEN_GITLAB: '${GITLAB_TOKEN}'
BITBUCKET_HOST: '${BITBUCKET_HOST}'
GITLAB_HOST: '${GITLAB_HOST}'
restart: unless-stopped
mcp:
build: .
command: mcp
ports:
- "3001:3001"
- '3001:3001'
environment:
TRUEREF_API_URL: http://web:3000
MCP_PORT: "3001"
MCP_PORT: '3001'
depends_on:
- web
restart: unless-stopped

View File

@@ -9,7 +9,11 @@
import { initializeDatabase } from '$lib/server/db/index.js';
import { getClient } from '$lib/server/db/client.js';
import { initializePipeline } from '$lib/server/pipeline/startup.js';
import { EMBEDDING_CONFIG_KEY, createProviderFromConfig, defaultEmbeddingConfig } from '$lib/server/embeddings/factory.js';
import {
EMBEDDING_CONFIG_KEY,
createProviderFromConfig,
defaultEmbeddingConfig
} from '$lib/server/embeddings/factory.js';
import { EmbeddingService } from '$lib/server/embeddings/embedding.service.js';
import type { EmbeddingConfig } from '$lib/server/embeddings/factory.js';
import type { Handle } from '@sveltejs/kit';

View File

@@ -115,7 +115,12 @@
/>
{:else}
<div class="mt-1">
<FolderPicker bind:value={sourceUrl} onselect={(p) => { if (!title) title = p.split('/').at(-1) ?? ''; }} />
<FolderPicker
bind:value={sourceUrl}
onselect={(p) => {
if (!title) title = p.split('/').at(-1) ?? '';
}}
/>
</div>
{/if}
</div>
@@ -133,7 +138,8 @@
{#if source === 'github'}
<label class="block">
<span class="text-sm font-medium text-gray-700"
>GitHub Token <span class="font-normal text-gray-500">(optional, for private repos)</span
>GitHub Token <span class="font-normal text-gray-500"
>(optional, for private repos)</span
></span
>
<input

View File

@@ -78,9 +78,7 @@
title="Browse folders"
>
<svg class="h-4 w-4" viewBox="0 0 20 20" fill="currentColor">
<path
d="M2 6a2 2 0 012-2h5l2 2h5a2 2 0 012 2v6a2 2 0 01-2 2H4a2 2 0 01-2-2V6z"
/>
<path d="M2 6a2 2 0 012-2h5l2 2h5a2 2 0 012 2v6a2 2 0 01-2 2H4a2 2 0 01-2-2V6z" />
</svg>
Browse
</button>
@@ -94,7 +92,10 @@
class="fixed inset-0 z-[60] flex items-center justify-center bg-black/50 p-4"
onclick={handleBackdropClick}
>
<div class="flex w-full max-w-lg flex-col rounded-xl bg-white shadow-xl" style="max-height: 70vh">
<div
class="flex w-full max-w-lg flex-col rounded-xl bg-white shadow-xl"
style="max-height: 70vh"
>
<!-- Header -->
<div class="flex items-center justify-between border-b border-gray-200 px-4 py-3">
<h3 class="text-sm font-semibold text-gray-900">Select Folder</h3>
@@ -133,11 +134,7 @@
{browsePath}
</span>
{#if loading}
<svg
class="h-4 w-4 animate-spin text-gray-400"
fill="none"
viewBox="0 0 24 24"
>
<svg class="h-4 w-4 animate-spin text-gray-400" fill="none" viewBox="0 0 24 24">
<circle class="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" stroke-width="4"
></circle>
<path
@@ -166,7 +163,9 @@
title="Click to navigate, double-click to select"
>
<svg
class="h-4 w-4 shrink-0 {entry.isGitRepo ? 'text-orange-400' : 'text-yellow-400'}"
class="h-4 w-4 shrink-0 {entry.isGitRepo
? 'text-orange-400'
: 'text-yellow-400'}"
viewBox="0 0 20 20"
fill="currentColor"
>
@@ -176,7 +175,9 @@
</svg>
<span class="flex-1 truncate text-gray-800">{entry.name}</span>
{#if entry.isGitRepo}
<span class="shrink-0 rounded bg-orange-100 px-1.5 py-0.5 text-xs text-orange-700">
<span
class="shrink-0 rounded bg-orange-100 px-1.5 py-0.5 text-xs text-orange-700"
>
git
</span>
{/if}

View File

@@ -36,8 +36,9 @@
<p class="mt-0.5 truncate font-mono text-sm text-gray-500">{repo.id}</p>
</div>
<span
class="ml-3 shrink-0 rounded-full px-2.5 py-0.5 text-xs font-medium {stateColors[repo.state] ??
'bg-gray-100 text-gray-600'}"
class="ml-3 shrink-0 rounded-full px-2.5 py-0.5 text-xs font-medium {stateColors[
repo.state
] ?? 'bg-gray-100 text-gray-600'}"
>
{stateLabels[repo.state] ?? repo.state}
</span>

View File

@@ -17,7 +17,10 @@
};
</script>
<div class="flex flex-col items-center rounded-lg p-3 {variantClasses[variant] ?? variantClasses.default}">
<div
class="flex flex-col items-center rounded-lg p-3 {variantClasses[variant] ??
variantClasses.default}"
>
<span class="text-lg font-bold">{value}</span>
<span class="mt-0.5 text-xs">{label}</span>
</div>

View File

@@ -17,6 +17,8 @@
const config = $derived(statusConfig[status]);
</script>
<span class="inline-flex items-center rounded-full px-2.5 py-0.5 text-xs font-medium {config.bg} {config.text}">
<span
class="inline-flex items-center rounded-full px-2.5 py-0.5 text-xs font-medium {config.bg} {config.text}"
>
{config.label}
</span>

View File

@@ -25,7 +25,7 @@
{placeholder}
onkeydown={handleKeydown}
disabled={loading}
class="flex-1 rounded-lg border border-gray-200 bg-white px-4 py-2.5 text-sm text-gray-900 placeholder-gray-400 shadow-sm outline-none transition-all focus:border-blue-400 focus:ring-2 focus:ring-blue-100 disabled:cursor-not-allowed disabled:opacity-60"
class="flex-1 rounded-lg border border-gray-200 bg-white px-4 py-2.5 text-sm text-gray-900 placeholder-gray-400 shadow-sm transition-all outline-none focus:border-blue-400 focus:ring-2 focus:ring-blue-100 disabled:cursor-not-allowed disabled:opacity-60"
/>
<button
onclick={onsubmit}

View File

@@ -5,21 +5,15 @@
const isCode = $derived(snippet.type === 'code');
const title = $derived(
snippet.type === 'code' ? snippet.title : null
);
const title = $derived(snippet.type === 'code' ? snippet.title : null);
const breadcrumb = $derived(
snippet.type === 'code' ? snippet.description : snippet.breadcrumb
);
const breadcrumb = $derived(snippet.type === 'code' ? snippet.description : snippet.breadcrumb);
const content = $derived(
snippet.type === 'code' ? snippet.codeList[0]?.code ?? '' : snippet.text
snippet.type === 'code' ? (snippet.codeList[0]?.code ?? '') : snippet.text
);
const language = $derived(
snippet.type === 'code' ? (snippet.codeList[0]?.language ?? '') : null
);
const language = $derived(snippet.type === 'code' ? (snippet.codeList[0]?.language ?? '') : null);
const tokenCount = $derived(snippet.tokenCount ?? 0);
</script>
@@ -40,14 +34,14 @@
</div>
{#if breadcrumb}
<p class="bg-gray-50 px-4 py-1.5 text-xs italic text-gray-500">{breadcrumb}</p>
<p class="bg-gray-50 px-4 py-1.5 text-xs text-gray-500 italic">{breadcrumb}</p>
{/if}
<div class="p-4">
{#if isCode}
<pre
class="overflow-x-auto rounded bg-gray-950 p-4 text-sm text-gray-100"
><code>{content}</code></pre>
<pre class="overflow-x-auto rounded bg-gray-950 p-4 text-sm text-gray-100"><code
>{content}</code
></pre>
{:else}
<div class="prose prose-sm max-w-none whitespace-pre-wrap text-gray-700">{content}</div>
{/if}

View File

@@ -83,9 +83,7 @@ function makeSnippetResult(snippet: Snippet): SnippetSearchResult {
});
}
function makeMetadata(
overrides: Partial<ContextResponseMetadata> = {}
): ContextResponseMetadata {
function makeMetadata(overrides: Partial<ContextResponseMetadata> = {}): ContextResponseMetadata {
return {
localSource: false,
resultCount: 1,
@@ -160,7 +158,11 @@ describe('formatLibrarySearchJson', () => {
it('maps non-indexed state to initial', () => {
const results: LibrarySearchResult[] = [
new LibrarySearchResult({ repository: makeRepo({ state: 'pending' }), versions: [], score: 0 })
new LibrarySearchResult({
repository: makeRepo({ state: 'pending' }),
versions: [],
score: 0
})
];
const response = formatLibrarySearchJson(results);
expect(response.results[0].state).toBe('initial');
@@ -168,7 +170,11 @@ describe('formatLibrarySearchJson', () => {
it('handles null lastIndexedAt', () => {
const results: LibrarySearchResult[] = [
new LibrarySearchResult({ repository: makeRepo({ lastIndexedAt: null }), versions: [], score: 0 })
new LibrarySearchResult({
repository: makeRepo({ lastIndexedAt: null }),
versions: [],
score: 0
})
];
const response = formatLibrarySearchJson(results);
expect(response.results[0].lastUpdateDate).toBeNull();

View File

@@ -66,7 +66,9 @@ export const CORS_HEADERS = {
/**
* Convert internal LibrarySearchResult[] to the context7-compatible JSON body.
*/
export function formatLibrarySearchJson(results: LibrarySearchResult[]): LibrarySearchJsonResponseDto {
export function formatLibrarySearchJson(
results: LibrarySearchResult[]
): LibrarySearchJsonResponseDto {
return ContextResponseMapper.toLibrarySearchJson(results);
}
@@ -80,7 +82,7 @@ export function formatContextJson(
snippets: SnippetSearchResult[],
rules: string[],
metadata?: ContextResponseMetadata
): ContextJsonResponseDto {
): ContextJsonResponseDto {
return ContextResponseMapper.toContextJson(snippets, rules, metadata);
}
@@ -94,7 +96,10 @@ export function formatContextJson(
* @param snippets - Ranked snippet search results (already token-budget trimmed).
* @param rules - Rules from `trueref.json` / `repository_configs`.
*/
function formatOriginLine(result: SnippetSearchResult, metadata?: ContextResponseMetadata): string | null {
function formatOriginLine(
result: SnippetSearchResult,
metadata?: ContextResponseMetadata
): string | null {
if (!metadata?.repository) return null;
const parts = [

View File

@@ -115,10 +115,7 @@ describe('parseConfigFile — description', () => {
describe('parseConfigFile — array path fields', () => {
it('accepts valid folders', () => {
const result = parseConfigFile(
JSON.stringify({ folders: ['src/', 'docs/'] }),
'trueref.json'
);
const result = parseConfigFile(JSON.stringify({ folders: ['src/', 'docs/'] }), 'trueref.json');
expect(result.config.folders).toEqual(['src/', 'docs/']);
expect(result.warnings).toHaveLength(0);
});
@@ -130,10 +127,7 @@ describe('parseConfigFile — array path fields', () => {
});
it('skips non-string entries in folders with a warning', () => {
const result = parseConfigFile(
JSON.stringify({ folders: ['src/', 42, true] }),
'trueref.json'
);
const result = parseConfigFile(JSON.stringify({ folders: ['src/', 42, true] }), 'trueref.json');
expect(result.config.folders).toEqual(['src/']);
expect(result.warnings.length).toBeGreaterThan(0);
});
@@ -174,7 +168,9 @@ describe('parseConfigFile — array path fields', () => {
describe('parseConfigFile — rules', () => {
it('accepts valid rules', () => {
const result = parseConfigFile(
JSON.stringify({ rules: ['Always use named imports.', 'Prefer async/await over callbacks.'] }),
JSON.stringify({
rules: ['Always use named imports.', 'Prefer async/await over callbacks.']
}),
'trueref.json'
);
expect(result.config.rules).toHaveLength(2);
@@ -204,10 +200,7 @@ describe('parseConfigFile — rules', () => {
});
it('ignores non-array rules with a warning', () => {
const result = parseConfigFile(
JSON.stringify({ rules: 'use named imports' }),
'trueref.json'
);
const result = parseConfigFile(JSON.stringify({ rules: 'use named imports' }), 'trueref.json');
expect(result.config.rules).toBeUndefined();
expect(result.warnings.some((w) => /rules must be an array/.test(w))).toBe(true);
});
@@ -243,10 +236,7 @@ describe('parseConfigFile — previousVersions', () => {
it('skips entries missing tag', () => {
const result = parseConfigFile(
JSON.stringify({
previousVersions: [
{ title: 'No tag here' },
{ tag: 'v1.0.0', title: 'Valid' }
]
previousVersions: [{ title: 'No tag here' }, { tag: 'v1.0.0', title: 'Valid' }]
}),
'trueref.json'
);
@@ -275,10 +265,7 @@ describe('parseConfigFile — previousVersions', () => {
});
it('ignores non-array previousVersions with a warning', () => {
const result = parseConfigFile(
JSON.stringify({ previousVersions: 'v1.0.0' }),
'trueref.json'
);
const result = parseConfigFile(JSON.stringify({ previousVersions: 'v1.0.0' }), 'trueref.json');
expect(result.config.previousVersions).toBeUndefined();
expect(result.warnings.some((w) => /previousVersions must be an array/.test(w))).toBe(true);
});
@@ -294,9 +281,7 @@ describe('resolveConfig', () => {
});
it('returns null when no matching filenames', () => {
expect(
resolveConfig([{ filename: 'package.json', content: '{"name":"x"}' }])
).toBeNull();
expect(resolveConfig([{ filename: 'package.json', content: '{"name":"x"}' }])).toBeNull();
});
it('prefers trueref.json over context7.json', () => {

View File

@@ -65,7 +65,9 @@ export function parseConfigFile(content: string, filename: string): ParsedConfig
// ---- 2. Root must be an object ------------------------------------------
if (typeof raw !== 'object' || raw === null || Array.isArray(raw)) {
throw new ConfigParseError(`${filename} must be a JSON object, got ${Array.isArray(raw) ? 'array' : typeof raw}`);
throw new ConfigParseError(
`${filename} must be a JSON object, got ${Array.isArray(raw) ? 'array' : typeof raw}`
);
}
const input = raw as Record<string, unknown>;
@@ -131,7 +133,9 @@ export function parseConfigFile(content: string, filename: string): ParsedConfig
})
.map((item) => {
if (item.length > maxLength) {
warnings.push(`${field} entry truncated to ${maxLength} characters: "${item.slice(0, 40)}..."`);
warnings.push(
`${field} entry truncated to ${maxLength} characters: "${item.slice(0, 40)}..."`
);
return item.slice(0, maxLength);
}
return item;
@@ -160,9 +164,7 @@ export function parseConfigFile(content: string, filename: string): ParsedConfig
return false;
}
if (r.length < minLength) {
warnings.push(
`rules entry too short (< ${minLength} chars) — skipping: "${r}"`
);
warnings.push(`rules entry too short (< ${minLength} chars) — skipping: "${r}"`);
return false;
}
return true;

View File

@@ -74,27 +74,46 @@ export const MAX_FILE_SIZE_BYTES = 500_000;
*/
export const IGNORED_DIR_NAMES = new Set([
// ── Version control ────────────────────────────────────────────────────
'.git', '.hg', '.svn',
'.git',
'.hg',
'.svn',
// ── JavaScript / TypeScript ─────────────────────────────────────────────
'node_modules',
'.npm', '.yarn', '.pnpm-store', '.pnp',
'.npm',
'.yarn',
'.pnpm-store',
'.pnp',
// Build outputs and framework caches
'dist', 'build', 'out',
'.next', '.nuxt', '.svelte-kit', '.vite',
'.turbo', '.parcel-cache', '.webpack',
'dist',
'build',
'out',
'.next',
'.nuxt',
'.svelte-kit',
'.vite',
'.turbo',
'.parcel-cache',
'.webpack',
// ── Python ──────────────────────────────────────────────────────────────
'__pycache__',
'.venv', 'venv', 'env',
'site-packages', '.eggs',
'.pytest_cache', '.mypy_cache', '.ruff_cache',
'.tox', '.nox',
'.venv',
'venv',
'env',
'site-packages',
'.eggs',
'.pytest_cache',
'.mypy_cache',
'.ruff_cache',
'.tox',
'.nox',
'htmlcov',
// ── Java / Kotlin / Scala ───────────────────────────────────────────────
'target', // Maven + sbt
'.gradle', '.mvn',
'.gradle',
'.mvn',
// ── Ruby ────────────────────────────────────────────────────────────────
'.bundle',
@@ -103,19 +122,24 @@ export const IGNORED_DIR_NAMES = new Set([
// 'vendor' below covers PHP Composer
// ── .NET ────────────────────────────────────────────────────────────────
'bin', 'obj', 'packages',
'bin',
'obj',
'packages',
// ── Haskell ─────────────────────────────────────────────────────────────
'.stack-work', 'dist-newstyle',
'.stack-work',
'dist-newstyle',
// ── Dart / Flutter ──────────────────────────────────────────────────────
'.dart_tool',
// ── Swift / iOS ─────────────────────────────────────────────────────────
'Pods', 'DerivedData',
'Pods',
'DerivedData',
// ── Elixir / Erlang ─────────────────────────────────────────────────────
'_build', 'deps',
'_build',
'deps',
// ── Clojure ─────────────────────────────────────────────────────────────
'.cpcache',
@@ -125,16 +149,25 @@ export const IGNORED_DIR_NAMES = new Set([
'vendor',
// ── Generic caches / temp ───────────────────────────────────────────────
'.cache', '.tmp', 'tmp', 'temp', '.temp', '.sass-cache',
'.cache',
'.tmp',
'tmp',
'temp',
'.temp',
'.sass-cache',
// ── Test coverage ───────────────────────────────────────────────────────
'coverage', '.nyc_output',
'coverage',
'.nyc_output',
// ── IDE / editor artefacts ──────────────────────────────────────────────
'.idea', '.vs',
'.idea',
'.vs',
// ── Generated code ──────────────────────────────────────────────────────
'generated', '__generated__', '_generated',
'generated',
'__generated__',
'_generated',
// ── Logs ────────────────────────────────────────────────────────────────
'logs'
@@ -264,11 +297,7 @@ export function detectLanguage(filePath: string): string {
* 7. Must not be under a config.excludeFolders path / regex.
* 8. Must be under a config.folders allowlist path / regex (if specified).
*/
export function shouldIndexFile(
filePath: string,
fileSize: number,
config?: RepoConfig
): boolean {
export function shouldIndexFile(filePath: string, fileSize: number, config?: RepoConfig): boolean {
const ext = extname(filePath).toLowerCase();
const base = basename(filePath);

View File

@@ -35,10 +35,9 @@ export async function listGitHubTags(
};
if (token) headers['Authorization'] = `Bearer ${token}`;
const response = await fetch(
`https://api.github.com/repos/${owner}/${repo}/tags?per_page=100`,
{ headers }
);
const response = await fetch(`https://api.github.com/repos/${owner}/${repo}/tags?per_page=100`, {
headers
});
if (!response.ok) throw new GitHubApiError(response.status);
return response.json() as Promise<GitHubTag[]>;

View File

@@ -8,13 +8,14 @@
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { crawl } from './github.crawler.js';
import { shouldIndexFile, detectLanguage, INDEXABLE_EXTENSIONS, MAX_FILE_SIZE_BYTES } from './file-filter.js';
import { GitHubRateLimiter, Semaphore, withRetry } from './rate-limiter.js';
import {
AuthenticationError,
PermissionError,
RepositoryNotFoundError
} from './types.js';
shouldIndexFile,
detectLanguage,
INDEXABLE_EXTENSIONS,
MAX_FILE_SIZE_BYTES
} from './file-filter.js';
import { GitHubRateLimiter, Semaphore, withRetry } from './rate-limiter.js';
import { AuthenticationError, PermissionError, RepositoryNotFoundError } from './types.js';
// ---------------------------------------------------------------------------
// Mock fetch helpers
@@ -112,7 +113,9 @@ describe('shouldIndexFile()', () => {
});
it('respects config.excludeFolders prefix', () => {
expect(shouldIndexFile('internal/config.ts', 100, { excludeFolders: ['internal/'] })).toBe(false);
expect(shouldIndexFile('internal/config.ts', 100, { excludeFolders: ['internal/'] })).toBe(
false
);
});
it('allows files outside of config.excludeFolders', () => {
@@ -169,8 +172,10 @@ describe('detectLanguage()', () => {
it('detects markdown', () => expect(detectLanguage('README.md')).toBe('markdown'));
it('detects svelte', () => expect(detectLanguage('App.svelte')).toBe('svelte'));
it('detects yaml', () => expect(detectLanguage('config.yaml')).toBe('yaml'));
it('returns empty string for unknown extension', () => expect(detectLanguage('file.xyz')).toBe(''));
it('is case-insensitive for extensions', () => expect(detectLanguage('FILE.TS')).toBe('typescript'));
it('returns empty string for unknown extension', () =>
expect(detectLanguage('file.xyz')).toBe(''));
it('is case-insensitive for extensions', () =>
expect(detectLanguage('FILE.TS')).toBe('typescript'));
});
// ---------------------------------------------------------------------------
@@ -267,9 +272,9 @@ describe('withRetry()', () => {
});
it('throws after exhausting all attempts', async () => {
await expect(
withRetry(() => Promise.reject(new Error('always fails')), 3)
).rejects.toThrow('always fails');
await expect(withRetry(() => Promise.reject(new Error('always fails')), 3)).rejects.toThrow(
'always fails'
);
});
});
@@ -425,7 +430,8 @@ describe('crawl()', () => {
});
it('throws AuthenticationError on 401', async () => {
stubFetch(() =>
stubFetch(
() =>
new Response('Unauthorized', {
status: 401,
headers: {
@@ -441,7 +447,8 @@ describe('crawl()', () => {
});
it('throws PermissionError on 403 without rate-limit exhaustion', async () => {
stubFetch(() =>
stubFetch(
() =>
new Response('Forbidden', {
status: 403,
headers: {

View File

@@ -106,9 +106,7 @@ async function throwForStatus(response: Response, rateLimiter: GitHubRateLimiter
);
}
case 404:
throw new RepositoryNotFoundError(
`Repository not found or not accessible: ${response.url}`
);
throw new RepositoryNotFoundError(`Repository not found or not accessible: ${response.url}`);
default: {
const body = await response.text().catch(() => '');
throw new Error(`GitHub API error ${response.status}: ${body}`);
@@ -129,7 +127,8 @@ async function fetchRepoInfo(
token: string | undefined,
rateLimiter: GitHubRateLimiter
): Promise<GitHubRepoResponse> {
return withRetry(async () => {
return withRetry(
async () => {
await rateLimiter.waitIfNeeded();
const response = await fetch(`${GITHUB_API}/repos/${owner}/${repo}`, {
@@ -140,7 +139,10 @@ async function fetchRepoInfo(
await throwForStatus(response, rateLimiter);
return (await response.json()) as GitHubRepoResponse;
}, 3, isRetryable);
},
3,
isRetryable
);
}
/**
@@ -155,7 +157,8 @@ async function fetchTree(
token: string | undefined,
rateLimiter: GitHubRateLimiter
): Promise<GitHubTreeResponse | null> {
return withRetry(async () => {
return withRetry(
async () => {
await rateLimiter.waitIfNeeded();
const url = `${GITHUB_API}/repos/${owner}/${repo}/git/trees/${ref}?recursive=1`;
@@ -169,7 +172,10 @@ async function fetchTree(
await throwForStatus(response, rateLimiter);
return (await response.json()) as GitHubTreeResponse;
}, 3, isRetryable);
},
3,
isRetryable
);
}
/**
@@ -184,7 +190,8 @@ async function fetchSubTree(
token: string | undefined,
rateLimiter: GitHubRateLimiter
): Promise<GitHubTreeResponse> {
return withRetry(async () => {
return withRetry(
async () => {
await rateLimiter.waitIfNeeded();
const url = `${GITHUB_API}/repos/${owner}/${repo}/git/trees/${treeSha}`;
@@ -194,7 +201,10 @@ async function fetchSubTree(
await throwForStatus(response, rateLimiter);
return (await response.json()) as GitHubTreeResponse;
}, 3, isRetryable);
},
3,
isRetryable
);
}
/**
@@ -208,7 +218,8 @@ async function fetchCommitSha(
token: string | undefined,
rateLimiter: GitHubRateLimiter
): Promise<string> {
return withRetry(async () => {
return withRetry(
async () => {
await rateLimiter.waitIfNeeded();
const url = `${GITHUB_API}/repos/${owner}/${repo}/commits/${ref}`;
@@ -222,7 +233,10 @@ async function fetchCommitSha(
// When Accept is 'application/vnd.github.sha', the response body is the
// bare SHA string.
return (await response.text()).trim();
}, 3, isRetryable);
},
3,
isRetryable
);
}
/**
@@ -347,14 +361,7 @@ async function fetchRepoConfig(
const content =
(await downloadRawFile(owner, repo, ref, configItem.path, token)) ??
(await downloadViaContentsApi(
owner,
repo,
ref,
configItem.path,
token,
rateLimiter
));
(await downloadViaContentsApi(owner, repo, ref, configItem.path, token, rateLimiter));
if (!content) return undefined;
@@ -435,14 +442,7 @@ export async function crawl(options: CrawlOptions): Promise<CrawlResult> {
// Prefer raw download (cheaper on rate limit); fall back to API.
const content =
(await downloadRawFile(owner, repo, ref!, item.path, token)) ??
(await downloadViaContentsApi(
owner,
repo,
ref!,
item.path,
token,
rateLimiter
));
(await downloadViaContentsApi(owner, repo, ref!, item.path, token, rateLimiter));
if (content === null) {
console.warn(`[GitHubCrawler] Could not download: ${item.path} — skipping.`);

View File

@@ -52,7 +52,9 @@ async function cleanupTempRepo(root: string): Promise<void> {
let root: string = '';
const crawler = new LocalCrawler();
async function crawlRoot(opts: Partial<LocalCrawlOptions> = {}): Promise<ReturnType<LocalCrawler['crawl']>> {
async function crawlRoot(
opts: Partial<LocalCrawlOptions> = {}
): Promise<ReturnType<LocalCrawler['crawl']>> {
return crawler.crawl({ rootPath: root, ...opts });
}

View File

@@ -141,7 +141,12 @@ export class LocalCrawler {
});
// Crawl the worktree and stamp the result with the git-resolved metadata.
const result = await this.crawlDirectory(worktreePath, options.config, options.onProgress, ref);
const result = await this.crawlDirectory(
worktreePath,
options.config,
options.onProgress,
ref
);
return { ...result, commitSha };
} finally {

View File

@@ -78,12 +78,8 @@
"name": "documents_repository_id_repositories_id_fk",
"tableFrom": "documents",
"tableTo": "repositories",
"columnsFrom": [
"repository_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["repository_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
},
@@ -91,12 +87,8 @@
"name": "documents_version_id_repository_versions_id_fk",
"tableFrom": "documents",
"tableTo": "repository_versions",
"columnsFrom": [
"version_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["version_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}
@@ -196,12 +188,8 @@
"name": "indexing_jobs_repository_id_repositories_id_fk",
"tableFrom": "indexing_jobs",
"tableTo": "repositories",
"columnsFrom": [
"repository_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["repository_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}
@@ -411,12 +399,8 @@
"name": "repository_configs_repository_id_repositories_id_fk",
"tableFrom": "repository_configs",
"tableTo": "repositories",
"columnsFrom": [
"repository_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["repository_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}
@@ -493,12 +477,8 @@
"name": "repository_versions_repository_id_repositories_id_fk",
"tableFrom": "repository_versions",
"tableTo": "repositories",
"columnsFrom": [
"repository_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["repository_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}
@@ -583,12 +563,8 @@
"name": "snippet_embeddings_snippet_id_snippets_id_fk",
"tableFrom": "snippet_embeddings",
"tableTo": "snippets",
"columnsFrom": [
"snippet_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["snippet_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}
@@ -685,12 +661,8 @@
"name": "snippets_document_id_documents_id_fk",
"tableFrom": "snippets",
"tableTo": "documents",
"columnsFrom": [
"document_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["document_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
},
@@ -698,12 +670,8 @@
"name": "snippets_repository_id_repositories_id_fk",
"tableFrom": "snippets",
"tableTo": "repositories",
"columnsFrom": [
"repository_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["repository_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
},
@@ -711,12 +679,8 @@
"name": "snippets_version_id_repository_versions_id_fk",
"tableFrom": "snippets",
"tableTo": "repository_versions",
"columnsFrom": [
"version_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["version_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}

View File

@@ -78,12 +78,8 @@
"name": "documents_repository_id_repositories_id_fk",
"tableFrom": "documents",
"tableTo": "repositories",
"columnsFrom": [
"repository_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["repository_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
},
@@ -91,12 +87,8 @@
"name": "documents_version_id_repository_versions_id_fk",
"tableFrom": "documents",
"tableTo": "repository_versions",
"columnsFrom": [
"version_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["version_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}
@@ -196,12 +188,8 @@
"name": "indexing_jobs_repository_id_repositories_id_fk",
"tableFrom": "indexing_jobs",
"tableTo": "repositories",
"columnsFrom": [
"repository_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["repository_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}
@@ -411,12 +399,8 @@
"name": "repository_configs_repository_id_repositories_id_fk",
"tableFrom": "repository_configs",
"tableTo": "repositories",
"columnsFrom": [
"repository_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["repository_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}
@@ -500,12 +484,8 @@
"name": "repository_versions_repository_id_repositories_id_fk",
"tableFrom": "repository_versions",
"tableTo": "repositories",
"columnsFrom": [
"repository_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["repository_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}
@@ -590,12 +570,8 @@
"name": "snippet_embeddings_snippet_id_snippets_id_fk",
"tableFrom": "snippet_embeddings",
"tableTo": "snippets",
"columnsFrom": [
"snippet_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["snippet_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}
@@ -692,12 +668,8 @@
"name": "snippets_document_id_documents_id_fk",
"tableFrom": "snippets",
"tableTo": "documents",
"columnsFrom": [
"document_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["document_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
},
@@ -705,12 +677,8 @@
"name": "snippets_repository_id_repositories_id_fk",
"tableFrom": "snippets",
"tableTo": "repositories",
"columnsFrom": [
"repository_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["repository_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
},
@@ -718,12 +686,8 @@
"name": "snippets_version_id_repository_versions_id_fk",
"tableFrom": "snippets",
"tableTo": "repository_versions",
"columnsFrom": [
"version_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["version_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}

View File

@@ -78,12 +78,8 @@
"name": "documents_repository_id_repositories_id_fk",
"tableFrom": "documents",
"tableTo": "repositories",
"columnsFrom": [
"repository_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["repository_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
},
@@ -91,12 +87,8 @@
"name": "documents_version_id_repository_versions_id_fk",
"tableFrom": "documents",
"tableTo": "repository_versions",
"columnsFrom": [
"version_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["version_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}
@@ -278,12 +270,8 @@
"name": "indexing_jobs_repository_id_repositories_id_fk",
"tableFrom": "indexing_jobs",
"tableTo": "repositories",
"columnsFrom": [
"repository_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["repository_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}
@@ -493,12 +481,8 @@
"name": "repository_configs_repository_id_repositories_id_fk",
"tableFrom": "repository_configs",
"tableTo": "repositories",
"columnsFrom": [
"repository_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["repository_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}
@@ -582,12 +566,8 @@
"name": "repository_versions_repository_id_repositories_id_fk",
"tableFrom": "repository_versions",
"tableTo": "repositories",
"columnsFrom": [
"repository_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["repository_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}
@@ -679,12 +659,8 @@
"name": "snippet_embeddings_snippet_id_snippets_id_fk",
"tableFrom": "snippet_embeddings",
"tableTo": "snippets",
"columnsFrom": [
"snippet_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["snippet_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
},
@@ -692,22 +668,15 @@
"name": "snippet_embeddings_profile_id_embedding_profiles_id_fk",
"tableFrom": "snippet_embeddings",
"tableTo": "embedding_profiles",
"columnsFrom": [
"profile_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["profile_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {
"snippet_embeddings_snippet_id_profile_id_pk": {
"columns": [
"snippet_id",
"profile_id"
],
"columns": ["snippet_id", "profile_id"],
"name": "snippet_embeddings_snippet_id_profile_id_pk"
}
},
@@ -802,12 +771,8 @@
"name": "snippets_document_id_documents_id_fk",
"tableFrom": "snippets",
"tableTo": "documents",
"columnsFrom": [
"document_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["document_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
},
@@ -815,12 +780,8 @@
"name": "snippets_repository_id_repositories_id_fk",
"tableFrom": "snippets",
"tableTo": "repositories",
"columnsFrom": [
"repository_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["repository_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
},
@@ -828,12 +789,8 @@
"name": "snippets_version_id_repository_versions_id_fk",
"tableFrom": "snippets",
"tableTo": "repository_versions",
"columnsFrom": [
"version_id"
],
"columnsTo": [
"id"
],
"columnsFrom": ["version_id"],
"columnsTo": ["id"],
"onDelete": "cascade",
"onUpdate": "no action"
}

View File

@@ -381,7 +381,12 @@ describe('EmbeddingService', () => {
.all(snippetId, 'local-default');
expect(rows).toHaveLength(1);
const row = rows[0] as { model: string; dimensions: number; embedding: Buffer; profile_id: string };
const row = rows[0] as {
model: string;
dimensions: number;
embedding: Buffer;
profile_id: string;
};
expect(row.model).toBe('test-model');
expect(row.dimensions).toBe(4);
expect(row.profile_id).toBe('local-default');
@@ -494,9 +499,7 @@ describe('createProviderFromConfig', () => {
});
it('throws when openai provider is selected without config', () => {
expect(() =>
createProviderFromConfig({ provider: 'openai' } as EmbeddingConfig)
).toThrow();
expect(() => createProviderFromConfig({ provider: 'openai' } as EmbeddingConfig)).toThrow();
});
it('defaultEmbeddingConfig returns provider=none', () => {

View File

@@ -41,18 +41,16 @@ export class EmbeddingService {
const placeholders = snippetIds.map(() => '?').join(',');
const snippets = this.db
.prepare<string[], SnippetRow>(
`SELECT id, title, breadcrumb, content FROM snippets WHERE id IN (${placeholders})`
)
.prepare<
string[],
SnippetRow
>(`SELECT id, title, breadcrumb, content FROM snippets WHERE id IN (${placeholders})`)
.all(...snippetIds);
if (snippets.length === 0) return;
const texts = snippets.map((s) =>
[s.title, s.breadcrumb, s.content]
.filter(Boolean)
.join('\n')
.slice(0, TEXT_MAX_CHARS)
[s.title, s.breadcrumb, s.content].filter(Boolean).join('\n').slice(0, TEXT_MAX_CHARS)
);
const insert = this.db.prepare<[string, string, string, number, Buffer]>(`
@@ -94,9 +92,10 @@ export class EmbeddingService {
*/
getEmbedding(snippetId: string, profileId: string = 'local-default'): Float32Array | null {
const row = this.db
.prepare<[string, string], { embedding: Buffer; dimensions: number }>(
`SELECT embedding, dimensions FROM snippet_embeddings WHERE snippet_id = ? AND profile_id = ?`
)
.prepare<
[string, string],
{ embedding: Buffer; dimensions: number }
>(`SELECT embedding, dimensions FROM snippet_embeddings WHERE snippet_id = ? AND profile_id = ?`)
.get(snippetId, profileId);
if (!row) return null;

View File

@@ -12,7 +12,11 @@ import { OpenAIEmbeddingProvider } from './openai.provider.js';
import { LocalEmbeddingProvider } from './local.provider.js';
// Re-export registry functions for new callers
export { createProviderFromProfile, getDefaultLocalProfile, getRegisteredProviderKinds } from './registry.js';
export {
createProviderFromProfile,
getDefaultLocalProfile,
getRegisteredProviderKinds
} from './registry.js';
export interface EmbeddingConfig {
provider: 'openai' | 'local' | 'none';

View File

@@ -43,7 +43,12 @@ export class ContextResponseMapper {
lastUpdateDate: repository.lastIndexedAt
? repository.lastIndexedAt.toISOString()
: null,
state: repository.state === 'indexed' ? 'finalized' : repository.state === 'error' ? 'error' : 'initial',
state:
repository.state === 'indexed'
? 'finalized'
: repository.state === 'error'
? 'error'
: 'initial',
totalTokens: repository.totalTokens ?? null,
totalSnippets: repository.totalSnippets ?? null,
stars: repository.stars ?? null,
@@ -68,7 +73,9 @@ export class ContextResponseMapper {
repositoryTitle: metadata.repository.title,
source: metadata.repository.source,
sourceUrl: metadata.repository.sourceUrl,
version: snippet.versionId ? metadata.snippetVersions[snippet.versionId] ?? null : null,
version: snippet.versionId
? (metadata.snippetVersions[snippet.versionId] ?? null)
: null,
versionId: snippet.versionId,
isLocal: metadata.localSource
})

View File

@@ -12,8 +12,7 @@ export class IndexingJobMapper {
processedFiles: entity.processed_files,
error: entity.error,
startedAt: entity.started_at != null ? new Date(entity.started_at * 1000) : null,
completedAt:
entity.completed_at != null ? new Date(entity.completed_at * 1000) : null,
completedAt: entity.completed_at != null ? new Date(entity.completed_at * 1000) : null,
createdAt: new Date(entity.created_at * 1000)
});
}

View File

@@ -1,4 +1,8 @@
import { LibrarySearchResult, SnippetRepositoryRef, SnippetSearchResult } from '$lib/server/models/search-result.js';
import {
LibrarySearchResult,
SnippetRepositoryRef,
SnippetSearchResult
} from '$lib/server/models/search-result.js';
import { RepositoryMapper } from '$lib/server/mappers/repository.mapper.js';
import { RepositoryVersionMapper } from '$lib/server/mappers/repository-version.mapper.js';
import { SnippetMapper } from '$lib/server/mappers/snippet.mapper.js';
@@ -26,9 +30,7 @@ export class SearchResultMapper {
): LibrarySearchResult {
return new LibrarySearchResult({
repository: RepositoryMapper.fromEntity(repositoryEntity),
versions: versionEntities.map((version) =>
RepositoryVersionMapper.fromEntity(version)
),
versions: versionEntities.map((version) => RepositoryVersionMapper.fromEntity(version)),
score
});
}

View File

@@ -71,7 +71,7 @@ export class SnippetOriginJsonDto {
this.versionId = props.versionId;
this.isLocal = props.isLocal;
}
}
}
export class LibrarySearchJsonResultDto {
id: string;

View File

@@ -286,7 +286,8 @@ This is the second paragraph that also has enough content to be included here.
});
it('skips paragraphs shorter than 20 characters', () => {
const content = 'Short.\n\nThis is a much longer paragraph that definitely passes the minimum length filter.';
const content =
'Short.\n\nThis is a much longer paragraph that definitely passes the minimum length filter.';
const snippets = parseCodeFile(content, 'notes.txt', 'text');
expect(snippets.length).toBe(1);
});
@@ -331,7 +332,10 @@ export function realFunction(): string {
describe('parseCodeFile — token count', () => {
it('all snippets have tokenCount within MAX_TOKENS', () => {
const lines = Array.from({ length: 300 }, (_, i) => `// comment line number ${i} here\nconst x${i} = ${i};`);
const lines = Array.from(
{ length: 300 },
(_, i) => `// comment line number ${i} here\nconst x${i} = ${i};`
);
const content = lines.join('\n');
const snippets = parseCodeFile(content, 'large.ts', 'typescript');

View File

@@ -26,15 +26,19 @@ import {
* The regex is tested line-by-line (multiline flag not needed).
*/
export const BOUNDARY_PATTERNS: Record<string, RegExp> = {
typescript: /^(export\s+)?(declare\s+)?(async\s+)?(function|class|interface|type|enum|const|let|var)\s+\w+/,
typescript:
/^(export\s+)?(declare\s+)?(async\s+)?(function|class|interface|type|enum|const|let|var)\s+\w+/,
javascript: /^(export\s+)?(async\s+)?(function|class|const|let|var)\s+\w+/,
python: /^(async\s+)?(def|class)\s+\w+/,
go: /^(func|type|var|const)\s+\w+/,
rust: /^(pub(\s*\(crate\))?\s+)?(async\s+)?(fn|impl|struct|enum|trait|type|const|static)\s+\w+/,
java: /^(\s*(public|private|protected|static|final|abstract|synchronized)\s+)+[\w<>\[\]]+\s+\w+\s*[({]/,
csharp: /^(\s*(public|private|protected|internal|static|override|virtual|abstract|sealed)\s+)+[\w<>\[\]]+\s+\w+\s*[({]/,
kotlin: /^(\s*(public|private|protected|internal|override|suspend|inline|open|abstract|sealed)\s+)*(fun|class|object|interface|data class|sealed class|enum class)\s+\w+/,
swift: /^(\s*(public|private|internal|fileprivate|open|override|static|final|class)\s+)*(func|class|struct|enum|protocol|extension)\s+\w+/,
csharp:
/^(\s*(public|private|protected|internal|static|override|virtual|abstract|sealed)\s+)+[\w<>\[\]]+\s+\w+\s*[({]/,
kotlin:
/^(\s*(public|private|protected|internal|override|suspend|inline|open|abstract|sealed)\s+)*(fun|class|object|interface|data class|sealed class|enum class)\s+\w+/,
swift:
/^(\s*(public|private|internal|fileprivate|open|override|static|final|class)\s+)*(func|class|struct|enum|protocol|extension)\s+\w+/,
ruby: /^(def|class|module)\s+\w+/
};
@@ -42,7 +46,10 @@ export const BOUNDARY_PATTERNS: Record<string, RegExp> = {
// Internal types
// ---------------------------------------------------------------------------
type RawSnippet = Omit<NewSnippet, 'id' | 'repositoryId' | 'documentId' | 'versionId' | 'createdAt'>;
type RawSnippet = Omit<
NewSnippet,
'id' | 'repositoryId' | 'documentId' | 'versionId' | 'createdAt'
>;
// ---------------------------------------------------------------------------
// Helpers
@@ -161,7 +168,10 @@ function parseHtmlLikeFile(content: string, filePath: string, language: string):
while ((match = scriptPattern.exec(content)) !== null) {
// Strip the outer tags, keep just the code
const inner = match[0].replace(/^<script[^>]*>/, '').replace(/<\/script>$/, '').trim();
const inner = match[0]
.replace(/^<script[^>]*>/, '')
.replace(/<\/script>$/, '')
.trim();
if (inner.length >= MIN_CONTENT_LENGTH) {
scriptBlocks.push(inner);
}

View File

@@ -48,6 +48,13 @@ export function parseFile(file: CrawledFile, options: ParseOptions): NewSnippet[
// Re-export helpers for consumers that need them individually
export { detectLanguage } from './language.js';
export { estimateTokens, chunkText, chunkLines, MAX_TOKENS, OVERLAP_TOKENS, MIN_CONTENT_LENGTH } from './chunker.js';
export {
estimateTokens,
chunkText,
chunkLines,
MAX_TOKENS,
OVERLAP_TOKENS,
MIN_CONTENT_LENGTH
} from './chunker.js';
export { parseMarkdown } from './markdown.parser.js';
export { parseCodeFile, BOUNDARY_PATTERNS } from './code.parser.js';

View File

@@ -99,7 +99,10 @@ describe('parseMarkdown — section splitting', () => {
describe('parseMarkdown — code block extraction', () => {
it('extracts a fenced code block as a code snippet', () => {
const codeBlock = fence('typescript', 'function hello(name: string): string {\n return `Hello, ${name}!`;\n}');
const codeBlock = fence(
'typescript',
'function hello(name: string): string {\n return `Hello, ${name}!`;\n}'
);
const source = [
'# Example',
'',
@@ -232,7 +235,10 @@ describe('parseMarkdown — large content chunking', () => {
describe('parseMarkdown — real-world sample', () => {
it('correctly parses a realistic README excerpt', () => {
const bashInstall = fence('bash', 'npm install my-library');
const tsUsage = fence('typescript', "import { doTheThing } from 'my-library';\n\ndoTheThing({ verbose: true });");
const tsUsage = fence(
'typescript',
"import { doTheThing } from 'my-library';\n\ndoTheThing({ verbose: true });"
);
const source = [
'# My Library',

View File

@@ -7,7 +7,13 @@
import { basename } from 'node:path';
import type { NewSnippet } from '$lib/server/db/schema.js';
import { estimateTokens, chunkText, MAX_TOKENS, OVERLAP_TOKENS, MIN_CONTENT_LENGTH } from './chunker.js';
import {
estimateTokens,
chunkText,
MAX_TOKENS,
OVERLAP_TOKENS,
MIN_CONTENT_LENGTH
} from './chunker.js';
// ---------------------------------------------------------------------------
// Internal types
@@ -121,7 +127,10 @@ function splitIntoSections(source: string): MarkdownSection[] {
// Public parser
// ---------------------------------------------------------------------------
type RawSnippet = Omit<NewSnippet, 'id' | 'repositoryId' | 'documentId' | 'versionId' | 'createdAt'>;
type RawSnippet = Omit<
NewSnippet,
'id' | 'repositoryId' | 'documentId' | 'versionId' | 'createdAt'
>;
/**
* Parse a Markdown/MDX file into raw snippets (before IDs and DB fields are

View File

@@ -22,10 +22,7 @@ function createTestDb(): Database.Database {
client.pragma('foreign_keys = ON');
const migrationsFolder = join(import.meta.dirname, '../db/migrations');
const migrationSql = readFileSync(
join(migrationsFolder, '0000_large_master_chief.sql'),
'utf-8'
);
const migrationSql = readFileSync(join(migrationsFolder, '0000_large_master_chief.sql'), 'utf-8');
const statements = migrationSql
.split('--> statement-breakpoint')
@@ -45,10 +42,7 @@ function createTestDb(): Database.Database {
const now = Math.floor(Date.now() / 1000);
function insertRepo(
db: Database.Database,
overrides: Partial<Record<string, unknown>> = {}
): void {
function insertRepo(db: Database.Database, overrides: Partial<Record<string, unknown>> = {}): void {
db.prepare(
`INSERT INTO repositories
(id, title, source, source_url, branch, state,
@@ -62,7 +56,15 @@ function insertRepo(
overrides.source_url ?? '/tmp/test-repo',
overrides.branch ?? 'main',
overrides.state ?? 'pending',
0, 0, 0, 0, null, null, null, now, now
0,
0,
0,
0,
null,
null,
null,
now,
now
);
}
@@ -108,9 +110,10 @@ describe('recoverStaleJobs', () => {
insertJob(db, { status: 'running' });
recoverStaleJobs(db);
const row = db
.prepare(`SELECT status, error FROM indexing_jobs LIMIT 1`)
.get() as { status: string; error: string };
const row = db.prepare(`SELECT status, error FROM indexing_jobs LIMIT 1`).get() as {
status: string;
error: string;
};
expect(row.status).toBe('failed');
expect(row.error).toMatch(/restarted/i);
});
@@ -119,9 +122,9 @@ describe('recoverStaleJobs', () => {
db.prepare(`UPDATE repositories SET state = 'indexing' WHERE id = '/test/repo'`).run();
recoverStaleJobs(db);
const row = db
.prepare(`SELECT state FROM repositories WHERE id = '/test/repo'`)
.get() as { state: string };
const row = db.prepare(`SELECT state FROM repositories WHERE id = '/test/repo'`).get() as {
state: string;
};
expect(row.state).toBe('error');
});
@@ -164,9 +167,7 @@ describe('JobQueue', () => {
const job2 = queue.enqueue('/test/repo');
expect(job1.id).toBe(job2.id);
const count = (
db.prepare(`SELECT COUNT(*) as n FROM indexing_jobs`).get() as { n: number }
).n;
const count = (db.prepare(`SELECT COUNT(*) as n FROM indexing_jobs`).get() as { n: number }).n;
expect(count).toBe(1);
});
@@ -255,19 +256,19 @@ describe('IndexingPipeline', () => {
})
};
return new IndexingPipeline(
db,
mockGithubCrawl as never,
mockLocalCrawler as never,
null
);
return new IndexingPipeline(db, mockGithubCrawl as never, mockLocalCrawler as never, null);
}
function makeJob(repositoryId = '/test/repo') {
const jobId = insertJob(db, { repository_id: repositoryId, status: 'queued' });
return db
.prepare(`SELECT * FROM indexing_jobs WHERE id = ?`)
.get(jobId) as { id: string; repositoryId?: string; repository_id?: string; status: string; versionId?: string; version_id?: string };
return db.prepare(`SELECT * FROM indexing_jobs WHERE id = ?`).get(jobId) as {
id: string;
repositoryId?: string;
repository_id?: string;
status: string;
versionId?: string;
version_id?: string;
};
}
it('marks job as done when there are no files to index', async () => {
@@ -289,9 +290,9 @@ describe('IndexingPipeline', () => {
await pipeline.run(job as never);
const updated = db
.prepare(`SELECT status FROM indexing_jobs WHERE id = ?`)
.get(job.id) as { status: string };
const updated = db.prepare(`SELECT status FROM indexing_jobs WHERE id = ?`).get(job.id) as {
status: string;
};
// The job should end in 'done' — the running→done transition is covered
// by the pipeline's internal updateJob calls.
expect(updated.status).toBe('done');
@@ -363,27 +364,24 @@ describe('IndexingPipeline', () => {
const job1 = makeJob();
await pipeline.run(job1 as never);
const firstDocCount = (
db.prepare(`SELECT COUNT(*) as n FROM documents`).get() as { n: number }
).n;
const firstSnippetIds = (
db.prepare(`SELECT id FROM snippets`).all() as { id: string }[]
).map((r) => r.id);
const firstDocCount = (db.prepare(`SELECT COUNT(*) as n FROM documents`).get() as { n: number })
.n;
const firstSnippetIds = (db.prepare(`SELECT id FROM snippets`).all() as { id: string }[]).map(
(r) => r.id
);
// Second run with identical files.
const job2Id = insertJob(db, { repository_id: '/test/repo', status: 'queued' });
const job2 = db
.prepare(`SELECT * FROM indexing_jobs WHERE id = ?`)
.get(job2Id) as never;
const job2 = db.prepare(`SELECT * FROM indexing_jobs WHERE id = ?`).get(job2Id) as never;
await pipeline.run(job2);
const secondDocCount = (
db.prepare(`SELECT COUNT(*) as n FROM documents`).get() as { n: number }
).n;
const secondSnippetIds = (
db.prepare(`SELECT id FROM snippets`).all() as { id: string }[]
).map((r) => r.id);
const secondSnippetIds = (db.prepare(`SELECT id FROM snippets`).all() as { id: string }[]).map(
(r) => r.id
);
// Document count stays the same and snippet IDs are unchanged.
expect(secondDocCount).toBe(firstDocCount);
@@ -395,7 +393,8 @@ describe('IndexingPipeline', () => {
files: [
{
path: 'README.md',
content: '# Original\n\nThis is the original version of the documentation with sufficient content.',
content:
'# Original\n\nThis is the original version of the documentation with sufficient content.',
sha: 'sha-v1',
language: 'markdown'
}
@@ -415,7 +414,8 @@ describe('IndexingPipeline', () => {
files: [
{
path: 'README.md',
content: '# Updated\n\nThis is a completely different version of the documentation with new content.',
content:
'# Updated\n\nThis is a completely different version of the documentation with new content.',
sha: 'sha-v2',
language: 'markdown'
}
@@ -423,14 +423,11 @@ describe('IndexingPipeline', () => {
totalFiles: 1
});
const job2Id = insertJob(db, { repository_id: '/test/repo', status: 'queued' });
const job2 = db
.prepare(`SELECT * FROM indexing_jobs WHERE id = ?`)
.get(job2Id) as never;
const job2 = db.prepare(`SELECT * FROM indexing_jobs WHERE id = ?`).get(job2Id) as never;
await pipeline2.run(job2);
const finalDocCount = (
db.prepare(`SELECT COUNT(*) as n FROM documents`).get() as { n: number }
).n;
const finalDocCount = (db.prepare(`SELECT COUNT(*) as n FROM documents`).get() as { n: number })
.n;
// Only one document should exist (the updated one).
expect(finalDocCount).toBe(1);
@@ -452,9 +449,9 @@ describe('IndexingPipeline', () => {
const job = makeJob();
await pipeline.run(job as never);
const updated = db
.prepare(`SELECT progress FROM indexing_jobs WHERE id = ?`)
.get(job.id) as { progress: number };
const updated = db.prepare(`SELECT progress FROM indexing_jobs WHERE id = ?`).get(job.id) as {
progress: number;
};
expect(updated.progress).toBe(100);
});
@@ -467,12 +464,7 @@ describe('IndexingPipeline', () => {
commitSha: 'abc'
});
const pipeline = new IndexingPipeline(
db,
vi.fn() as never,
{ crawl } as never,
null
);
const pipeline = new IndexingPipeline(db, vi.fn() as never, { crawl } as never, null);
const job = makeJob();
await pipeline.run(job as never);
@@ -511,7 +503,10 @@ describe('IndexingPipeline', () => {
await pipeline1.run(job1 as never);
const afterFirstRun = {
docs: db.prepare(`SELECT file_path, checksum FROM documents ORDER BY file_path`).all() as { file_path: string; checksum: string }[],
docs: db.prepare(`SELECT file_path, checksum FROM documents ORDER BY file_path`).all() as {
file_path: string;
checksum: string;
}[],
snippetCount: (db.prepare(`SELECT COUNT(*) as n FROM snippets`).get() as { n: number }).n
};
expect(afterFirstRun.docs).toHaveLength(3);

View File

@@ -250,7 +250,10 @@ export class IndexingPipeline {
private async crawl(
repo: Repository,
job: IndexingJob
): Promise<{ files: Array<{ path: string; content: string; sha: string; size: number; language: string }>; totalFiles: number }> {
): Promise<{
files: Array<{ path: string; content: string; sha: string; size: number; language: string }>;
totalFiles: number;
}> {
if (repo.source === 'github') {
// Parse owner/repo from the canonical ID: "/owner/repo"
const parts = repo.id.replace(/^\//, '').split('/');

View File

@@ -133,9 +133,7 @@ export class JobQueue {
// Check whether another job was queued while this one ran.
const next = this.db
.prepare<[], { id: string }>(
`SELECT id FROM indexing_jobs WHERE status = 'queued' LIMIT 1`
)
.prepare<[], { id: string }>(`SELECT id FROM indexing_jobs WHERE status = 'queued' LIMIT 1`)
.get();
if (next) {
setImmediate(() => this.processNext());
@@ -147,9 +145,7 @@ export class JobQueue {
* Retrieve a single job by ID.
*/
getJob(id: string): IndexingJob | null {
const raw = this.db
.prepare<[string], IndexingJobEntity>(`${JOB_SELECT} WHERE id = ?`)
.get(id);
const raw = this.db.prepare<[string], IndexingJobEntity>(`${JOB_SELECT} WHERE id = ?`).get(id);
return raw ? IndexingJobMapper.fromEntity(new IndexingJobEntity(raw)) : null;
}
@@ -178,9 +174,9 @@ export class JobQueue {
const sql = `${JOB_SELECT} ${where} ORDER BY created_at DESC LIMIT ?`;
params.push(limit);
return (this.db.prepare<unknown[], IndexingJobEntity>(sql).all(...params) as IndexingJobEntity[]).map(
(row) => IndexingJobMapper.fromEntity(new IndexingJobEntity(row))
);
return (
this.db.prepare<unknown[], IndexingJobEntity>(sql).all(...params) as IndexingJobEntity[]
).map((row) => IndexingJobMapper.fromEntity(new IndexingJobEntity(row)));
}
/**
@@ -228,9 +224,7 @@ export class JobQueue {
return false;
}
this.db
.prepare(`UPDATE indexing_jobs SET status = 'paused' WHERE id = ?`)
.run(id);
this.db.prepare(`UPDATE indexing_jobs SET status = 'paused' WHERE id = ?`).run(id);
return true;
}
@@ -249,9 +243,7 @@ export class JobQueue {
return false;
}
this.db
.prepare(`UPDATE indexing_jobs SET status = 'queued' WHERE id = ?`)
.run(id);
this.db.prepare(`UPDATE indexing_jobs SET status = 'queued' WHERE id = ?`).run(id);
// Trigger queue processing in case the queue was idle
this.drainQueued();

View File

@@ -27,7 +27,11 @@ function createTestDb(): Database.Database {
const migrationsFolder = join(import.meta.dirname, '../db/migrations');
// Run all migrations in order
const migrations = ['0000_large_master_chief.sql', '0001_quick_nighthawk.sql', '0002_silky_stellaris.sql'];
const migrations = [
'0000_large_master_chief.sql',
'0001_quick_nighthawk.sql',
'0002_silky_stellaris.sql'
];
for (const migrationFile of migrations) {
const migrationSql = readFileSync(join(migrationsFolder, migrationFile), 'utf-8');
const statements = migrationSql
@@ -123,9 +127,7 @@ function seedEmbedding(
// Mock EmbeddingProvider
// ---------------------------------------------------------------------------
function makeMockProvider(
returnValues: number[][] = [[1, 0, 0, 0]]
): EmbeddingProvider {
function makeMockProvider(returnValues: number[][] = [[1, 0, 0, 0]]): EmbeddingProvider {
return {
name: 'mock',
dimensions: returnValues[0]?.length ?? 4,
@@ -254,9 +256,18 @@ describe('reciprocalRankFusion', () => {
});
it('handles three lists correctly', () => {
const r1 = [{ id: 'a', score: 1 }, { id: 'b', score: 0 }];
const r2 = [{ id: 'b', score: 1 }, { id: 'c', score: 0 }];
const r3 = [{ id: 'a', score: 1 }, { id: 'c', score: 0 }];
const r1 = [
{ id: 'a', score: 1 },
{ id: 'b', score: 0 }
];
const r2 = [
{ id: 'b', score: 1 },
{ id: 'c', score: 0 }
];
const r3 = [
{ id: 'a', score: 1 },
{ id: 'c', score: 0 }
];
const result = reciprocalRankFusion(r1, r2, r3);
// 'a' appears first in r1 and r3 → higher combined score than 'b' or 'c'.
expect(result[0].id).toBe('a');

View File

@@ -103,10 +103,7 @@ export class HybridSearchService {
* @param options - Search parameters including repositoryId and alpha blend.
* @returns Ranked array of SnippetSearchResult, deduplicated by snippet ID.
*/
async search(
query: string,
options: HybridSearchOptions
): Promise<SnippetSearchResult[]> {
async search(query: string, options: HybridSearchOptions): Promise<SnippetSearchResult[]> {
const limit = options.limit ?? 20;
const mode = options.searchMode ?? 'auto';

View File

@@ -20,7 +20,7 @@
*/
export function preprocessQuery(raw: string): string {
// 1. Trim and collapse whitespace.
let q = raw.trim().replace(/\s+/g, ' ');
const q = raw.trim().replace(/\s+/g, ' ');
if (!q) return q;
@@ -91,7 +91,10 @@ export function preprocessQuery(raw: string): string {
}
// Remove trailing operators
while (finalTokens.length > 0 && ['AND', 'OR', 'NOT'].includes(finalTokens[finalTokens.length - 1])) {
while (
finalTokens.length > 0 &&
['AND', 'OR', 'NOT'].includes(finalTokens[finalTokens.length - 1])
) {
finalTokens.pop();
}

View File

@@ -32,9 +32,7 @@ export interface FusedItem {
* descending relevance (index 0 = most relevant).
* @returns Fused array sorted by descending rrfScore, deduplicated by id.
*/
export function reciprocalRankFusion(
...rankings: Array<Array<RankedItem>>
): Array<FusedItem> {
export function reciprocalRankFusion(...rankings: Array<Array<RankedItem>>): Array<FusedItem> {
const K = 60; // Standard RRF constant.
const scores = new Map<string, number>();

View File

@@ -674,7 +674,9 @@ describe('formatLibraryResults', () => {
id: '/facebook/react/v18',
repositoryId: '/facebook/react',
tag: 'v18',
title: 'React 18', commitHash: null, state: 'indexed',
title: 'React 18',
commitHash: null,
state: 'indexed',
totalSnippets: 1000,
indexedAt: null,
createdAt: now
@@ -731,7 +733,9 @@ describe('formatLibraryResults', () => {
describe('formatSnippetResults', () => {
const now = new Date();
function makeSnippetResult(overrides: Partial<Parameters<typeof formatSnippetResults>[0][number]> = {}): Parameters<typeof formatSnippetResults>[0][number] {
function makeSnippetResult(
overrides: Partial<Parameters<typeof formatSnippetResults>[0][number]> = {}
): Parameters<typeof formatSnippetResults>[0][number] {
return {
snippet: {
id: crypto.randomUUID(),

View File

@@ -87,10 +87,7 @@ export class SearchService {
if (!processedQuery) return [];
// Build the WHERE clause dynamically based on optional filters.
const conditions: string[] = [
'snippets_fts MATCH ?',
's.repository_id = ?'
];
const conditions: string[] = ['snippets_fts MATCH ?', 's.repository_id = ?'];
const params: unknown[] = [processedQuery, repositoryId];
if (versionId !== undefined) {
@@ -132,10 +129,14 @@ export class SearchService {
const rows = this.db.prepare(sql).all(...params) as RawSnippetRow[];
return rows.map((row) =>
SearchResultMapper.snippetFromEntity(new SnippetEntity(row), {
SearchResultMapper.snippetFromEntity(
new SnippetEntity(row),
{
id: row.repo_id,
title: row.repo_title
}, row.score)
},
row.score
)
);
}
@@ -188,7 +189,11 @@ export class SearchService {
return rows.map((row) => {
const compositeScore =
row.exact_match + row.prefix_match + row.desc_match + row.snippet_score + row.trust_component;
row.exact_match +
row.prefix_match +
row.desc_match +
row.snippet_score +
row.trust_component;
return SearchResultMapper.libraryFromEntity(
new RepositoryEntity(row),
this.getVersionEntities(row.id),
@@ -203,9 +208,7 @@ export class SearchService {
private getVersionEntities(repositoryId: string): RepositoryVersionEntity[] {
return this.db
.prepare(
`SELECT * FROM repository_versions WHERE repository_id = ? ORDER BY created_at DESC`
)
.prepare(`SELECT * FROM repository_versions WHERE repository_id = ? ORDER BY created_at DESC`)
.all(repositoryId) as RawVersionRow[];
}
}

View File

@@ -46,9 +46,7 @@ interface RawEmbeddingRow {
*/
export function cosineSimilarity(a: Float32Array, b: Float32Array): number {
if (a.length !== b.length) {
throw new Error(
`Embedding dimension mismatch: ${a.length} vs ${b.length}`
);
throw new Error(`Embedding dimension mismatch: ${a.length} vs ${b.length}`);
}
let dot = 0;

View File

@@ -27,10 +27,7 @@ function createTestDb(): Database.Database {
client.pragma('foreign_keys = ON');
const migrationsFolder = join(import.meta.dirname, '../db/migrations');
const migrationSql = readFileSync(
join(migrationsFolder, '0000_large_master_chief.sql'),
'utf-8'
);
const migrationSql = readFileSync(join(migrationsFolder, '0000_large_master_chief.sql'), 'utf-8');
// Drizzle migration files use `--> statement-breakpoint` as separator.
const statements = migrationSql
@@ -261,9 +258,7 @@ describe('RepositoryService.add()', () => {
});
it('throws InvalidInputError when sourceUrl is empty', () => {
expect(() =>
service.add({ source: 'github', sourceUrl: '' })
).toThrow(InvalidInputError);
expect(() => service.add({ source: 'github', sourceUrl: '' })).toThrow(InvalidInputError);
});
it('stores description and branch when provided', () => {
@@ -321,9 +316,7 @@ describe('RepositoryService.update()', () => {
});
it('throws NotFoundError for a non-existent repository', () => {
expect(() =>
service.update('/not/found', { title: 'New Title' })
).toThrow(NotFoundError);
expect(() => service.update('/not/found', { title: 'New Title' })).toThrow(NotFoundError);
});
});

View File

@@ -74,9 +74,7 @@ export class RepositoryService {
.get(state) as { n: number };
return row.n;
}
const row = this.db
.prepare(`SELECT COUNT(*) as n FROM repositories`)
.get() as { n: number };
const row = this.db.prepare(`SELECT COUNT(*) as n FROM repositories`).get() as { n: number };
return row.n;
}
@@ -115,13 +113,13 @@ export class RepositoryService {
}
// Default title from owner/repo
const parts = id.split('/').filter(Boolean);
title = input.title ?? (parts[1] ?? id);
title = input.title ?? parts[1] ?? id;
} else {
// local
const existing = this.list({ limit: 9999 }).map((r) => r.id);
id = resolveLocalId(input.sourceUrl, existing);
const parts = input.sourceUrl.split('/');
title = input.title ?? (parts.at(-1) ?? 'local-repo');
title = input.title ?? parts.at(-1) ?? 'local-repo';
}
// Check for collision

View File

@@ -25,14 +25,8 @@ function createTestDb(): Database.Database {
const migrationsFolder = join(import.meta.dirname, '../db/migrations');
// Apply all migration files in order
const migration0 = readFileSync(
join(migrationsFolder, '0000_large_master_chief.sql'),
'utf-8'
);
const migration1 = readFileSync(
join(migrationsFolder, '0001_quick_nighthawk.sql'),
'utf-8'
);
const migration0 = readFileSync(join(migrationsFolder, '0000_large_master_chief.sql'), 'utf-8');
const migration1 = readFileSync(join(migrationsFolder, '0001_quick_nighthawk.sql'), 'utf-8');
// Apply first migration
const statements0 = migration0
@@ -201,9 +195,7 @@ describe('VersionService.remove()', () => {
versionService.remove('/facebook/react', 'v18.3.0');
const doc = client
.prepare(`SELECT id FROM documents WHERE id = ?`)
.get(docId);
const doc = client.prepare(`SELECT id FROM documents WHERE id = ?`).get(docId);
expect(doc).toBeUndefined();
});
});

View File

@@ -40,12 +40,7 @@ export class VersionService {
* @throws NotFoundError when the parent repository does not exist
* @throws AlreadyExistsError when the tag is already registered
*/
add(
repositoryId: string,
tag: string,
title?: string,
commitHash?: string
): RepositoryVersion {
add(repositoryId: string, tag: string, title?: string, commitHash?: string): RepositoryVersion {
// Verify parent repository exists.
const repo = this.db
.prepare(`SELECT id, source, source_url FROM repositories WHERE id = ?`)
@@ -115,9 +110,7 @@ export class VersionService {
*/
getByTag(repositoryId: string, tag: string): RepositoryVersion | null {
const row = this.db
.prepare(
`SELECT * FROM repository_versions WHERE repository_id = ? AND tag = ?`
)
.prepare(`SELECT * FROM repository_versions WHERE repository_id = ? AND tag = ?`)
.get(repositoryId, tag) as RepositoryVersionEntity | undefined;
return row ? RepositoryVersionMapper.fromEntity(new RepositoryVersionEntity(row)) : null;
}
@@ -137,9 +130,9 @@ export class VersionService {
previousVersions: { tag: string; title: string; commitHash?: string }[]
): RepositoryVersion[] {
// Verify parent repository exists.
const repo = this.db
.prepare(`SELECT id FROM repositories WHERE id = ?`)
.get(repositoryId) as { id: string } | undefined;
const repo = this.db.prepare(`SELECT id FROM repositories WHERE id = ?`).get(repositoryId) as
| { id: string }
| undefined;
if (!repo) {
throw new NotFoundError(`Repository ${repositoryId} not found`);

View File

@@ -65,13 +65,10 @@ export function discoverVersionTags(options: DiscoverTagsOptions): string[] {
try {
// List all tags, sorted by commit date (newest first)
const output = execSync(
`git -C "${repoPath}" tag -l --sort=-creatordate`,
{
const output = execSync(`git -C "${repoPath}" tag -l --sort=-creatordate`, {
encoding: 'utf-8',
stdio: ['ignore', 'pipe', 'pipe']
}
).trim();
}).trim();
if (!output) return [];

View File

@@ -33,10 +33,11 @@ export function resolveLocalId(path: string, existingIds: string[]): string {
* Slugify a string to be safe for use in IDs.
*/
function slugify(str: string): string {
return str
return (
str
.toLowerCase()
.replace(/[^a-z0-9-_]/g, '-')
.replace(/-+/g, '-')
.replace(/^-|-$/g, '')
|| 'repo';
.replace(/^-|-$/g, '') || 'repo'
);
}

View File

@@ -59,13 +59,10 @@ export function errorResponse(
status: number,
details?: Record<string, unknown>
): Response {
return new Response(
JSON.stringify({ error, code, ...(details ? { details } : {}) }),
{
return new Response(JSON.stringify({ error, code, ...(details ? { details } : {}) }), {
status,
headers: { 'Content-Type': 'application/json' }
}
);
});
}
/**

View File

@@ -20,15 +20,9 @@ import { createServer } from 'node:http';
import { Server } from '@modelcontextprotocol/sdk/server/index.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import { StreamableHTTPServerTransport } from '@modelcontextprotocol/sdk/server/streamableHttp.js';
import {
CallToolRequestSchema,
ListToolsRequestSchema
} from '@modelcontextprotocol/sdk/types.js';
import { CallToolRequestSchema, ListToolsRequestSchema } from '@modelcontextprotocol/sdk/types.js';
import {
RESOLVE_LIBRARY_ID_TOOL,
handleResolveLibraryId
} from './tools/resolve-library-id.js';
import { RESOLVE_LIBRARY_ID_TOOL, handleResolveLibraryId } from './tools/resolve-library-id.js';
import { QUERY_DOCS_TOOL, handleQueryDocs } from './tools/query-docs.js';
// ---------------------------------------------------------------------------

View File

@@ -9,7 +9,9 @@ import { z } from 'zod';
import { searchLibraries } from '../client.js';
export const ResolveLibraryIdSchema = z.object({
libraryName: z.string().describe('Library name to search for and resolve to a TrueRef library ID'),
libraryName: z
.string()
.describe('Library name to search for and resolve to a TrueRef library ID'),
query: z.string().describe("The user's question or context to help rank results")
});

View File

@@ -150,7 +150,9 @@
{#if loading && jobs.length === 0}
<div class="flex items-center justify-center py-12">
<div class="text-center">
<div class="inline-block h-8 w-8 animate-spin rounded-full border-4 border-solid border-blue-600 border-r-transparent"></div>
<div
class="inline-block h-8 w-8 animate-spin rounded-full border-4 border-solid border-blue-600 border-r-transparent"
></div>
<p class="mt-2 text-gray-600">Loading jobs...</p>
</div>
</div>
@@ -160,26 +162,38 @@
</div>
{:else if jobs.length === 0}
<div class="rounded-md bg-gray-50 p-8 text-center">
<p class="text-gray-600">No jobs found. Jobs will appear here when repositories are indexed.</p>
<p class="text-gray-600">
No jobs found. Jobs will appear here when repositories are indexed.
</p>
</div>
{:else}
<div class="overflow-x-auto rounded-lg border border-gray-200 bg-white shadow">
<table class="min-w-full divide-y divide-gray-200">
<thead class="bg-gray-50">
<tr>
<th class="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider text-gray-500">
<th
class="px-6 py-3 text-left text-xs font-medium tracking-wider text-gray-500 uppercase"
>
Repository
</th>
<th class="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider text-gray-500">
<th
class="px-6 py-3 text-left text-xs font-medium tracking-wider text-gray-500 uppercase"
>
Status
</th>
<th class="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider text-gray-500">
<th
class="px-6 py-3 text-left text-xs font-medium tracking-wider text-gray-500 uppercase"
>
Progress
</th>
<th class="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider text-gray-500">
<th
class="px-6 py-3 text-left text-xs font-medium tracking-wider text-gray-500 uppercase"
>
Created
</th>
<th class="px-6 py-3 text-right text-xs font-medium uppercase tracking-wider text-gray-500">
<th
class="px-6 py-3 text-right text-xs font-medium tracking-wider text-gray-500 uppercase"
>
Actions
</th>
</tr>
@@ -187,16 +201,16 @@
<tbody class="divide-y divide-gray-200 bg-white">
{#each jobs as job (job.id)}
<tr class="hover:bg-gray-50">
<td class="whitespace-nowrap px-6 py-4 text-sm font-medium text-gray-900">
<td class="px-6 py-4 text-sm font-medium whitespace-nowrap text-gray-900">
{job.repositoryId}
{#if job.versionId}
<span class="ml-1 text-xs text-gray-500">@{job.versionId}</span>
{/if}
</td>
<td class="whitespace-nowrap px-6 py-4 text-sm text-gray-500">
<td class="px-6 py-4 text-sm whitespace-nowrap text-gray-500">
<JobStatusBadge status={job.status} />
</td>
<td class="whitespace-nowrap px-6 py-4 text-sm text-gray-500">
<td class="px-6 py-4 text-sm whitespace-nowrap text-gray-500">
<div class="flex items-center">
<span class="mr-2">{job.progress}%</span>
<div class="h-2 w-32 rounded-full bg-gray-200">
@@ -212,10 +226,10 @@
{/if}
</div>
</td>
<td class="whitespace-nowrap px-6 py-4 text-sm text-gray-500">
<td class="px-6 py-4 text-sm whitespace-nowrap text-gray-500">
{formatDate(job.createdAt)}
</td>
<td class="whitespace-nowrap px-6 py-4 text-right text-sm font-medium">
<td class="px-6 py-4 text-right text-sm font-medium whitespace-nowrap">
<div class="flex justify-end gap-2">
{#if canPause(job.status)}
<button
@@ -256,9 +270,7 @@
</div>
{#if loading}
<div class="mt-4 text-center text-sm text-gray-500">
Refreshing...
</div>
<div class="mt-4 text-center text-sm text-gray-500">Refreshing...</div>
{/if}
{/if}
</div>

View File

@@ -20,11 +20,7 @@ import { createProviderFromProfile } from '$lib/server/embeddings/registry';
import type { EmbeddingProfile } from '$lib/server/db/schema';
import { parseLibraryId } from '$lib/server/api/library-id';
import { selectSnippetsWithinBudget, DEFAULT_TOKEN_BUDGET } from '$lib/server/api/token-budget';
import {
formatContextJson,
formatContextTxt,
CORS_HEADERS
} from '$lib/server/api/formatters';
import { formatContextJson, formatContextTxt, CORS_HEADERS } from '$lib/server/api/formatters';
import type { ContextResponseMetadata } from '$lib/server/mappers/context-response.mapper';
// ---------------------------------------------------------------------------
@@ -36,9 +32,10 @@ function getServices(db: ReturnType<typeof getClient>) {
// Load the active embedding profile from the database
const profileRow = db
.prepare<[], EmbeddingProfile>(
'SELECT * FROM embedding_profiles WHERE is_default = 1 AND enabled = 1 LIMIT 1'
)
.prepare<
[],
EmbeddingProfile
>('SELECT * FROM embedding_profiles WHERE is_default = 1 AND enabled = 1 LIMIT 1')
.get();
const provider = profileRow ? createProviderFromProfile(profileRow) : null;
@@ -53,7 +50,10 @@ interface RawRepoConfig {
function getRules(db: ReturnType<typeof getClient>, repositoryId: string): string[] {
const row = db
.prepare<[string], RawRepoConfig>(`SELECT rules FROM repository_configs WHERE repository_id = ?`)
.prepare<
[string],
RawRepoConfig
>(`SELECT rules FROM repository_configs WHERE repository_id = ?`)
.get(repositoryId);
if (!row?.rules) return [];
@@ -88,9 +88,10 @@ function getSnippetVersionTags(
const placeholders = versionIds.map(() => '?').join(', ');
const rows = db
.prepare<string[], RawVersionRow>(
`SELECT id, tag FROM repository_versions WHERE id IN (${placeholders})`
)
.prepare<
string[],
RawVersionRow
>(`SELECT id, tag FROM repository_versions WHERE id IN (${placeholders})`)
.all(...versionIds);
return Object.fromEntries(rows.map((row) => [row.id, row.tag]));
@@ -116,13 +117,10 @@ export const GET: RequestHandler = async ({ url }) => {
const query = url.searchParams.get('query');
if (!query || !query.trim()) {
return new Response(
JSON.stringify({ error: 'query is required', code: 'MISSING_PARAMETER' }),
{
return new Response(JSON.stringify({ error: 'query is required', code: 'MISSING_PARAMETER' }), {
status: 400,
headers: { 'Content-Type': 'application/json', ...CORS_HEADERS }
}
);
});
}
const responseType = url.searchParams.get('type') ?? 'json';
@@ -157,9 +155,10 @@ export const GET: RequestHandler = async ({ url }) => {
// Verify the repository exists and check its state.
const repo = db
.prepare<[string], RawRepoState>(
`SELECT id, state, title, source, source_url, branch FROM repositories WHERE id = ?`
)
.prepare<
[string],
RawRepoState
>(`SELECT id, state, title, source, source_url, branch FROM repositories WHERE id = ?`)
.get(parsed.repositoryId);
if (!repo) {
@@ -193,9 +192,10 @@ export const GET: RequestHandler = async ({ url }) => {
let resolvedVersion: RawVersionRow | undefined;
if (parsed.version) {
resolvedVersion = db
.prepare<[string, string], RawVersionRow>(
`SELECT id, tag FROM repository_versions WHERE repository_id = ? AND tag = ?`
)
.prepare<
[string, string],
RawVersionRow
>(`SELECT id, tag FROM repository_versions WHERE repository_id = ? AND tag = ?`)
.get(parsed.repositoryId, parsed.version);
// Version not found is not fatal — fall back to default branch.
@@ -240,7 +240,8 @@ export const GET: RequestHandler = async ({ url }) => {
sourceUrl: repo.source_url,
branch: repo.branch
},
version: parsed.version || resolvedVersion
version:
parsed.version || resolvedVersion
? {
requested: parsed.version ?? null,
resolved: resolvedVersion?.tag ?? null,

View File

@@ -10,7 +10,7 @@ export const GET: RequestHandler = ({ url }) => {
let entries: { name: string; path: string; isGitRepo: boolean }[] = [];
let error: string | null = null;
let resolved = target;
const resolved = target;
try {
const items = fs.readdirSync(target, { withFileTypes: true });

View File

@@ -7,7 +7,11 @@ import type { RequestHandler } from './$types';
import { getClient } from '$lib/server/db/client.js';
import { IndexingJobMapper } from '$lib/server/mappers/indexing-job.mapper.js';
import { JobQueue } from '$lib/server/pipeline/job-queue.js';
import { handleServiceError, NotFoundError, InvalidInputError } from '$lib/server/utils/validation.js';
import {
handleServiceError,
NotFoundError,
InvalidInputError
} from '$lib/server/utils/validation.js';
export const POST: RequestHandler = ({ params }) => {
try {
@@ -19,9 +23,7 @@ export const POST: RequestHandler = ({ params }) => {
const success = queue.cancelJob(params.id);
if (!success) {
throw new InvalidInputError(
`Cannot cancel job ${params.id} - job is already done or failed`
);
throw new InvalidInputError(`Cannot cancel job ${params.id} - job is already done or failed`);
}
// Fetch updated job

View File

@@ -7,7 +7,11 @@ import type { RequestHandler } from './$types';
import { getClient } from '$lib/server/db/client.js';
import { IndexingJobMapper } from '$lib/server/mappers/indexing-job.mapper.js';
import { JobQueue } from '$lib/server/pipeline/job-queue.js';
import { handleServiceError, NotFoundError, InvalidInputError } from '$lib/server/utils/validation.js';
import {
handleServiceError,
NotFoundError,
InvalidInputError
} from '$lib/server/utils/validation.js';
export const POST: RequestHandler = ({ params }) => {
try {

View File

@@ -7,7 +7,11 @@ import type { RequestHandler } from './$types';
import { getClient } from '$lib/server/db/client.js';
import { IndexingJobMapper } from '$lib/server/mappers/indexing-job.mapper.js';
import { JobQueue } from '$lib/server/pipeline/job-queue.js';
import { handleServiceError, NotFoundError, InvalidInputError } from '$lib/server/utils/validation.js';
import {
handleServiceError,
NotFoundError,
InvalidInputError
} from '$lib/server/utils/validation.js';
export const POST: RequestHandler = ({ params }) => {
try {
@@ -19,7 +23,9 @@ export const POST: RequestHandler = ({ params }) => {
const success = queue.resumeJob(params.id);
if (!success) {
throw new InvalidInputError(`Cannot resume job ${params.id} - only paused jobs can be resumed`);
throw new InvalidInputError(
`Cannot resume job ${params.id} - only paused jobs can be resumed`
);
}
// Fetch updated job

View File

@@ -58,9 +58,7 @@ export const POST: RequestHandler = async ({ request }) => {
let jobResponse: ReturnType<typeof IndexingJobMapper.toDto> | null = null;
if (body.autoIndex !== false) {
const queue = getQueue();
const job = queue
? queue.enqueue(repo.id)
: service.createIndexingJob(repo.id);
const job = queue ? queue.enqueue(repo.id) : service.createIndexingJob(repo.id);
jobResponse = IndexingJobMapper.toDto(job);
}

View File

@@ -28,9 +28,7 @@ export const POST: RequestHandler = async ({ params, request }) => {
// Use the queue so processNext() is triggered immediately.
// Falls back to direct DB insert if the queue isn't initialised yet.
const queue = getQueue();
const job = queue
? queue.enqueue(id, versionId)
: service.createIndexingJob(id, versionId);
const job = queue ? queue.enqueue(id, versionId) : service.createIndexingJob(id, versionId);
return json({ job: IndexingJobMapper.toDto(job) }, { status: 202 });
} catch (err) {

View File

@@ -146,4 +146,3 @@ function sanitizeProfile(profile: EmbeddingProfile): EmbeddingProfile {
}
return profile;
}

View File

@@ -16,9 +16,10 @@ export const GET: RequestHandler = async () => {
try {
const db = getClient();
const profile = db
.prepare<[], EmbeddingProfile>(
'SELECT * FROM embedding_profiles WHERE is_default = 1 AND enabled = 1 LIMIT 1'
)
.prepare<
[],
EmbeddingProfile
>('SELECT * FROM embedding_profiles WHERE is_default = 1 AND enabled = 1 LIMIT 1')
.get();
if (!profile) {
@@ -42,7 +43,6 @@ export const GET: RequestHandler = async () => {
}
};
export const POST: RequestHandler = async ({ request }) => {
try {
const body = await request.json();

View File

@@ -8,7 +8,9 @@
let { data }: { data: PageData } = $props();
// Initialized empty; $effect syncs from data prop on every navigation/reload.
let repo = $state<Repository & { versions?: RepositoryVersion[] }>({} as Repository & { versions?: RepositoryVersion[] });
let repo = $state<Repository & { versions?: RepositoryVersion[] }>(
{} as Repository & { versions?: RepositoryVersion[] }
);
let recentJobs = $state<IndexingJob[]>([]);
$effect(() => {
if (data.repo) repo = data.repo;
@@ -189,7 +191,7 @@
<dl class="grid grid-cols-1 gap-y-2 text-sm sm:grid-cols-2">
<div class="flex gap-2">
<dt class="text-gray-500">Source</dt>
<dd class="font-medium capitalize text-gray-900">{repo.source}</dd>
<dd class="font-medium text-gray-900 capitalize">{repo.source}</dd>
</div>
<div class="flex gap-2">
<dt class="text-gray-500">Branch</dt>

View File

@@ -227,7 +227,9 @@
<div class="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
<div class="mb-4 flex items-center gap-3">
<div class="flex min-w-0 flex-1 items-center gap-2">
<span class="shrink-0 rounded bg-green-100 px-2 py-0.5 text-xs font-medium text-green-700">
<span
class="shrink-0 rounded bg-green-100 px-2 py-0.5 text-xs font-medium text-green-700"
>
Selected
</span>
<span class="truncate font-mono text-sm text-gray-700">{selectedLibraryTitle}</span>
@@ -285,7 +287,9 @@
{:else if query && !loadingSnippets && snippets.length === 0 && !snippetError}
<div class="flex flex-col items-center py-16 text-center">
<p class="text-sm text-gray-500">No snippets found for that query.</p>
<p class="mt-1 text-xs text-gray-400">Try a different question or select another library.</p>
<p class="mt-1 text-xs text-gray-400">
Try a different question or select another library.
</p>
</div>
{/if}
{/if}

View File

@@ -203,7 +203,11 @@
: 'border border-gray-200 text-gray-700 hover:bg-gray-50'
].join(' ')}
>
{p === 'none' ? 'None (FTS5 only)' : p === 'openai' ? 'OpenAI-compatible' : 'Local Model'}
{p === 'none'
? 'None (FTS5 only)'
: p === 'openai'
? 'OpenAI-compatible'
: 'Local Model'}
</button>
{/each}
</div>