chore(FEEDBACK-0001): linting
This commit is contained in:
18
README.md
18
README.md
@@ -602,6 +602,7 @@ alwaysApply: true
|
||||
---
|
||||
|
||||
When answering questions about indexed libraries, always use the TrueRef MCP tools:
|
||||
|
||||
1. Call `resolve-library-id` with the library name and the user's question to get the library ID.
|
||||
2. Call `query-docs` with the library ID and question to retrieve relevant documentation.
|
||||
3. Use the returned documentation to answer accurately.
|
||||
@@ -639,7 +640,7 @@ docker compose up --build
|
||||
This builds the image and starts two services:
|
||||
|
||||
| Service | Default port | Purpose |
|
||||
|---------|-------------|---------|
|
||||
| ------- | ------------ | ----------------------------- |
|
||||
| `web` | `3000` | SvelteKit web UI and REST API |
|
||||
| `mcp` | `3001` | MCP HTTP server |
|
||||
|
||||
@@ -687,10 +688,10 @@ services:
|
||||
- ${USERPROFILE:-$HOME}/.gitconfig:/root/.gitconfig:ro
|
||||
- ${CORP_CA_CERT}:/certs/corp-ca.crt:ro
|
||||
environment:
|
||||
BITBUCKET_HOST: "${BITBUCKET_HOST}"
|
||||
GITLAB_HOST: "${GITLAB_HOST}"
|
||||
GIT_TOKEN_BITBUCKET: "${GIT_TOKEN_BITBUCKET}"
|
||||
GIT_TOKEN_GITLAB: "${GIT_TOKEN_GITLAB}"
|
||||
BITBUCKET_HOST: '${BITBUCKET_HOST}'
|
||||
GITLAB_HOST: '${GITLAB_HOST}'
|
||||
GIT_TOKEN_BITBUCKET: '${GIT_TOKEN_BITBUCKET}'
|
||||
GIT_TOKEN_GITLAB: '${GIT_TOKEN_GITLAB}'
|
||||
```
|
||||
|
||||
5. **Start the services**:
|
||||
@@ -708,6 +709,7 @@ The Docker entrypoint script (`docker-entrypoint.sh`) runs these steps in order:
|
||||
3. **Configure git credentials**: Sets up per-host credential helpers that provide the correct username and token for each remote.
|
||||
|
||||
This setup works for:
|
||||
|
||||
- HTTPS cloning with personal access tokens
|
||||
- SSH cloning with mounted SSH keys
|
||||
- On-premise servers with custom CA certificates
|
||||
@@ -718,6 +720,7 @@ This setup works for:
|
||||
For long-lived deployments, SSH authentication is recommended:
|
||||
|
||||
1. Generate an SSH key pair if you don't have one:
|
||||
|
||||
```sh
|
||||
ssh-keygen -t ed25519 -C "trueref@your-company.com"
|
||||
```
|
||||
@@ -727,6 +730,7 @@ For long-lived deployments, SSH authentication is recommended:
|
||||
- GitLab: User Settings → SSH Keys
|
||||
|
||||
3. Ensure your `~/.ssh/config` has the correct host entries:
|
||||
|
||||
```
|
||||
Host bitbucket.corp.example.com
|
||||
IdentityFile ~/.ssh/id_ed25519
|
||||
@@ -738,7 +742,7 @@ For long-lived deployments, SSH authentication is recommended:
|
||||
### Environment variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| ----------------- | ----------------------- | -------------------------------------------------- |
|
||||
| `DATABASE_URL` | `/data/trueref.db` | Path to the SQLite database inside the container |
|
||||
| `PORT` | `3000` | Port the web app listens on |
|
||||
| `HOST` | `0.0.0.0` | Bind address for the web app |
|
||||
@@ -807,7 +811,7 @@ Verify the connection inside Claude Code:
|
||||
### Health checks
|
||||
|
||||
| Endpoint | Expected response |
|
||||
|----------|------------------|
|
||||
| ----------------------------------- | ------------------------------- |
|
||||
| `http://localhost:3000/api/v1/libs` | JSON array of indexed libraries |
|
||||
| `http://localhost:3001/ping` | `{"ok":true}` |
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@ services:
|
||||
web:
|
||||
build: .
|
||||
ports:
|
||||
- "3000:3000"
|
||||
- '3000:3000'
|
||||
volumes:
|
||||
- trueref-data:/data
|
||||
# Corporate deployment support (TRUEREF-0019)
|
||||
@@ -24,10 +24,10 @@ services:
|
||||
build: .
|
||||
command: mcp
|
||||
ports:
|
||||
- "3001:3001"
|
||||
- '3001:3001'
|
||||
environment:
|
||||
TRUEREF_API_URL: http://web:3000
|
||||
MCP_PORT: "3001"
|
||||
MCP_PORT: '3001'
|
||||
depends_on:
|
||||
- web
|
||||
restart: unless-stopped
|
||||
|
||||
37
docs/PRD.md
37
docs/PRD.md
@@ -17,6 +17,7 @@ The core use case is enabling AI coding assistants (Claude Code, Cursor, Zed, et
|
||||
## 2. Problem Statement
|
||||
|
||||
### 2.1 Context7's Limitations
|
||||
|
||||
- The indexing and crawling backend is entirely private and closed-source.
|
||||
- Only public libraries already in the context7.com catalog are available.
|
||||
- Private, internal, or niche repositories cannot be added.
|
||||
@@ -24,6 +25,7 @@ The core use case is enabling AI coding assistants (Claude Code, Cursor, Zed, et
|
||||
- No way to self-host for air-gapped or compliance-constrained environments.
|
||||
|
||||
### 2.2 The Gap
|
||||
|
||||
Teams with internal SDKs, private libraries, proprietary documentation, or a need for data sovereignty have no tooling that provides context7-equivalent LLM documentation retrieval.
|
||||
|
||||
---
|
||||
@@ -31,6 +33,7 @@ Teams with internal SDKs, private libraries, proprietary documentation, or a nee
|
||||
## 3. Goals & Non-Goals
|
||||
|
||||
### Goals
|
||||
|
||||
- Replicate all context7 capabilities: library search, documentation retrieval, MCP tools (`resolve-library-id`, `query-docs`).
|
||||
- Support both GitHub-hosted and local filesystem repositories.
|
||||
- Provide a full indexing pipeline: crawl → parse → chunk → embed → store → query.
|
||||
@@ -42,6 +45,7 @@ Teams with internal SDKs, private libraries, proprietary documentation, or a nee
|
||||
- Self-hostable with minimal dependencies (SQLite-first, no external vector DB required).
|
||||
|
||||
### Non-Goals (v1)
|
||||
|
||||
- Authentication & authorization (deferred to a future version).
|
||||
- Skill generation (context7 CLI skill feature).
|
||||
- Multi-tenant SaaS mode.
|
||||
@@ -54,9 +58,11 @@ Teams with internal SDKs, private libraries, proprietary documentation, or a nee
|
||||
## 4. Users & Personas
|
||||
|
||||
### Primary: The Developer / Tech Lead
|
||||
|
||||
Configures TrueRef, adds repositories, integrates the MCP server with their AI coding assistant. Technical, comfortable with CLI and config files.
|
||||
|
||||
### Secondary: The AI Coding Assistant
|
||||
|
||||
The "user" at query time. Calls `resolve-library-id` and `query-docs` via MCP to retrieve documentation snippets for code generation.
|
||||
|
||||
---
|
||||
@@ -100,8 +106,9 @@ The "user" at query time. Calls `resolve-library-id` and `query-docs` via MCP to
|
||||
```
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Technology |
|
||||
|-------|-----------|
|
||||
| ---------------- | ------------------------------------------------------------------ |
|
||||
| Framework | SvelteKit (Node adapter) |
|
||||
| Language | TypeScript |
|
||||
| Database | SQLite via better-sqlite3 + drizzle-orm |
|
||||
@@ -119,6 +126,7 @@ The "user" at query time. Calls `resolve-library-id` and `query-docs` via MCP to
|
||||
## 6. Data Model
|
||||
|
||||
### 6.1 Repositories
|
||||
|
||||
A `Repository` is the top-level entity. It maps to a GitHub repo or local directory.
|
||||
|
||||
```
|
||||
@@ -141,6 +149,7 @@ Repository {
|
||||
```
|
||||
|
||||
### 6.2 Repository Versions
|
||||
|
||||
```
|
||||
RepositoryVersion {
|
||||
id TEXT PRIMARY KEY
|
||||
@@ -153,6 +162,7 @@ RepositoryVersion {
|
||||
```
|
||||
|
||||
### 6.3 Documents (parsed files)
|
||||
|
||||
```
|
||||
Document {
|
||||
id TEXT PRIMARY KEY
|
||||
@@ -169,6 +179,7 @@ Document {
|
||||
```
|
||||
|
||||
### 6.4 Snippets (indexed chunks)
|
||||
|
||||
```
|
||||
Snippet {
|
||||
id TEXT PRIMARY KEY
|
||||
@@ -186,6 +197,7 @@ Snippet {
|
||||
```
|
||||
|
||||
### 6.5 Indexing Jobs
|
||||
|
||||
```
|
||||
IndexingJob {
|
||||
id TEXT PRIMARY KEY
|
||||
@@ -203,6 +215,7 @@ IndexingJob {
|
||||
```
|
||||
|
||||
### 6.6 Repository Configuration (`trueref.json`)
|
||||
|
||||
```
|
||||
RepositoryConfig {
|
||||
repositoryId TEXT FK → Repository
|
||||
@@ -221,15 +234,19 @@ RepositoryConfig {
|
||||
## 7. Core Features
|
||||
|
||||
### F1: Repository Management
|
||||
|
||||
Add, remove, update, and list repositories. Support GitHub (public/private via token) and local filesystem sources. Trigger indexing on demand or on schedule.
|
||||
|
||||
### F2: GitHub Crawler
|
||||
|
||||
Fetch repository file trees via GitHub Trees API. Download file contents. Respect `trueref.json` include/exclude rules. Support rate limiting and incremental re-indexing (checksum-based).
|
||||
|
||||
### F3: Local Filesystem Crawler
|
||||
|
||||
Walk directory trees. Apply include/exclude rules from `trueref.json`. Watch for file changes (optional).
|
||||
|
||||
### F4: Document Parser & Chunker
|
||||
|
||||
- Parse Markdown files into sections (heading-based splitting).
|
||||
- Extract code blocks from Markdown.
|
||||
- Parse standalone code files into function/class-level chunks.
|
||||
@@ -237,16 +254,19 @@ Walk directory trees. Apply include/exclude rules from `trueref.json`. Watch for
|
||||
- Produce structured `Snippet` records (type: "code" or "info").
|
||||
|
||||
### F5: Embedding & Vector Storage
|
||||
|
||||
- Generate embeddings for each snippet using a pluggable embeddings backend.
|
||||
- Store embeddings as binary blobs in SQLite (sqlite-vec).
|
||||
- Support fallback to FTS5-only search when no embedding provider is configured.
|
||||
|
||||
### F6: Semantic Search Engine
|
||||
|
||||
- Hybrid search: vector similarity + FTS5 keyword matching (BM25) with reciprocal rank fusion.
|
||||
- Query-time retrieval: given `libraryId + query`, return ranked snippets.
|
||||
- Library search: given `libraryName + query`, return matching repositories.
|
||||
|
||||
### F7: REST API (`/api/v1/*`)
|
||||
|
||||
- `GET /api/v1/libs/search?query=&libraryName=` — search libraries (context7-compatible)
|
||||
- `GET /api/v1/context?query=&libraryId=&type=json|txt` — fetch documentation
|
||||
- `GET /api/v1/libs` — list all indexed libraries
|
||||
@@ -256,12 +276,14 @@ Walk directory trees. Apply include/exclude rules from `trueref.json`. Watch for
|
||||
- `GET /api/v1/jobs/:id` — get indexing job status
|
||||
|
||||
### F8: MCP Server
|
||||
|
||||
- Tool: `resolve-library-id` — search for libraries by name
|
||||
- Tool: `query-docs` — fetch documentation by libraryId + query
|
||||
- Transport: stdio (primary), HTTP (optional)
|
||||
- Compatible with Claude Code, Cursor, and other MCP-aware tools
|
||||
|
||||
### F9: Web UI — Repository Dashboard
|
||||
|
||||
- List all repositories with status, snippet count, last indexed date
|
||||
- Add/remove repositories (GitHub URL or local path)
|
||||
- Trigger re-indexing
|
||||
@@ -269,23 +291,27 @@ Walk directory trees. Apply include/exclude rules from `trueref.json`. Watch for
|
||||
- View repository config (`trueref.json`)
|
||||
|
||||
### F10: Web UI — Search Explorer
|
||||
|
||||
- Interactive search interface (resolve library → query docs)
|
||||
- Preview snippets with syntax highlighting
|
||||
- View raw document content
|
||||
|
||||
### F11: `trueref.json` Config Support
|
||||
|
||||
- Parse `trueref.json` from repo root (or `context7.json` for compatibility)
|
||||
- Apply `folders`, `excludeFolders`, `excludeFiles` during crawling
|
||||
- Inject `rules` into LLM context alongside snippets
|
||||
- Support `previousVersions` for versioned documentation
|
||||
|
||||
### F12: Indexing Pipeline & Job Queue
|
||||
|
||||
- SQLite-backed job queue (no external message broker required)
|
||||
- Sequential processing with progress tracking
|
||||
- Error recovery and retry logic
|
||||
- Incremental re-indexing using file checksums
|
||||
|
||||
### F13: Version Support
|
||||
|
||||
- Index specific git tags/branches per repository
|
||||
- Serve version-specific context when libraryId includes version (`/owner/repo/v1.2.3`)
|
||||
- UI for managing available versions
|
||||
@@ -297,11 +323,12 @@ Walk directory trees. Apply include/exclude rules from `trueref.json`. Watch for
|
||||
TrueRef's REST API mirrors context7's `/api/v2/*` interface to allow drop-in compatibility:
|
||||
|
||||
| context7 Endpoint | TrueRef Endpoint | Notes |
|
||||
|-------------------|-----------------|-------|
|
||||
| ------------------------- | ------------------------- | -------------------------------------- |
|
||||
| `GET /api/v2/libs/search` | `GET /api/v1/libs/search` | Same query params |
|
||||
| `GET /api/v2/context` | `GET /api/v1/context` | Same query params, same response shape |
|
||||
|
||||
The MCP tool names and input schemas are identical:
|
||||
|
||||
- `resolve-library-id` with `libraryName` + `query`
|
||||
- `query-docs` with `libraryId` + `query`
|
||||
|
||||
@@ -312,20 +339,24 @@ Library IDs follow the same convention: `/owner/repo` or `/owner/repo/version`.
|
||||
## 9. Non-Functional Requirements
|
||||
|
||||
### Performance
|
||||
|
||||
- Library search: < 200ms p99
|
||||
- Documentation retrieval: < 500ms p99 for 20 snippets
|
||||
- Indexing throughput: > 1,000 files/minute (GitHub API rate-limited)
|
||||
|
||||
### Reliability
|
||||
|
||||
- Failed indexing jobs must not corrupt existing indexed data
|
||||
- Atomic snippet replacement during re-indexing
|
||||
|
||||
### Portability
|
||||
|
||||
- Single SQLite file for all data
|
||||
- Runs on Linux, macOS, Windows (Node.js 20+)
|
||||
- No required external services beyond optional embedding API
|
||||
|
||||
### Scalability (v1 constraints)
|
||||
|
||||
- Designed for single-node deployment
|
||||
- SQLite suitable for up to ~500 repositories, ~500k snippets
|
||||
|
||||
@@ -334,7 +365,7 @@ Library IDs follow the same convention: `/owner/repo` or `/owner/repo/version`.
|
||||
## 10. Milestones & Feature Order
|
||||
|
||||
| ID | Feature | Priority | Depends On |
|
||||
|----|---------|----------|-----------|
|
||||
| ------------ | ---------------------------------------- | -------- | -------------------------- |
|
||||
| TRUEREF-0001 | Database schema & core data models | P0 | — |
|
||||
| TRUEREF-0002 | Repository management service & REST API | P0 | TRUEREF-0001 |
|
||||
| TRUEREF-0003 | GitHub repository crawler | P0 | TRUEREF-0001 |
|
||||
|
||||
@@ -39,7 +39,9 @@ export const repositories = sqliteTable('repositories', {
|
||||
branch: text('branch').default('main'),
|
||||
state: text('state', {
|
||||
enum: ['pending', 'indexing', 'indexed', 'error']
|
||||
}).notNull().default('pending'),
|
||||
})
|
||||
.notNull()
|
||||
.default('pending'),
|
||||
totalSnippets: integer('total_snippets').default(0),
|
||||
totalTokens: integer('total_tokens').default(0),
|
||||
trustScore: real('trust_score').default(0), // 0.0–10.0
|
||||
@@ -48,7 +50,7 @@ export const repositories = sqliteTable('repositories', {
|
||||
githubToken: text('github_token'), // encrypted PAT for private repos
|
||||
lastIndexedAt: integer('last_indexed_at', { mode: 'timestamp' }),
|
||||
createdAt: integer('created_at', { mode: 'timestamp' }).notNull(),
|
||||
updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull(),
|
||||
updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull()
|
||||
});
|
||||
```
|
||||
|
||||
@@ -59,16 +61,19 @@ Tracks indexed git tags/branches beyond the default branch.
|
||||
```typescript
|
||||
export const repositoryVersions = sqliteTable('repository_versions', {
|
||||
id: text('id').primaryKey(), // e.g. "/facebook/react/v18.3.0"
|
||||
repositoryId: text('repository_id').notNull()
|
||||
repositoryId: text('repository_id')
|
||||
.notNull()
|
||||
.references(() => repositories.id, { onDelete: 'cascade' }),
|
||||
tag: text('tag').notNull(), // git tag or branch name
|
||||
title: text('title'),
|
||||
state: text('state', {
|
||||
enum: ['pending', 'indexing', 'indexed', 'error']
|
||||
}).notNull().default('pending'),
|
||||
})
|
||||
.notNull()
|
||||
.default('pending'),
|
||||
totalSnippets: integer('total_snippets').default(0),
|
||||
indexedAt: integer('indexed_at', { mode: 'timestamp' }),
|
||||
createdAt: integer('created_at', { mode: 'timestamp' }).notNull(),
|
||||
createdAt: integer('created_at', { mode: 'timestamp' }).notNull()
|
||||
});
|
||||
```
|
||||
|
||||
@@ -79,16 +84,16 @@ A parsed source file within a repository.
|
||||
```typescript
|
||||
export const documents = sqliteTable('documents', {
|
||||
id: text('id').primaryKey(), // UUID
|
||||
repositoryId: text('repository_id').notNull()
|
||||
repositoryId: text('repository_id')
|
||||
.notNull()
|
||||
.references(() => repositories.id, { onDelete: 'cascade' }),
|
||||
versionId: text('version_id')
|
||||
.references(() => repositoryVersions.id, { onDelete: 'cascade' }),
|
||||
versionId: text('version_id').references(() => repositoryVersions.id, { onDelete: 'cascade' }),
|
||||
filePath: text('file_path').notNull(), // relative path within repo
|
||||
title: text('title'),
|
||||
language: text('language'), // e.g. "typescript", "markdown"
|
||||
tokenCount: integer('token_count').default(0),
|
||||
checksum: text('checksum').notNull(), // SHA-256 of file content
|
||||
indexedAt: integer('indexed_at', { mode: 'timestamp' }).notNull(),
|
||||
indexedAt: integer('indexed_at', { mode: 'timestamp' }).notNull()
|
||||
});
|
||||
```
|
||||
|
||||
@@ -99,19 +104,20 @@ An indexed chunk of content, the atomic unit of search.
|
||||
```typescript
|
||||
export const snippets = sqliteTable('snippets', {
|
||||
id: text('id').primaryKey(), // UUID
|
||||
documentId: text('document_id').notNull()
|
||||
documentId: text('document_id')
|
||||
.notNull()
|
||||
.references(() => documents.id, { onDelete: 'cascade' }),
|
||||
repositoryId: text('repository_id').notNull()
|
||||
repositoryId: text('repository_id')
|
||||
.notNull()
|
||||
.references(() => repositories.id, { onDelete: 'cascade' }),
|
||||
versionId: text('version_id')
|
||||
.references(() => repositoryVersions.id, { onDelete: 'cascade' }),
|
||||
versionId: text('version_id').references(() => repositoryVersions.id, { onDelete: 'cascade' }),
|
||||
type: text('type', { enum: ['code', 'info'] }).notNull(),
|
||||
title: text('title'),
|
||||
content: text('content').notNull(), // searchable text / code
|
||||
language: text('language'),
|
||||
breadcrumb: text('breadcrumb'), // e.g. "Installation > Getting Started"
|
||||
tokenCount: integer('token_count').default(0),
|
||||
createdAt: integer('created_at', { mode: 'timestamp' }).notNull(),
|
||||
createdAt: integer('created_at', { mode: 'timestamp' }).notNull()
|
||||
});
|
||||
```
|
||||
|
||||
@@ -121,12 +127,13 @@ Stores vector embeddings separately to keep snippets table lean.
|
||||
|
||||
```typescript
|
||||
export const snippetEmbeddings = sqliteTable('snippet_embeddings', {
|
||||
snippetId: text('snippet_id').primaryKey()
|
||||
snippetId: text('snippet_id')
|
||||
.primaryKey()
|
||||
.references(() => snippets.id, { onDelete: 'cascade' }),
|
||||
model: text('model').notNull(), // embedding model identifier
|
||||
dimensions: integer('dimensions').notNull(),
|
||||
embedding: blob('embedding').notNull(), // Float32Array as binary blob
|
||||
createdAt: integer('created_at', { mode: 'timestamp' }).notNull(),
|
||||
createdAt: integer('created_at', { mode: 'timestamp' }).notNull()
|
||||
});
|
||||
```
|
||||
|
||||
@@ -137,19 +144,22 @@ Tracks asynchronous indexing operations.
|
||||
```typescript
|
||||
export const indexingJobs = sqliteTable('indexing_jobs', {
|
||||
id: text('id').primaryKey(), // UUID
|
||||
repositoryId: text('repository_id').notNull()
|
||||
repositoryId: text('repository_id')
|
||||
.notNull()
|
||||
.references(() => repositories.id, { onDelete: 'cascade' }),
|
||||
versionId: text('version_id'),
|
||||
status: text('status', {
|
||||
enum: ['queued', 'running', 'done', 'failed']
|
||||
}).notNull().default('queued'),
|
||||
})
|
||||
.notNull()
|
||||
.default('queued'),
|
||||
progress: integer('progress').default(0), // 0–100
|
||||
totalFiles: integer('total_files').default(0),
|
||||
processedFiles: integer('processed_files').default(0),
|
||||
error: text('error'),
|
||||
startedAt: integer('started_at', { mode: 'timestamp' }),
|
||||
completedAt: integer('completed_at', { mode: 'timestamp' }),
|
||||
createdAt: integer('created_at', { mode: 'timestamp' }).notNull(),
|
||||
createdAt: integer('created_at', { mode: 'timestamp' }).notNull()
|
||||
});
|
||||
```
|
||||
|
||||
@@ -159,7 +169,8 @@ Stores parsed `trueref.json` / `context7.json` configuration.
|
||||
|
||||
```typescript
|
||||
export const repositoryConfigs = sqliteTable('repository_configs', {
|
||||
repositoryId: text('repository_id').primaryKey()
|
||||
repositoryId: text('repository_id')
|
||||
.primaryKey()
|
||||
.references(() => repositories.id, { onDelete: 'cascade' }),
|
||||
projectTitle: text('project_title'),
|
||||
description: text('description'),
|
||||
@@ -167,9 +178,10 @@ export const repositoryConfigs = sqliteTable('repository_configs', {
|
||||
excludeFolders: text('exclude_folders', { mode: 'json' }).$type<string[]>(),
|
||||
excludeFiles: text('exclude_files', { mode: 'json' }).$type<string[]>(),
|
||||
rules: text('rules', { mode: 'json' }).$type<string[]>(),
|
||||
previousVersions: text('previous_versions', { mode: 'json' })
|
||||
.$type<{ tag: string; title: string }[]>(),
|
||||
updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull(),
|
||||
previousVersions: text('previous_versions', { mode: 'json' }).$type<
|
||||
{ tag: string; title: string }[]
|
||||
>(),
|
||||
updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull()
|
||||
});
|
||||
```
|
||||
|
||||
@@ -181,7 +193,7 @@ Key-value store for global application settings.
|
||||
export const settings = sqliteTable('settings', {
|
||||
key: text('key').primaryKey(),
|
||||
value: text('value', { mode: 'json' }),
|
||||
updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull(),
|
||||
updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull()
|
||||
});
|
||||
```
|
||||
|
||||
|
||||
@@ -31,10 +31,12 @@ Implement the core `RepositoryService` that handles CRUD operations for reposito
|
||||
## Repository ID Generation
|
||||
|
||||
GitHub repositories:
|
||||
|
||||
- Input URL: `https://github.com/facebook/react` or `github.com/facebook/react`
|
||||
- Generated ID: `/facebook/react`
|
||||
|
||||
Local repositories:
|
||||
|
||||
- Input path: `/home/user/projects/my-sdk`
|
||||
- Generated ID: `/local/my-sdk` (basename of path, slugified)
|
||||
- Collision resolution: append `-2`, `-3`, etc.
|
||||
@@ -71,22 +73,22 @@ export class RepositoryService {
|
||||
state?: Repository['state'];
|
||||
limit?: number;
|
||||
offset?: number;
|
||||
}): Promise<Repository[]>
|
||||
}): Promise<Repository[]>;
|
||||
|
||||
async get(id: string): Promise<Repository | null>
|
||||
async get(id: string): Promise<Repository | null>;
|
||||
|
||||
async add(input: AddRepositoryInput): Promise<Repository>
|
||||
async add(input: AddRepositoryInput): Promise<Repository>;
|
||||
|
||||
async update(id: string, input: UpdateRepositoryInput): Promise<Repository>
|
||||
async update(id: string, input: UpdateRepositoryInput): Promise<Repository>;
|
||||
|
||||
async remove(id: string): Promise<void>
|
||||
async remove(id: string): Promise<void>;
|
||||
|
||||
async getStats(id: string): Promise<{
|
||||
totalSnippets: number;
|
||||
totalTokens: number;
|
||||
totalDocuments: number;
|
||||
lastIndexedAt: Date | null;
|
||||
}>
|
||||
}>;
|
||||
}
|
||||
```
|
||||
|
||||
@@ -97,11 +99,13 @@ export class RepositoryService {
|
||||
### `GET /api/v1/libs`
|
||||
|
||||
Query parameters:
|
||||
|
||||
- `state` (optional): filter by state (`pending`, `indexed`, `error`, etc.)
|
||||
- `limit` (optional, default 50): max results
|
||||
- `offset` (optional, default 0): pagination offset
|
||||
|
||||
Response `200`:
|
||||
|
||||
```json
|
||||
{
|
||||
"libraries": [
|
||||
@@ -128,6 +132,7 @@ Response `200`:
|
||||
### `POST /api/v1/libs`
|
||||
|
||||
Request body:
|
||||
|
||||
```json
|
||||
{
|
||||
"source": "github",
|
||||
@@ -139,6 +144,7 @@ Request body:
|
||||
```
|
||||
|
||||
Response `201`:
|
||||
|
||||
```json
|
||||
{
|
||||
"library": { ...Repository },
|
||||
@@ -149,6 +155,7 @@ Response `201`:
|
||||
`autoIndex: true` (default) immediately queues an indexing job.
|
||||
|
||||
Response `409` if repository already exists:
|
||||
|
||||
```json
|
||||
{ "error": "Repository /facebook/react already exists" }
|
||||
```
|
||||
@@ -176,11 +183,13 @@ Response `404`: not found.
|
||||
Triggers a new indexing job. If a job is already running for this repo, returns the existing job.
|
||||
|
||||
Request body (optional):
|
||||
|
||||
```json
|
||||
{ "version": "v18.3.0" }
|
||||
```
|
||||
|
||||
Response `202`:
|
||||
|
||||
```json
|
||||
{
|
||||
"job": {
|
||||
@@ -198,6 +207,7 @@ Response `202`:
|
||||
## Error Response Shape
|
||||
|
||||
All error responses follow:
|
||||
|
||||
```json
|
||||
{
|
||||
"error": "Human-readable message",
|
||||
@@ -207,6 +217,7 @@ All error responses follow:
|
||||
```
|
||||
|
||||
Error codes:
|
||||
|
||||
- `NOT_FOUND`
|
||||
- `ALREADY_EXISTS`
|
||||
- `INVALID_INPUT`
|
||||
|
||||
@@ -38,16 +38,45 @@ The crawler only downloads files with these extensions:
|
||||
```typescript
|
||||
const INDEXABLE_EXTENSIONS = new Set([
|
||||
// Documentation
|
||||
'.md', '.mdx', '.txt', '.rst',
|
||||
'.md',
|
||||
'.mdx',
|
||||
'.txt',
|
||||
'.rst',
|
||||
// Code
|
||||
'.ts', '.tsx', '.js', '.jsx',
|
||||
'.py', '.rb', '.go', '.rs', '.java', '.cs', '.cpp', '.c', '.h',
|
||||
'.swift', '.kt', '.php', '.scala', '.clj', '.ex', '.exs',
|
||||
'.sh', '.bash', '.zsh', '.fish',
|
||||
'.ts',
|
||||
'.tsx',
|
||||
'.js',
|
||||
'.jsx',
|
||||
'.py',
|
||||
'.rb',
|
||||
'.go',
|
||||
'.rs',
|
||||
'.java',
|
||||
'.cs',
|
||||
'.cpp',
|
||||
'.c',
|
||||
'.h',
|
||||
'.swift',
|
||||
'.kt',
|
||||
'.php',
|
||||
'.scala',
|
||||
'.clj',
|
||||
'.ex',
|
||||
'.exs',
|
||||
'.sh',
|
||||
'.bash',
|
||||
'.zsh',
|
||||
'.fish',
|
||||
// Config / data
|
||||
'.json', '.yaml', '.yml', '.toml',
|
||||
'.json',
|
||||
'.yaml',
|
||||
'.yml',
|
||||
'.toml',
|
||||
// Web
|
||||
'.html', '.css', '.svelte', '.vue',
|
||||
'.html',
|
||||
'.css',
|
||||
'.svelte',
|
||||
'.vue'
|
||||
]);
|
||||
|
||||
const MAX_FILE_SIZE_BYTES = 500_000; // 500 KB — skip large generated files
|
||||
@@ -89,12 +118,14 @@ export interface CrawlOptions {
|
||||
## GitHub API Usage
|
||||
|
||||
### Step 1: Get default branch (if ref not specified)
|
||||
|
||||
```
|
||||
GET https://api.github.com/repos/{owner}/{repo}
|
||||
→ { default_branch: "main", stargazers_count: 12345 }
|
||||
```
|
||||
|
||||
### Step 2: Fetch file tree (recursive)
|
||||
|
||||
```
|
||||
GET https://api.github.com/repos/{owner}/{repo}/git/trees/{ref}?recursive=1
|
||||
→ {
|
||||
@@ -109,12 +140,14 @@ GET https://api.github.com/repos/{owner}/{repo}/git/trees/{ref}?recursive=1
|
||||
If `truncated: true`, the tree has >100k items. Use `--depth` pagination or filter top-level directories first.
|
||||
|
||||
### Step 3: Download file contents (parallel)
|
||||
|
||||
```
|
||||
GET https://api.github.com/repos/{owner}/{repo}/contents/{path}?ref={ref}
|
||||
→ { content: "<base64>", encoding: "base64", size: 1234, sha: "abc123" }
|
||||
```
|
||||
|
||||
Alternative for large repos: use raw content URL:
|
||||
|
||||
```
|
||||
GET https://raw.githubusercontent.com/{owner}/{repo}/{ref}/{path}
|
||||
```
|
||||
@@ -124,11 +157,7 @@ GET https://raw.githubusercontent.com/{owner}/{repo}/{ref}/{path}
|
||||
## Filtering Logic
|
||||
|
||||
```typescript
|
||||
function shouldIndexFile(
|
||||
filePath: string,
|
||||
fileSize: number,
|
||||
config?: RepoConfig
|
||||
): boolean {
|
||||
function shouldIndexFile(filePath: string, fileSize: number, config?: RepoConfig): boolean {
|
||||
const ext = path.extname(filePath).toLowerCase();
|
||||
const base = path.basename(filePath);
|
||||
|
||||
@@ -153,14 +182,17 @@ function shouldIndexFile(
|
||||
if (isInIgnoredDirectory(filePath)) return false;
|
||||
|
||||
// 7. Apply config excludeFolders (regex or prefix match)
|
||||
if (config?.excludeFolders?.some(folder =>
|
||||
filePath.startsWith(folder) || new RegExp(folder).test(filePath)
|
||||
)) return false;
|
||||
if (
|
||||
config?.excludeFolders?.some(
|
||||
(folder) => filePath.startsWith(folder) || new RegExp(folder).test(filePath)
|
||||
)
|
||||
)
|
||||
return false;
|
||||
|
||||
// 8. Apply config folders allowlist (if specified, only index those paths)
|
||||
if (config?.folders?.length) {
|
||||
const inAllowedFolder = config.folders.some(folder =>
|
||||
filePath.startsWith(folder) || new RegExp(folder).test(filePath)
|
||||
const inAllowedFolder = config.folders.some(
|
||||
(folder) => filePath.startsWith(folder) || new RegExp(folder).test(filePath)
|
||||
);
|
||||
if (!inAllowedFolder) return false;
|
||||
}
|
||||
@@ -201,7 +233,7 @@ Requests are made with a concurrency limit of 10 parallel downloads using a sema
|
||||
## Error Handling
|
||||
|
||||
| Scenario | Behavior |
|
||||
|----------|---------|
|
||||
| ------------------------- | --------------------------------------------------------------------------- |
|
||||
| 404 Not Found | Throw `RepositoryNotFoundError` |
|
||||
| 401 Unauthorized | Throw `AuthenticationError` (invalid or missing token) |
|
||||
| 403 Forbidden | If `X-RateLimit-Remaining: 0`, wait and retry; else throw `PermissionError` |
|
||||
|
||||
@@ -58,18 +58,14 @@ export class LocalCrawler {
|
||||
const allFiles = await this.walkDirectory(options.rootPath, '', gitignore);
|
||||
|
||||
// 3. Look for trueref.json / context7.json first
|
||||
const configFile = allFiles.find(f =>
|
||||
f === 'trueref.json' || f === 'context7.json'
|
||||
);
|
||||
const configFile = allFiles.find((f) => f === 'trueref.json' || f === 'context7.json');
|
||||
let config = options.config;
|
||||
if (configFile && !config) {
|
||||
config = await this.parseConfigFile(
|
||||
path.join(options.rootPath, configFile)
|
||||
);
|
||||
config = await this.parseConfigFile(path.join(options.rootPath, configFile));
|
||||
}
|
||||
|
||||
// 4. Filter files
|
||||
const filteredFiles = allFiles.filter(relPath => {
|
||||
const filteredFiles = allFiles.filter((relPath) => {
|
||||
const stat = statCache.get(relPath);
|
||||
return shouldIndexFile(relPath, stat.size, config);
|
||||
});
|
||||
@@ -85,7 +81,7 @@ export class LocalCrawler {
|
||||
content,
|
||||
size: Buffer.byteLength(content, 'utf-8'),
|
||||
sha,
|
||||
language: detectLanguage(relPath),
|
||||
language: detectLanguage(relPath)
|
||||
});
|
||||
options.onProgress?.(i + 1, filteredFiles.length);
|
||||
}
|
||||
@@ -95,11 +91,15 @@ export class LocalCrawler {
|
||||
totalFiles: filteredFiles.length,
|
||||
skippedFiles: allFiles.length - filteredFiles.length,
|
||||
branch: 'local',
|
||||
commitSha: computeSHA256(crawledFiles.map(f => f.sha).join('')),
|
||||
commitSha: computeSHA256(crawledFiles.map((f) => f.sha).join(''))
|
||||
};
|
||||
}
|
||||
|
||||
private async walkDirectory(dir: string, rel = '', gitignore?: GitignoreFilter): Promise<string[]> {
|
||||
private async walkDirectory(
|
||||
dir: string,
|
||||
rel = '',
|
||||
gitignore?: GitignoreFilter
|
||||
): Promise<string[]> {
|
||||
const entries = await fs.readdir(dir, { withFileTypes: true });
|
||||
const files: string[] = [];
|
||||
for (const entry of entries) {
|
||||
@@ -109,9 +109,7 @@ export class LocalCrawler {
|
||||
if (shouldPruneDirectory(relPath) || gitignore?.isIgnored(relPath, true)) {
|
||||
continue;
|
||||
}
|
||||
files.push(...await this.walkDirectory(
|
||||
path.join(dir, entry.name), relPath, gitignore
|
||||
));
|
||||
files.push(...(await this.walkDirectory(path.join(dir, entry.name), relPath, gitignore)));
|
||||
} else {
|
||||
if (gitignore?.isIgnored(relPath, false)) continue;
|
||||
files.push(relPath);
|
||||
|
||||
@@ -31,7 +31,7 @@ Implement the document parsing and chunking pipeline that transforms raw file co
|
||||
## Supported File Types
|
||||
|
||||
| Extension | Parser Strategy |
|
||||
|-----------|----------------|
|
||||
| --------------------------------- | ------------------------------------------------------- |
|
||||
| `.md`, `.mdx` | Heading-based section splitting + code block extraction |
|
||||
| `.txt`, `.rst` | Paragraph-based splitting |
|
||||
| `.ts`, `.tsx`, `.js`, `.jsx` | AST-free: function/class boundary detection via regex |
|
||||
@@ -96,7 +96,7 @@ function parseMarkdown(content: string, filePath: string): Snippet[] {
|
||||
title,
|
||||
content: chunk,
|
||||
breadcrumb,
|
||||
tokenCount: estimateTokens(chunk),
|
||||
tokenCount: estimateTokens(chunk)
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -110,7 +110,7 @@ function parseMarkdown(content: string, filePath: string): Snippet[] {
|
||||
content: block.code,
|
||||
language: block.language || detectLanguage('.' + block.language),
|
||||
breadcrumb,
|
||||
tokenCount: estimateTokens(block.code),
|
||||
tokenCount: estimateTokens(block.code)
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -139,14 +139,10 @@ const BOUNDARY_PATTERNS: Record<string, RegExp> = {
|
||||
python: /^(async\s+)?(def|class)\s+\w+/m,
|
||||
go: /^(func|type|var|const)\s+\w+/m,
|
||||
rust: /^(pub\s+)?(fn|impl|struct|enum|trait)\s+\w+/m,
|
||||
java: /^(public|private|protected|static).*?(class|interface|enum|void|\w+)\s+\w+\s*[({]/m,
|
||||
java: /^(public|private|protected|static).*?(class|interface|enum|void|\w+)\s+\w+\s*[({]/m
|
||||
};
|
||||
|
||||
function parseCodeFile(
|
||||
content: string,
|
||||
filePath: string,
|
||||
language: string
|
||||
): Snippet[] {
|
||||
function parseCodeFile(content: string, filePath: string, language: string): Snippet[] {
|
||||
const pattern = BOUNDARY_PATTERNS[language];
|
||||
const breadcrumb = filePath;
|
||||
const title = path.basename(filePath);
|
||||
@@ -158,17 +154,19 @@ function parseCodeFile(
|
||||
|
||||
const chunks = splitAtBoundaries(content, pattern);
|
||||
return chunks
|
||||
.filter(chunk => chunk.trim().length >= 20)
|
||||
.flatMap(chunk => {
|
||||
.filter((chunk) => chunk.trim().length >= 20)
|
||||
.flatMap((chunk) => {
|
||||
if (estimateTokens(chunk) <= MAX_TOKENS) {
|
||||
return [{
|
||||
return [
|
||||
{
|
||||
type: 'code' as const,
|
||||
title,
|
||||
content: chunk,
|
||||
language,
|
||||
breadcrumb,
|
||||
tokenCount: estimateTokens(chunk),
|
||||
}];
|
||||
tokenCount: estimateTokens(chunk)
|
||||
}
|
||||
];
|
||||
}
|
||||
return slidingWindowChunks(chunk, filePath, language);
|
||||
});
|
||||
@@ -188,11 +186,7 @@ const MIN_CONTENT_LENGTH = 20; // characters
|
||||
### Sliding Window Chunker
|
||||
|
||||
```typescript
|
||||
function chunkText(
|
||||
text: string,
|
||||
maxTokens: number,
|
||||
overlapTokens: number
|
||||
): string[] {
|
||||
function chunkText(text: string, maxTokens: number, overlapTokens: number): string[] {
|
||||
const words = text.split(/\s+/);
|
||||
const wordsPerToken = 0.75; // ~0.75 words per token
|
||||
const maxWords = Math.floor(maxTokens * wordsPerToken);
|
||||
@@ -218,29 +212,37 @@ function chunkText(
|
||||
|
||||
```typescript
|
||||
const LANGUAGE_MAP: Record<string, string> = {
|
||||
'.ts': 'typescript', '.tsx': 'typescript',
|
||||
'.js': 'javascript', '.jsx': 'javascript',
|
||||
'.ts': 'typescript',
|
||||
'.tsx': 'typescript',
|
||||
'.js': 'javascript',
|
||||
'.jsx': 'javascript',
|
||||
'.py': 'python',
|
||||
'.rb': 'ruby',
|
||||
'.go': 'go',
|
||||
'.rs': 'rust',
|
||||
'.java': 'java',
|
||||
'.cs': 'csharp',
|
||||
'.cpp': 'cpp', '.c': 'c', '.h': 'c',
|
||||
'.cpp': 'cpp',
|
||||
'.c': 'c',
|
||||
'.h': 'c',
|
||||
'.swift': 'swift',
|
||||
'.kt': 'kotlin',
|
||||
'.php': 'php',
|
||||
'.scala': 'scala',
|
||||
'.sh': 'bash', '.bash': 'bash', '.zsh': 'bash',
|
||||
'.md': 'markdown', '.mdx': 'markdown',
|
||||
'.sh': 'bash',
|
||||
'.bash': 'bash',
|
||||
'.zsh': 'bash',
|
||||
'.md': 'markdown',
|
||||
'.mdx': 'markdown',
|
||||
'.json': 'json',
|
||||
'.yaml': 'yaml', '.yml': 'yaml',
|
||||
'.yaml': 'yaml',
|
||||
'.yml': 'yaml',
|
||||
'.toml': 'toml',
|
||||
'.html': 'html',
|
||||
'.css': 'css',
|
||||
'.svelte': 'svelte',
|
||||
'.vue': 'vue',
|
||||
'.sql': 'sql',
|
||||
'.sql': 'sql'
|
||||
};
|
||||
|
||||
function detectLanguage(filePath: string): string {
|
||||
@@ -260,12 +262,12 @@ export interface ParseOptions {
|
||||
versionId?: string;
|
||||
}
|
||||
|
||||
export function parseFile(
|
||||
file: CrawledFile,
|
||||
options: ParseOptions
|
||||
): NewSnippet[] {
|
||||
export function parseFile(file: CrawledFile, options: ParseOptions): NewSnippet[] {
|
||||
const language = detectLanguage(file.path);
|
||||
let rawSnippets: Omit<NewSnippet, 'id' | 'repositoryId' | 'documentId' | 'versionId' | 'createdAt'>[];
|
||||
let rawSnippets: Omit<
|
||||
NewSnippet,
|
||||
'id' | 'repositoryId' | 'documentId' | 'versionId' | 'createdAt'
|
||||
>[];
|
||||
|
||||
if (language === 'markdown') {
|
||||
rawSnippets = parseMarkdown(file.content, file.path);
|
||||
@@ -273,13 +275,13 @@ export function parseFile(
|
||||
rawSnippets = parseCodeFile(file.content, file.path, language);
|
||||
}
|
||||
|
||||
return rawSnippets.map(s => ({
|
||||
return rawSnippets.map((s) => ({
|
||||
...s,
|
||||
id: crypto.randomUUID(),
|
||||
repositoryId: options.repositoryId,
|
||||
documentId: options.documentId,
|
||||
versionId: options.versionId ?? null,
|
||||
createdAt: new Date(),
|
||||
createdAt: new Date()
|
||||
}));
|
||||
}
|
||||
```
|
||||
|
||||
@@ -61,14 +61,9 @@ export interface LibrarySearchResult {
|
||||
export class SearchService {
|
||||
constructor(private db: BetterSQLite3.Database) {}
|
||||
|
||||
searchSnippets(
|
||||
query: string,
|
||||
options: SnippetSearchOptions
|
||||
): SnippetSearchResult[]
|
||||
searchSnippets(query: string, options: SnippetSearchOptions): SnippetSearchResult[];
|
||||
|
||||
searchRepositories(
|
||||
options: LibrarySearchOptions
|
||||
): LibrarySearchResult[]
|
||||
searchRepositories(options: LibrarySearchOptions): LibrarySearchResult[];
|
||||
}
|
||||
```
|
||||
|
||||
@@ -174,52 +169,61 @@ searchRepositories(options: LibrarySearchOptions): LibrarySearchResult[] {
|
||||
The search results must be formatted for the REST API and MCP tool responses:
|
||||
|
||||
### Library search response (for `resolve-library-id`):
|
||||
|
||||
```typescript
|
||||
function formatLibraryResults(results: LibrarySearchResult[]): string {
|
||||
if (results.length === 0) {
|
||||
return 'No libraries found matching your search.';
|
||||
}
|
||||
|
||||
return results.map((r, i) => {
|
||||
return results
|
||||
.map((r, i) => {
|
||||
const repo = r.repository;
|
||||
const versions = r.versions.map(v => v.tag).join(', ') || 'default branch';
|
||||
const versions = r.versions.map((v) => v.tag).join(', ') || 'default branch';
|
||||
return [
|
||||
`${i + 1}. ${repo.title}`,
|
||||
` Library ID: ${repo.id}`,
|
||||
` Description: ${repo.description ?? 'No description'}`,
|
||||
` Snippets: ${repo.totalSnippets} | Trust Score: ${repo.trustScore.toFixed(1)}/10`,
|
||||
` Available Versions: ${versions}`,
|
||||
` Available Versions: ${versions}`
|
||||
].join('\n');
|
||||
}).join('\n\n');
|
||||
})
|
||||
.join('\n\n');
|
||||
}
|
||||
```
|
||||
|
||||
### Snippet search response (for `query-docs`):
|
||||
|
||||
```typescript
|
||||
function formatSnippetResults(
|
||||
results: SnippetSearchResult[],
|
||||
rules?: string[]
|
||||
): string {
|
||||
function formatSnippetResults(results: SnippetSearchResult[], rules?: string[]): string {
|
||||
const parts: string[] = [];
|
||||
|
||||
// Prepend repository rules if present
|
||||
if (rules?.length) {
|
||||
parts.push('## Library Rules\n' + rules.map(r => `- ${r}`).join('\n'));
|
||||
parts.push('## Library Rules\n' + rules.map((r) => `- ${r}`).join('\n'));
|
||||
}
|
||||
|
||||
for (const { snippet } of results) {
|
||||
if (snippet.type === 'code') {
|
||||
parts.push([
|
||||
parts.push(
|
||||
[
|
||||
snippet.title ? `### ${snippet.title}` : '',
|
||||
snippet.breadcrumb ? `*${snippet.breadcrumb}*` : '',
|
||||
`\`\`\`${snippet.language ?? ''}\n${snippet.content}\n\`\`\``,
|
||||
].filter(Boolean).join('\n'));
|
||||
`\`\`\`${snippet.language ?? ''}\n${snippet.content}\n\`\`\``
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join('\n')
|
||||
);
|
||||
} else {
|
||||
parts.push([
|
||||
parts.push(
|
||||
[
|
||||
snippet.title ? `### ${snippet.title}` : '',
|
||||
snippet.breadcrumb ? `*${snippet.breadcrumb}*` : '',
|
||||
snippet.content,
|
||||
].filter(Boolean).join('\n'));
|
||||
snippet.content
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join('\n')
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -74,14 +74,14 @@ export class OpenAIEmbeddingProvider implements EmbeddingProvider {
|
||||
const response = await fetch(`${this.config.baseUrl}/embeddings`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Authorization': `Bearer ${this.config.apiKey}`,
|
||||
'Content-Type': 'application/json',
|
||||
Authorization: `Bearer ${this.config.apiKey}`,
|
||||
'Content-Type': 'application/json'
|
||||
},
|
||||
body: JSON.stringify({
|
||||
model: this.config.model,
|
||||
input: batch,
|
||||
dimensions: this.config.dimensions,
|
||||
}),
|
||||
dimensions: this.config.dimensions
|
||||
})
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
@@ -93,7 +93,7 @@ export class OpenAIEmbeddingProvider implements EmbeddingProvider {
|
||||
allEmbeddings.push({
|
||||
values: new Float32Array(item.embedding),
|
||||
dimensions: item.embedding.length,
|
||||
model: this.config.model,
|
||||
model: this.config.model
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -126,12 +126,12 @@ export class LocalEmbeddingProvider implements EmbeddingProvider {
|
||||
for (const text of texts) {
|
||||
const output = await (this.pipeline as Function)(text, {
|
||||
pooling: 'mean',
|
||||
normalize: true,
|
||||
normalize: true
|
||||
});
|
||||
results.push({
|
||||
values: new Float32Array(output.data),
|
||||
dimensions: this.dimensions,
|
||||
model: this.model,
|
||||
model: this.model
|
||||
});
|
||||
}
|
||||
return results;
|
||||
@@ -163,12 +163,14 @@ export class EmbeddingService {
|
||||
snippetIds: string[],
|
||||
onProgress?: (done: number, total: number) => void
|
||||
): Promise<void> {
|
||||
const snippets = this.db.prepare(
|
||||
const snippets = this.db
|
||||
.prepare(
|
||||
`SELECT id, content, type FROM snippets WHERE id IN (${snippetIds.map(() => '?').join(',')})`
|
||||
).all(...snippetIds) as Snippet[];
|
||||
)
|
||||
.all(...snippetIds) as Snippet[];
|
||||
|
||||
// Prepare text for embedding: combine title + content
|
||||
const texts = snippets.map(s =>
|
||||
const texts = snippets.map((s) =>
|
||||
[s.title, s.breadcrumb, s.content].filter(Boolean).join('\n').slice(0, 2048)
|
||||
);
|
||||
|
||||
@@ -227,6 +229,7 @@ export interface EmbeddingConfig {
|
||||
### API Endpoints
|
||||
|
||||
`GET /api/v1/settings/embedding`
|
||||
|
||||
```json
|
||||
{
|
||||
"provider": "openai",
|
||||
@@ -251,11 +254,7 @@ Embeddings are stored as raw `Float32Array` binary blobs:
|
||||
const buffer = Buffer.from(float32Array.buffer);
|
||||
|
||||
// Retrieve
|
||||
const float32Array = new Float32Array(
|
||||
buffer.buffer,
|
||||
buffer.byteOffset,
|
||||
buffer.byteLength / 4
|
||||
);
|
||||
const float32Array = new Float32Array(buffer.buffer, buffer.byteOffset, buffer.byteLength / 4);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@@ -137,13 +137,10 @@ export class HybridSearchService {
|
||||
constructor(
|
||||
private db: BetterSQLite3.Database,
|
||||
private searchService: SearchService,
|
||||
private embeddingProvider: EmbeddingProvider | null,
|
||||
private embeddingProvider: EmbeddingProvider | null
|
||||
) {}
|
||||
|
||||
async search(
|
||||
query: string,
|
||||
options: HybridSearchOptions
|
||||
): Promise<SnippetSearchResult[]> {
|
||||
async search(query: string, options: HybridSearchOptions): Promise<SnippetSearchResult[]> {
|
||||
const limit = options.limit ?? 20;
|
||||
const alpha = options.alpha ?? 0.5;
|
||||
|
||||
@@ -152,7 +149,7 @@ export class HybridSearchService {
|
||||
repositoryId: options.repositoryId,
|
||||
versionId: options.versionId,
|
||||
type: options.type,
|
||||
limit: limit * 3, // get more candidates for fusion
|
||||
limit: limit * 3 // get more candidates for fusion
|
||||
});
|
||||
|
||||
// If no embedding provider or alpha = 0, return FTS5 results directly
|
||||
@@ -171,18 +168,18 @@ export class HybridSearchService {
|
||||
// Normalize result lists for RRF
|
||||
const ftsRanked = ftsResults.map((r, i) => ({
|
||||
id: r.snippet.id,
|
||||
score: i,
|
||||
score: i
|
||||
}));
|
||||
const vecRanked = vectorResults.map((r, i) => ({
|
||||
id: r.snippetId,
|
||||
score: i,
|
||||
score: i
|
||||
}));
|
||||
|
||||
// Apply RRF
|
||||
const fused = reciprocalRankFusion(ftsRanked, vecRanked);
|
||||
|
||||
// Fetch full snippet data for top results
|
||||
const topIds = fused.slice(0, limit).map(r => r.id);
|
||||
const topIds = fused.slice(0, limit).map((r) => r.id);
|
||||
return this.fetchSnippetsByIds(topIds, options.repositoryId);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -72,12 +72,16 @@ export class JobQueue {
|
||||
error: null,
|
||||
startedAt: null,
|
||||
completedAt: null,
|
||||
createdAt: new Date(),
|
||||
createdAt: new Date()
|
||||
};
|
||||
|
||||
this.db.prepare(`
|
||||
this.db
|
||||
.prepare(
|
||||
`
|
||||
INSERT INTO indexing_jobs VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
`).run(...Object.values(job));
|
||||
`
|
||||
)
|
||||
.run(...Object.values(job));
|
||||
|
||||
// Kick off processing if not already running
|
||||
if (!this.isRunning) {
|
||||
@@ -90,12 +94,16 @@ export class JobQueue {
|
||||
private async processNext(): Promise<void> {
|
||||
if (this.isRunning) return;
|
||||
|
||||
const job = this.db.prepare(`
|
||||
const job = this.db
|
||||
.prepare(
|
||||
`
|
||||
SELECT * FROM indexing_jobs
|
||||
WHERE status = 'queued'
|
||||
ORDER BY created_at ASC
|
||||
LIMIT 1
|
||||
`).get() as IndexingJob | undefined;
|
||||
`
|
||||
)
|
||||
.get() as IndexingJob | undefined;
|
||||
|
||||
if (!job) return;
|
||||
|
||||
@@ -105,17 +113,17 @@ export class JobQueue {
|
||||
} finally {
|
||||
this.isRunning = false;
|
||||
// Check for next queued job
|
||||
const nextJob = this.db.prepare(
|
||||
`SELECT id FROM indexing_jobs WHERE status = 'queued' LIMIT 1`
|
||||
).get();
|
||||
const nextJob = this.db
|
||||
.prepare(`SELECT id FROM indexing_jobs WHERE status = 'queued' LIMIT 1`)
|
||||
.get();
|
||||
if (nextJob) setImmediate(() => this.processNext());
|
||||
}
|
||||
}
|
||||
|
||||
getJob(id: string): IndexingJob | null {
|
||||
return this.db.prepare(
|
||||
`SELECT * FROM indexing_jobs WHERE id = ?`
|
||||
).get(id) as IndexingJob | null;
|
||||
return this.db
|
||||
.prepare(`SELECT * FROM indexing_jobs WHERE id = ?`)
|
||||
.get(id) as IndexingJob | null;
|
||||
}
|
||||
|
||||
listJobs(repositoryId?: string, limit = 20): IndexingJob[] {
|
||||
@@ -140,7 +148,7 @@ export class IndexingPipeline {
|
||||
private db: BetterSQLite3.Database,
|
||||
private githubCrawler: GitHubCrawler,
|
||||
private localCrawler: LocalCrawler,
|
||||
private embeddingService: EmbeddingService | null,
|
||||
private embeddingService: EmbeddingService | null
|
||||
) {}
|
||||
|
||||
async run(job: IndexingJob): Promise<void> {
|
||||
@@ -157,8 +165,11 @@ export class IndexingPipeline {
|
||||
const crawlResult = await this.crawl(repo, job);
|
||||
|
||||
// Step 2: Parse and diff
|
||||
const { newSnippets, changedDocIds, newDocuments } =
|
||||
await this.parseAndDiff(crawlResult, repo, job);
|
||||
const { newSnippets, changedDocIds, newDocuments } = await this.parseAndDiff(
|
||||
crawlResult,
|
||||
repo,
|
||||
job
|
||||
);
|
||||
|
||||
// Step 3: Atomic replacement
|
||||
this.replaceSnippets(repo.id, changedDocIds, newDocuments, newSnippets);
|
||||
@@ -166,7 +177,7 @@ export class IndexingPipeline {
|
||||
// Step 4: Embeddings (async, non-blocking for job completion)
|
||||
if (this.embeddingService && newSnippets.length > 0) {
|
||||
await this.embeddingService.embedSnippets(
|
||||
newSnippets.map(s => s.id),
|
||||
newSnippets.map((s) => s.id),
|
||||
(done, total) => {
|
||||
// Update job progress for embedding phase
|
||||
}
|
||||
@@ -180,20 +191,19 @@ export class IndexingPipeline {
|
||||
totalSnippets: stats.totalSnippets,
|
||||
totalTokens: stats.totalTokens,
|
||||
trustScore: computeTrustScore({ ...repo, ...stats }),
|
||||
lastIndexedAt: new Date(),
|
||||
lastIndexedAt: new Date()
|
||||
});
|
||||
|
||||
this.updateJob(job.id, {
|
||||
status: 'done',
|
||||
progress: 100,
|
||||
completedAt: new Date(),
|
||||
completedAt: new Date()
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
this.updateJob(job.id, {
|
||||
status: 'failed',
|
||||
error: (error as Error).message,
|
||||
completedAt: new Date(),
|
||||
completedAt: new Date()
|
||||
});
|
||||
this.updateRepo(job.repositoryId, { state: 'error' });
|
||||
throw error;
|
||||
@@ -210,9 +220,9 @@ export class IndexingPipeline {
|
||||
this.db.transaction(() => {
|
||||
if (changedDocIds.length > 0) {
|
||||
// Cascade deletes snippets via FK constraint
|
||||
this.db.prepare(
|
||||
`DELETE FROM documents WHERE id IN (${changedDocIds.map(() => '?').join(',')})`
|
||||
).run(...changedDocIds);
|
||||
this.db
|
||||
.prepare(`DELETE FROM documents WHERE id IN (${changedDocIds.map(() => '?').join(',')})`)
|
||||
.run(...changedDocIds);
|
||||
}
|
||||
|
||||
for (const doc of newDocuments) {
|
||||
@@ -248,9 +258,7 @@ function calculateProgress(
|
||||
|
||||
// Crawl+parse = 80%, embeddings = 20%
|
||||
const parseProgress = (processedFiles / totalFiles) * 80;
|
||||
const embedProgress = embeddingsTotal > 0
|
||||
? (embeddingsDone / embeddingsTotal) * 20
|
||||
: 0;
|
||||
const embedProgress = embeddingsTotal > 0 ? (embeddingsDone / embeddingsTotal) * 20 : 0;
|
||||
|
||||
return Math.round(parseProgress + embedProgress);
|
||||
}
|
||||
@@ -263,6 +271,7 @@ function calculateProgress(
|
||||
### `GET /api/v1/jobs/:id`
|
||||
|
||||
Response `200`:
|
||||
|
||||
```json
|
||||
{
|
||||
"job": {
|
||||
@@ -285,6 +294,7 @@ Response `200`:
|
||||
Query params: `repositoryId` (optional), `status` (optional), `limit` (default 20).
|
||||
|
||||
Response `200`:
|
||||
|
||||
```json
|
||||
{
|
||||
"jobs": [...],
|
||||
@@ -300,20 +310,24 @@ On application start, mark any jobs in `running` state as `failed` (they were in
|
||||
|
||||
```typescript
|
||||
function recoverStaleJobs(db: BetterSQLite3.Database): void {
|
||||
db.prepare(`
|
||||
db.prepare(
|
||||
`
|
||||
UPDATE indexing_jobs
|
||||
SET status = 'failed',
|
||||
error = 'Server restarted while job was running',
|
||||
completed_at = unixepoch()
|
||||
WHERE status = 'running'
|
||||
`).run();
|
||||
`
|
||||
).run();
|
||||
|
||||
// Also reset any repositories stuck in 'indexing' state
|
||||
db.prepare(`
|
||||
db.prepare(
|
||||
`
|
||||
UPDATE repositories
|
||||
SET state = 'error'
|
||||
WHERE state = 'indexing'
|
||||
`).run();
|
||||
`
|
||||
).run();
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
@@ -33,7 +33,7 @@ Implement the public-facing REST API endpoints that replicate context7's `/api/v
|
||||
### Query Parameters
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| ------------- | ------- | -------- | ------------------------------------- |
|
||||
| `libraryName` | string | Yes | Library name to search for |
|
||||
| `query` | string | No | User's question for relevance ranking |
|
||||
| `limit` | integer | No | Max results (default: 10, max: 50) |
|
||||
@@ -67,7 +67,7 @@ Note: `state: "finalized"` maps from TrueRef's `state: "indexed"` for compatibil
|
||||
### State Mapping
|
||||
|
||||
| TrueRef state | context7 state |
|
||||
|---------------|---------------|
|
||||
| ------------- | -------------- |
|
||||
| `pending` | `initial` |
|
||||
| `indexing` | `initial` |
|
||||
| `indexed` | `finalized` |
|
||||
@@ -82,7 +82,7 @@ Note: `state: "finalized"` maps from TrueRef's `state: "indexed"` for compatibil
|
||||
### Query Parameters
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| ----------- | ------- | -------- | --------------------------------------------------------------- |
|
||||
| `libraryId` | string | Yes | Library ID, e.g. `/facebook/react` or `/facebook/react/v18.3.0` |
|
||||
| `query` | string | Yes | Specific question about the library |
|
||||
| `type` | string | No | `json` (default) or `txt` (plain text for LLM injection) |
|
||||
@@ -125,7 +125,7 @@ Note: `state: "finalized"` maps from TrueRef's `state: "indexed"` for compatibil
|
||||
|
||||
Plain text formatted for direct LLM context injection:
|
||||
|
||||
```
|
||||
````
|
||||
## Library Rules
|
||||
- Always use functional components
|
||||
- Use hooks for state management
|
||||
@@ -139,15 +139,17 @@ Plain text formatted for direct LLM context injection:
|
||||
function MyComponent() {
|
||||
return <div>Hello</div>;
|
||||
}
|
||||
```
|
||||
````
|
||||
|
||||
---
|
||||
|
||||
### React components let you split the UI...
|
||||
*Core Concepts > Components*
|
||||
|
||||
_Core Concepts > Components_
|
||||
|
||||
React components let you split the UI into independent, reusable pieces...
|
||||
```
|
||||
|
||||
````
|
||||
|
||||
---
|
||||
|
||||
@@ -167,7 +169,7 @@ function parseLibraryId(libraryId: string): {
|
||||
version: match[3],
|
||||
};
|
||||
}
|
||||
```
|
||||
````
|
||||
|
||||
---
|
||||
|
||||
@@ -176,10 +178,7 @@ function parseLibraryId(libraryId: string): {
|
||||
The `tokens` parameter limits the total response size. Snippets are added greedily until the budget is exhausted:
|
||||
|
||||
```typescript
|
||||
function selectSnippetsWithinBudget(
|
||||
snippets: Snippet[],
|
||||
maxTokens: number
|
||||
): Snippet[] {
|
||||
function selectSnippetsWithinBudget(snippets: Snippet[], maxTokens: number): Snippet[] {
|
||||
const selected: Snippet[] = [];
|
||||
let usedTokens = 0;
|
||||
|
||||
@@ -215,6 +214,7 @@ Default token budget: 10,000 tokens (~7,500 words) — enough for ~20 medium sni
|
||||
## CORS Configuration
|
||||
|
||||
All API routes include:
|
||||
|
||||
```
|
||||
Access-Control-Allow-Origin: *
|
||||
Access-Control-Allow-Methods: GET, POST, PATCH, DELETE, OPTIONS
|
||||
|
||||
@@ -46,10 +46,7 @@ Implement a Model Context Protocol (MCP) server that exposes `resolve-library-id
|
||||
|
||||
import { Server } from '@modelcontextprotocol/sdk/server/index.js';
|
||||
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
|
||||
import {
|
||||
CallToolRequestSchema,
|
||||
ListToolsRequestSchema,
|
||||
} from '@modelcontextprotocol/sdk/types.js';
|
||||
import { CallToolRequestSchema, ListToolsRequestSchema } from '@modelcontextprotocol/sdk/types.js';
|
||||
import { z } from 'zod';
|
||||
|
||||
const API_BASE = process.env.TRUEREF_API_URL ?? 'http://localhost:5173';
|
||||
@@ -57,33 +54,29 @@ const API_BASE = process.env.TRUEREF_API_URL ?? 'http://localhost:5173';
|
||||
const server = new Server(
|
||||
{
|
||||
name: 'io.github.trueref/trueref',
|
||||
version: '1.0.0',
|
||||
version: '1.0.0'
|
||||
},
|
||||
{
|
||||
capabilities: { tools: {} },
|
||||
capabilities: { tools: {} }
|
||||
}
|
||||
);
|
||||
|
||||
// Tool schemas — identical to context7 for drop-in compatibility
|
||||
const ResolveLibraryIdSchema = z.object({
|
||||
libraryName: z.string().describe(
|
||||
'Library name to search for and resolve to a TrueRef library ID'
|
||||
),
|
||||
query: z.string().describe(
|
||||
"The user's question or context to help rank results"
|
||||
),
|
||||
libraryName: z
|
||||
.string()
|
||||
.describe('Library name to search for and resolve to a TrueRef library ID'),
|
||||
query: z.string().describe("The user's question or context to help rank results")
|
||||
});
|
||||
|
||||
const QueryDocsSchema = z.object({
|
||||
libraryId: z.string().describe(
|
||||
'The TrueRef library ID obtained from resolve-library-id, e.g. /facebook/react'
|
||||
),
|
||||
query: z.string().describe(
|
||||
'Specific question about the library to retrieve relevant documentation'
|
||||
),
|
||||
tokens: z.number().optional().describe(
|
||||
'Maximum token budget for the response (default: 10000)'
|
||||
),
|
||||
libraryId: z
|
||||
.string()
|
||||
.describe('The TrueRef library ID obtained from resolve-library-id, e.g. /facebook/react'),
|
||||
query: z
|
||||
.string()
|
||||
.describe('Specific question about the library to retrieve relevant documentation'),
|
||||
tokens: z.number().optional().describe('Maximum token budget for the response (default: 10000)')
|
||||
});
|
||||
|
||||
server.setRequestHandler(ListToolsRequestSchema, async () => ({
|
||||
@@ -94,22 +87,22 @@ server.setRequestHandler(ListToolsRequestSchema, async () => ({
|
||||
'Searches TrueRef to find a library matching the given name.',
|
||||
'Returns a list of matching libraries with their IDs.',
|
||||
'ALWAYS call this tool before query-docs to get the correct library ID.',
|
||||
'Call at most 3 times per user question.',
|
||||
'Call at most 3 times per user question.'
|
||||
].join(' '),
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
libraryName: {
|
||||
type: 'string',
|
||||
description: 'Library name to search for',
|
||||
description: 'Library name to search for'
|
||||
},
|
||||
query: {
|
||||
type: 'string',
|
||||
description: "User's question for relevance ranking",
|
||||
},
|
||||
},
|
||||
required: ['libraryName', 'query'],
|
||||
description: "User's question for relevance ranking"
|
||||
}
|
||||
},
|
||||
required: ['libraryName', 'query']
|
||||
}
|
||||
},
|
||||
{
|
||||
name: 'query-docs',
|
||||
@@ -117,28 +110,28 @@ server.setRequestHandler(ListToolsRequestSchema, async () => ({
|
||||
'Fetches documentation and code examples from TrueRef for a specific library.',
|
||||
'Requires a library ID obtained from resolve-library-id.',
|
||||
'Returns relevant snippets formatted for LLM consumption.',
|
||||
'Call at most 3 times per user question.',
|
||||
'Call at most 3 times per user question.'
|
||||
].join(' '),
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
libraryId: {
|
||||
type: 'string',
|
||||
description: 'TrueRef library ID, e.g. /facebook/react',
|
||||
description: 'TrueRef library ID, e.g. /facebook/react'
|
||||
},
|
||||
query: {
|
||||
type: 'string',
|
||||
description: 'Specific question about the library',
|
||||
description: 'Specific question about the library'
|
||||
},
|
||||
tokens: {
|
||||
type: 'number',
|
||||
description: 'Max token budget (default: 10000)',
|
||||
description: 'Max token budget (default: 10000)'
|
||||
}
|
||||
},
|
||||
},
|
||||
required: ['libraryId', 'query'],
|
||||
},
|
||||
},
|
||||
],
|
||||
required: ['libraryId', 'query']
|
||||
}
|
||||
}
|
||||
]
|
||||
}));
|
||||
|
||||
server.setRequestHandler(CallToolRequestSchema, async (request) => {
|
||||
@@ -155,17 +148,19 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
|
||||
const response = await fetch(url.toString());
|
||||
if (!response.ok) {
|
||||
return {
|
||||
content: [{
|
||||
content: [
|
||||
{
|
||||
type: 'text',
|
||||
text: `Error searching libraries: ${response.status} ${response.statusText}`,
|
||||
}],
|
||||
isError: true,
|
||||
text: `Error searching libraries: ${response.status} ${response.statusText}`
|
||||
}
|
||||
],
|
||||
isError: true
|
||||
};
|
||||
}
|
||||
|
||||
const text = await response.text();
|
||||
return {
|
||||
content: [{ type: 'text', text }],
|
||||
content: [{ type: 'text', text }]
|
||||
};
|
||||
}
|
||||
|
||||
@@ -183,40 +178,46 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
|
||||
const status = response.status;
|
||||
if (status === 404) {
|
||||
return {
|
||||
content: [{
|
||||
content: [
|
||||
{
|
||||
type: 'text',
|
||||
text: `Library "${libraryId}" not found. Please run resolve-library-id first.`,
|
||||
}],
|
||||
isError: true,
|
||||
text: `Library "${libraryId}" not found. Please run resolve-library-id first.`
|
||||
}
|
||||
],
|
||||
isError: true
|
||||
};
|
||||
}
|
||||
if (status === 503) {
|
||||
return {
|
||||
content: [{
|
||||
content: [
|
||||
{
|
||||
type: 'text',
|
||||
text: `Library "${libraryId}" is currently being indexed. Please try again in a moment.`,
|
||||
}],
|
||||
isError: true,
|
||||
text: `Library "${libraryId}" is currently being indexed. Please try again in a moment.`
|
||||
}
|
||||
],
|
||||
isError: true
|
||||
};
|
||||
}
|
||||
return {
|
||||
content: [{
|
||||
content: [
|
||||
{
|
||||
type: 'text',
|
||||
text: `Error fetching documentation: ${response.status} ${response.statusText}`,
|
||||
}],
|
||||
isError: true,
|
||||
text: `Error fetching documentation: ${response.status} ${response.statusText}`
|
||||
}
|
||||
],
|
||||
isError: true
|
||||
};
|
||||
}
|
||||
|
||||
const text = await response.text();
|
||||
return {
|
||||
content: [{ type: 'text', text }],
|
||||
content: [{ type: 'text', text }]
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
content: [{ type: 'text', text: `Unknown tool: ${name}` }],
|
||||
isError: true,
|
||||
isError: true
|
||||
};
|
||||
});
|
||||
|
||||
@@ -245,6 +246,7 @@ main().catch((err) => {
|
||||
```
|
||||
|
||||
Or with `tsx` for TypeScript-direct execution:
|
||||
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
@@ -274,6 +276,7 @@ Users add to `.mcp.json`:
|
||||
```
|
||||
|
||||
Or with tsx for development:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
@@ -295,13 +298,15 @@ Or with tsx for development:
|
||||
The MCP server should include a `resources` list item (optional) or the library responses themselves prepend rules. Additionally, users should add a Claude rule file:
|
||||
|
||||
```markdown
|
||||
<!-- .claude/rules/trueref.md -->
|
||||
---
|
||||
## <!-- .claude/rules/trueref.md -->
|
||||
|
||||
description: Use TrueRef to retrieve documentation for indexed libraries
|
||||
alwaysApply: true
|
||||
|
||||
---
|
||||
|
||||
When answering questions about indexed libraries, always use the TrueRef MCP tools:
|
||||
|
||||
1. Call `resolve-library-id` with the library name and the user's question to get the library ID
|
||||
2. Call `query-docs` with the library ID and question to retrieve relevant documentation
|
||||
3. Use the returned documentation to answer the question accurately
|
||||
|
||||
@@ -52,8 +52,8 @@ import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js'
|
||||
const { values: args } = parseArgs({
|
||||
options: {
|
||||
transport: { type: 'string', default: 'stdio' },
|
||||
port: { type: 'string', default: process.env.PORT ?? '3001' },
|
||||
},
|
||||
port: { type: 'string', default: process.env.PORT ?? '3001' }
|
||||
}
|
||||
});
|
||||
|
||||
async function startHttp(server: Server, port: number): Promise<void> {
|
||||
@@ -81,7 +81,7 @@ async function startHttp(server: Server, port: number): Promise<void> {
|
||||
}
|
||||
|
||||
const transport = new StreamableHTTPServerTransport({
|
||||
sessionIdGenerator: () => crypto.randomUUID(),
|
||||
sessionIdGenerator: () => crypto.randomUUID()
|
||||
});
|
||||
|
||||
await server.connect(transport);
|
||||
|
||||
@@ -95,7 +95,7 @@ const CONFIG_CONSTRAINTS = {
|
||||
excludeFiles: { maxItems: 100, maxLength: 200 },
|
||||
rules: { maxItems: 20, minLength: 5, maxLength: 500 },
|
||||
previousVersions: { maxItems: 50 },
|
||||
versionTag: { pattern: /^v?\d+\.\d+(\.\d+)?(-.*)?$/ },
|
||||
versionTag: { pattern: /^v?\d+\.\d+(\.\d+)?(-.*)?$/ }
|
||||
};
|
||||
```
|
||||
|
||||
@@ -172,7 +172,7 @@ export function parseConfigFile(content: string, filename: string): ParsedConfig
|
||||
if (Array.isArray(config.rules)) {
|
||||
validated.rules = (config.rules as unknown[])
|
||||
.filter((r): r is string => typeof r === 'string' && r.length >= 5)
|
||||
.map(r => r.slice(0, 500))
|
||||
.map((r) => r.slice(0, 500))
|
||||
.slice(0, 20);
|
||||
}
|
||||
}
|
||||
@@ -181,8 +181,10 @@ export function parseConfigFile(content: string, filename: string): ParsedConfig
|
||||
if (config.previousVersions !== undefined) {
|
||||
if (Array.isArray(config.previousVersions)) {
|
||||
validated.previousVersions = (config.previousVersions as unknown[])
|
||||
.filter((v): v is { tag: string; title: string } =>
|
||||
typeof v === 'object' && v !== null &&
|
||||
.filter(
|
||||
(v): v is { tag: string; title: string } =>
|
||||
typeof v === 'object' &&
|
||||
v !== null &&
|
||||
typeof (v as Record<string, unknown>).tag === 'string' &&
|
||||
typeof (v as Record<string, unknown>).title === 'string'
|
||||
)
|
||||
@@ -193,7 +195,7 @@ export function parseConfigFile(content: string, filename: string): ParsedConfig
|
||||
return {
|
||||
config: validated,
|
||||
source: filename.startsWith('trueref') ? 'trueref.json' : 'context7.json',
|
||||
warnings,
|
||||
warnings
|
||||
};
|
||||
}
|
||||
```
|
||||
@@ -219,17 +221,11 @@ When `query-docs` returns results, `rules` from `repository_configs` are prepend
|
||||
|
||||
```typescript
|
||||
// In formatters.ts
|
||||
function buildContextResponse(
|
||||
snippets: Snippet[],
|
||||
config: RepositoryConfig | null
|
||||
): string {
|
||||
function buildContextResponse(snippets: Snippet[], config: RepositoryConfig | null): string {
|
||||
const parts: string[] = [];
|
||||
|
||||
if (config?.rules?.length) {
|
||||
parts.push(
|
||||
'## Library Best Practices\n' +
|
||||
config.rules.map(r => `- ${r}`).join('\n')
|
||||
);
|
||||
parts.push('## Library Best Practices\n' + config.rules.map((r) => `- ${r}`).join('\n'));
|
||||
}
|
||||
|
||||
// ... append snippet content
|
||||
|
||||
@@ -45,6 +45,7 @@ Examples:
|
||||
### `GET /api/v1/libs/:id/versions`
|
||||
|
||||
Response `200`:
|
||||
|
||||
```json
|
||||
{
|
||||
"versions": [
|
||||
@@ -64,6 +65,7 @@ Response `200`:
|
||||
### `POST /api/v1/libs/:id/versions`
|
||||
|
||||
Request body:
|
||||
|
||||
```json
|
||||
{
|
||||
"tag": "v18.3.0",
|
||||
@@ -73,6 +75,7 @@ Request body:
|
||||
```
|
||||
|
||||
Response `201`:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": { ...RepositoryVersion },
|
||||
@@ -101,15 +104,14 @@ async function listGitHubTags(
|
||||
token?: string
|
||||
): Promise<Array<{ name: string; commit: { sha: string } }>> {
|
||||
const headers: Record<string, string> = {
|
||||
'Accept': 'application/vnd.github.v3+json',
|
||||
'User-Agent': 'TrueRef/1.0',
|
||||
Accept: 'application/vnd.github.v3+json',
|
||||
'User-Agent': 'TrueRef/1.0'
|
||||
};
|
||||
if (token) headers['Authorization'] = `Bearer ${token}`;
|
||||
|
||||
const response = await fetch(
|
||||
`https://api.github.com/repos/${owner}/${repo}/tags?per_page=100`,
|
||||
{ headers }
|
||||
);
|
||||
const response = await fetch(`https://api.github.com/repos/${owner}/${repo}/tags?per_page=100`, {
|
||||
headers
|
||||
});
|
||||
|
||||
if (!response.ok) throw new GitHubApiError(response.status);
|
||||
return response.json();
|
||||
@@ -135,14 +137,12 @@ function resolveSearchTarget(libraryId: string): {
|
||||
}
|
||||
|
||||
// Look up versionId from tag
|
||||
const versionRecord = db.prepare(
|
||||
`SELECT id FROM repository_versions WHERE repository_id = ? AND tag = ?`
|
||||
).get(repositoryId, version) as { id: string } | undefined;
|
||||
const versionRecord = db
|
||||
.prepare(`SELECT id FROM repository_versions WHERE repository_id = ? AND tag = ?`)
|
||||
.get(repositoryId, version) as { id: string } | undefined;
|
||||
|
||||
if (!versionRecord) {
|
||||
throw new NotFoundError(
|
||||
`Version "${version}" not found for library "${repositoryId}"`
|
||||
);
|
||||
throw new NotFoundError(`Version "${version}" not found for library "${repositoryId}"`);
|
||||
}
|
||||
|
||||
return { repositoryId, versionId: versionRecord.id };
|
||||
@@ -159,18 +159,18 @@ Snippets with `version_id IS NULL` belong to the default branch; snippets with a
|
||||
export class VersionService {
|
||||
constructor(private db: BetterSQLite3.Database) {}
|
||||
|
||||
list(repositoryId: string): RepositoryVersion[]
|
||||
list(repositoryId: string): RepositoryVersion[];
|
||||
|
||||
add(repositoryId: string, tag: string, title?: string): RepositoryVersion
|
||||
add(repositoryId: string, tag: string, title?: string): RepositoryVersion;
|
||||
|
||||
remove(repositoryId: string, tag: string): void
|
||||
remove(repositoryId: string, tag: string): void;
|
||||
|
||||
getByTag(repositoryId: string, tag: string): RepositoryVersion | null
|
||||
getByTag(repositoryId: string, tag: string): RepositoryVersion | null;
|
||||
|
||||
registerFromConfig(
|
||||
repositoryId: string,
|
||||
previousVersions: { tag: string; title: string }[]
|
||||
): RepositoryVersion[]
|
||||
): RepositoryVersion[];
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
@@ -61,14 +61,14 @@ Implement the main web interface for managing repositories. Built with SvelteKit
|
||||
pending: 'bg-gray-100 text-gray-600',
|
||||
indexing: 'bg-blue-100 text-blue-700',
|
||||
indexed: 'bg-green-100 text-green-700',
|
||||
error: 'bg-red-100 text-red-700',
|
||||
error: 'bg-red-100 text-red-700'
|
||||
};
|
||||
|
||||
const stateLabels = {
|
||||
pending: 'Pending',
|
||||
indexing: 'Indexing...',
|
||||
indexed: 'Indexed',
|
||||
error: 'Error',
|
||||
error: 'Error'
|
||||
};
|
||||
</script>
|
||||
|
||||
@@ -150,7 +150,7 @@ Implement the main web interface for managing repositories. Built with SvelteKit
|
||||
const res = await fetch('/api/v1/libs', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ source, sourceUrl, githubToken: githubToken || undefined }),
|
||||
body: JSON.stringify({ source, sourceUrl, githubToken: githubToken || undefined })
|
||||
});
|
||||
if (!res.ok) {
|
||||
const data = await res.json();
|
||||
@@ -172,13 +172,17 @@ Implement the main web interface for managing repositories. Built with SvelteKit
|
||||
|
||||
<div class="mb-4 flex gap-2">
|
||||
<button
|
||||
class="flex-1 rounded-lg py-2 text-sm {source === 'github' ? 'bg-blue-600 text-white' : 'border border-gray-200 text-gray-700'}"
|
||||
onclick={() => source = 'github'}
|
||||
>GitHub</button>
|
||||
class="flex-1 rounded-lg py-2 text-sm {source === 'github'
|
||||
? 'bg-blue-600 text-white'
|
||||
: 'border border-gray-200 text-gray-700'}"
|
||||
onclick={() => (source = 'github')}>GitHub</button
|
||||
>
|
||||
<button
|
||||
class="flex-1 rounded-lg py-2 text-sm {source === 'local' ? 'bg-blue-600 text-white' : 'border border-gray-200 text-gray-700'}"
|
||||
onclick={() => source = 'local'}
|
||||
>Local Path</button>
|
||||
class="flex-1 rounded-lg py-2 text-sm {source === 'local'
|
||||
? 'bg-blue-600 text-white'
|
||||
: 'border border-gray-200 text-gray-700'}"
|
||||
onclick={() => (source = 'local')}>Local Path</button
|
||||
>
|
||||
</div>
|
||||
|
||||
<label class="block">
|
||||
@@ -197,7 +201,9 @@ Implement the main web interface for managing repositories. Built with SvelteKit
|
||||
|
||||
{#if source === 'github'}
|
||||
<label class="mt-3 block">
|
||||
<span class="text-sm font-medium text-gray-700">GitHub Token (optional, for private repos)</span>
|
||||
<span class="text-sm font-medium text-gray-700"
|
||||
>GitHub Token (optional, for private repos)</span
|
||||
>
|
||||
<input
|
||||
type="password"
|
||||
bind:value={githubToken}
|
||||
|
||||
@@ -58,14 +58,20 @@ An interactive search interface within the web UI that lets users test the docum
|
||||
<!-- src/lib/components/search/LibraryResult.svelte -->
|
||||
<script lang="ts">
|
||||
let { result, onSelect } = $props<{
|
||||
result: { id: string; title: string; description: string; totalSnippets: number; trustScore: number };
|
||||
result: {
|
||||
id: string;
|
||||
title: string;
|
||||
description: string;
|
||||
totalSnippets: number;
|
||||
trustScore: number;
|
||||
};
|
||||
onSelect: (id: string) => void;
|
||||
}>();
|
||||
</script>
|
||||
|
||||
<button
|
||||
onclick={() => onSelect(result.id)}
|
||||
class="w-full rounded-xl border border-gray-200 bg-white p-4 text-left shadow-sm hover:border-blue-300 hover:shadow-md transition-all"
|
||||
class="w-full rounded-xl border border-gray-200 bg-white p-4 text-left shadow-sm transition-all hover:border-blue-300 hover:shadow-md"
|
||||
>
|
||||
<div class="flex items-center justify-between">
|
||||
<span class="font-semibold text-gray-900">{result.title}</span>
|
||||
@@ -73,7 +79,7 @@ An interactive search interface within the web UI that lets users test the docum
|
||||
</div>
|
||||
<p class="font-mono text-xs text-gray-400">{result.id}</p>
|
||||
{#if result.description}
|
||||
<p class="mt-1.5 text-sm text-gray-600 line-clamp-2">{result.description}</p>
|
||||
<p class="mt-1.5 line-clamp-2 text-sm text-gray-600">{result.description}</p>
|
||||
{/if}
|
||||
<p class="mt-2 text-xs text-gray-400">{result.totalSnippets.toLocaleString()} snippets</p>
|
||||
</button>
|
||||
@@ -91,7 +97,7 @@ An interactive search interface within the web UI that lets users test the docum
|
||||
let { snippet } = $props<{ snippet: Snippet }>();
|
||||
</script>
|
||||
|
||||
<div class="rounded-xl border border-gray-200 bg-white overflow-hidden">
|
||||
<div class="overflow-hidden rounded-xl border border-gray-200 bg-white">
|
||||
<div class="flex items-center justify-between border-b border-gray-100 px-4 py-2.5">
|
||||
<div class="flex items-center gap-2">
|
||||
{#if snippet.type === 'code'}
|
||||
@@ -112,7 +118,9 @@ An interactive search interface within the web UI that lets users test the docum
|
||||
|
||||
<div class="p-4">
|
||||
{#if snippet.type === 'code'}
|
||||
<pre class="overflow-x-auto rounded bg-gray-950 p-4 text-sm text-gray-100"><code>{snippet.content}</code></pre>
|
||||
<pre class="overflow-x-auto rounded bg-gray-950 p-4 text-sm text-gray-100"><code
|
||||
>{snippet.content}</code
|
||||
></pre>
|
||||
{:else}
|
||||
<div class="prose prose-sm max-w-none text-gray-700">{snippet.content}</div>
|
||||
{/if}
|
||||
@@ -162,7 +170,7 @@ An interactive search interface within the web UI that lets users test the docum
|
||||
// Update URL
|
||||
goto(`/search?lib=${encodeURIComponent(selectedLibraryId)}&q=${encodeURIComponent(query)}`, {
|
||||
replaceState: true,
|
||||
keepFocus: true,
|
||||
keepFocus: true
|
||||
});
|
||||
}
|
||||
</script>
|
||||
|
||||
@@ -40,8 +40,8 @@ function computeDiff(
|
||||
crawledFiles: CrawledFile[],
|
||||
existingDocs: Document[] // documents currently in DB for this repo
|
||||
): FileDiff {
|
||||
const existingMap = new Map(existingDocs.map(d => [d.filePath, d]));
|
||||
const crawledMap = new Map(crawledFiles.map(f => [f.path, f]));
|
||||
const existingMap = new Map(existingDocs.map((d) => [d.filePath, d]));
|
||||
const crawledMap = new Map(crawledFiles.map((f) => [f.path, f]));
|
||||
|
||||
const added: CrawledFile[] = [];
|
||||
const modified: CrawledFile[] = [];
|
||||
@@ -59,8 +59,8 @@ function computeDiff(
|
||||
}
|
||||
|
||||
const deleted = existingDocs
|
||||
.filter(doc => !crawledMap.has(doc.filePath))
|
||||
.map(doc => doc.filePath);
|
||||
.filter((doc) => !crawledMap.has(doc.filePath))
|
||||
.map((doc) => doc.filePath);
|
||||
|
||||
return { added, modified, deleted, unchanged };
|
||||
}
|
||||
@@ -78,7 +78,7 @@ const diff = computeDiff(crawledResult.files, existingDocs);
|
||||
|
||||
// Log diff summary
|
||||
this.updateJob(job.id, {
|
||||
totalFiles: crawledResult.files.length,
|
||||
totalFiles: crawledResult.files.length
|
||||
});
|
||||
|
||||
// Process only changed/new files
|
||||
@@ -89,13 +89,13 @@ const docIdsToDelete: string[] = [];
|
||||
|
||||
// Map modified files to their existing document IDs for deletion
|
||||
for (const file of diff.modified) {
|
||||
const existing = existingDocs.find(d => d.filePath === file.path);
|
||||
const existing = existingDocs.find((d) => d.filePath === file.path);
|
||||
if (existing) docIdsToDelete.push(existing.id);
|
||||
}
|
||||
|
||||
// Map deleted file paths to document IDs
|
||||
for (const filePath of diff.deleted) {
|
||||
const existing = existingDocs.find(d => d.filePath === filePath);
|
||||
const existing = existingDocs.find((d) => d.filePath === filePath);
|
||||
if (existing) docIdsToDelete.push(existing.id);
|
||||
}
|
||||
|
||||
@@ -110,7 +110,7 @@ for (const [i, file] of filesToProcess.entries()) {
|
||||
const progress = Math.round((totalProcessed / crawledResult.files.length) * 80);
|
||||
this.updateJob(job.id, {
|
||||
processedFiles: totalProcessed,
|
||||
progress,
|
||||
progress
|
||||
});
|
||||
}
|
||||
|
||||
@@ -123,6 +123,7 @@ this.replaceSnippets(repo.id, docIdsToDelete, newDocuments, newSnippets);
|
||||
## Performance Impact
|
||||
|
||||
For a typical repository with 1,000 files where 50 changed:
|
||||
|
||||
- **Without incremental**: 1,000 files parsed + 1,000 embed batches
|
||||
- **With incremental**: 50 files parsed + 50 embed batches
|
||||
- Estimated speedup: ~20x for re-indexing
|
||||
|
||||
@@ -36,20 +36,20 @@ const PROVIDER_PRESETS = [
|
||||
name: 'OpenAI',
|
||||
baseUrl: 'https://api.openai.com/v1',
|
||||
model: 'text-embedding-3-small',
|
||||
dimensions: 1536,
|
||||
dimensions: 1536
|
||||
},
|
||||
{
|
||||
name: 'Ollama (local)',
|
||||
baseUrl: 'http://localhost:11434/v1',
|
||||
model: 'nomic-embed-text',
|
||||
dimensions: 768,
|
||||
dimensions: 768
|
||||
},
|
||||
{
|
||||
name: 'Azure OpenAI',
|
||||
baseUrl: 'https://{resource}.openai.azure.com/openai/deployments/{deployment}/v1',
|
||||
model: 'text-embedding-3-small',
|
||||
dimensions: 1536,
|
||||
},
|
||||
dimensions: 1536
|
||||
}
|
||||
];
|
||||
```
|
||||
|
||||
@@ -76,7 +76,7 @@ const PROVIDER_PRESETS = [
|
||||
const res = await fetch('/api/v1/settings/embedding/test', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ provider, openai: { baseUrl, apiKey, model, dimensions } }),
|
||||
body: JSON.stringify({ provider, openai: { baseUrl, apiKey, model, dimensions } })
|
||||
});
|
||||
if (res.ok) {
|
||||
testStatus = 'ok';
|
||||
@@ -96,7 +96,7 @@ const PROVIDER_PRESETS = [
|
||||
await fetch('/api/v1/settings/embedding', {
|
||||
method: 'PUT',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ provider, openai: { baseUrl, apiKey, model, dimensions } }),
|
||||
body: JSON.stringify({ provider, openai: { baseUrl, apiKey, model, dimensions } })
|
||||
});
|
||||
saving = false;
|
||||
}
|
||||
@@ -114,7 +114,7 @@ const PROVIDER_PRESETS = [
|
||||
<div class="mb-4 flex gap-2">
|
||||
{#each ['none', 'openai', 'local'] as p}
|
||||
<button
|
||||
onclick={() => provider = p}
|
||||
onclick={() => (provider = p)}
|
||||
class="rounded-lg px-4 py-2 text-sm {provider === p
|
||||
? 'bg-blue-600 text-white'
|
||||
: 'border border-gray-200 text-gray-700 hover:bg-gray-50'}"
|
||||
@@ -125,7 +125,7 @@ const PROVIDER_PRESETS = [
|
||||
</div>
|
||||
|
||||
{#if provider === 'none'}
|
||||
<div class="rounded-lg bg-amber-50 border border-amber-200 p-3 text-sm text-amber-700">
|
||||
<div class="rounded-lg border border-amber-200 bg-amber-50 p-3 text-sm text-amber-700">
|
||||
Search will use keyword matching only. Results may be less relevant for complex questions.
|
||||
</div>
|
||||
{/if}
|
||||
@@ -133,10 +133,14 @@ const PROVIDER_PRESETS = [
|
||||
{#if provider === 'openai'}
|
||||
<div class="space-y-3">
|
||||
<!-- Preset buttons -->
|
||||
<div class="flex gap-2 flex-wrap">
|
||||
<div class="flex flex-wrap gap-2">
|
||||
{#each PROVIDER_PRESETS as preset}
|
||||
<button
|
||||
onclick={() => { baseUrl = preset.baseUrl; model = preset.model; dimensions = preset.dimensions; }}
|
||||
onclick={() => {
|
||||
baseUrl = preset.baseUrl;
|
||||
model = preset.model;
|
||||
dimensions = preset.dimensions;
|
||||
}}
|
||||
class="rounded border border-gray-200 px-2.5 py-1 text-xs text-gray-600 hover:bg-gray-50"
|
||||
>
|
||||
{preset.name}
|
||||
@@ -146,26 +150,46 @@ const PROVIDER_PRESETS = [
|
||||
|
||||
<label class="block">
|
||||
<span class="text-sm font-medium">Base URL</span>
|
||||
<input type="text" bind:value={baseUrl} class="mt-1 w-full rounded-lg border px-3 py-2 text-sm" />
|
||||
<input
|
||||
type="text"
|
||||
bind:value={baseUrl}
|
||||
class="mt-1 w-full rounded-lg border px-3 py-2 text-sm"
|
||||
/>
|
||||
</label>
|
||||
|
||||
<label class="block">
|
||||
<span class="text-sm font-medium">API Key</span>
|
||||
<input type="password" bind:value={apiKey} class="mt-1 w-full rounded-lg border px-3 py-2 text-sm" placeholder="sk-..." />
|
||||
<input
|
||||
type="password"
|
||||
bind:value={apiKey}
|
||||
class="mt-1 w-full rounded-lg border px-3 py-2 text-sm"
|
||||
placeholder="sk-..."
|
||||
/>
|
||||
</label>
|
||||
|
||||
<label class="block">
|
||||
<span class="text-sm font-medium">Model</span>
|
||||
<input type="text" bind:value={model} class="mt-1 w-full rounded-lg border px-3 py-2 text-sm" />
|
||||
<input
|
||||
type="text"
|
||||
bind:value={model}
|
||||
class="mt-1 w-full rounded-lg border px-3 py-2 text-sm"
|
||||
/>
|
||||
</label>
|
||||
|
||||
<label class="block">
|
||||
<span class="text-sm font-medium">Dimensions (optional override)</span>
|
||||
<input type="number" bind:value={dimensions} class="mt-1 w-full rounded-lg border px-3 py-2 text-sm" />
|
||||
<input
|
||||
type="number"
|
||||
bind:value={dimensions}
|
||||
class="mt-1 w-full rounded-lg border px-3 py-2 text-sm"
|
||||
/>
|
||||
</label>
|
||||
|
||||
<div class="flex items-center gap-3">
|
||||
<button onclick={testConnection} class="rounded-lg border border-gray-300 px-3 py-1.5 text-sm">
|
||||
<button
|
||||
onclick={testConnection}
|
||||
class="rounded-lg border border-gray-300 px-3 py-1.5 text-sm"
|
||||
>
|
||||
{testStatus === 'testing' ? 'Testing...' : 'Test Connection'}
|
||||
</button>
|
||||
{#if testStatus === 'ok'}
|
||||
|
||||
@@ -80,6 +80,7 @@ git -C /path/to/repo archive <commit-hash> | tar -x -C /tmp/trueref-idx/<repo>-<
|
||||
```
|
||||
|
||||
Advantages over `git checkout` or worktrees:
|
||||
|
||||
- Working directory is completely untouched
|
||||
- No `.git` directory in the output (cleaner for parsing)
|
||||
- Temp directory deleted after indexing with no git state to clean up
|
||||
@@ -115,7 +116,7 @@ Allow commit hashes to be pinned explicitly per version, overriding tag resoluti
|
||||
### Edge Cases
|
||||
|
||||
| Case | Handling |
|
||||
|------|----------|
|
||||
| ---------------------------- | ---------------------------------------------------------------------------------- |
|
||||
| Annotated tags | `rev-parse <tag>^{commit}` peels to commit automatically |
|
||||
| Mutable tags (e.g. `latest`) | Re-resolve on re-index; warn in UI if hash has changed |
|
||||
| Branch as version | `rev-parse origin/<branch>^{commit}` gives tip; re-resolves on re-index |
|
||||
@@ -170,7 +171,7 @@ fi
|
||||
Username conventions by server type:
|
||||
|
||||
| Server | HTTPS username | Password |
|
||||
|--------|---------------|----------|
|
||||
| ------------------------------ | --------------------- | --------------------- |
|
||||
| Bitbucket Server / Data Center | `x-token-auth` | HTTP access token |
|
||||
| Bitbucket Cloud | account username | App password |
|
||||
| GitLab (self-hosted or cloud) | `oauth2` | Personal access token |
|
||||
@@ -220,7 +221,7 @@ services:
|
||||
web:
|
||||
build: .
|
||||
ports:
|
||||
- "3000:3000"
|
||||
- '3000:3000'
|
||||
volumes:
|
||||
- trueref-data:/data
|
||||
- ${USERPROFILE}/.ssh:/root/.ssh:ro
|
||||
@@ -228,20 +229,20 @@ services:
|
||||
- ${CORP_CA_CERT}:/certs/corp-ca.crt:ro
|
||||
environment:
|
||||
DATABASE_URL: /data/trueref.db
|
||||
GIT_TOKEN_BITBUCKET: "${BITBUCKET_TOKEN}"
|
||||
GIT_TOKEN_GITLAB: "${GITLAB_TOKEN}"
|
||||
BITBUCKET_HOST: "${BITBUCKET_HOST}"
|
||||
GITLAB_HOST: "${GITLAB_HOST}"
|
||||
GIT_TOKEN_BITBUCKET: '${BITBUCKET_TOKEN}'
|
||||
GIT_TOKEN_GITLAB: '${GITLAB_TOKEN}'
|
||||
BITBUCKET_HOST: '${BITBUCKET_HOST}'
|
||||
GITLAB_HOST: '${GITLAB_HOST}'
|
||||
restart: unless-stopped
|
||||
|
||||
mcp:
|
||||
build: .
|
||||
command: mcp
|
||||
ports:
|
||||
- "3001:3001"
|
||||
- '3001:3001'
|
||||
environment:
|
||||
TRUEREF_API_URL: http://web:3000
|
||||
MCP_PORT: "3001"
|
||||
MCP_PORT: '3001'
|
||||
depends_on:
|
||||
- web
|
||||
restart: unless-stopped
|
||||
|
||||
@@ -9,7 +9,11 @@
|
||||
import { initializeDatabase } from '$lib/server/db/index.js';
|
||||
import { getClient } from '$lib/server/db/client.js';
|
||||
import { initializePipeline } from '$lib/server/pipeline/startup.js';
|
||||
import { EMBEDDING_CONFIG_KEY, createProviderFromConfig, defaultEmbeddingConfig } from '$lib/server/embeddings/factory.js';
|
||||
import {
|
||||
EMBEDDING_CONFIG_KEY,
|
||||
createProviderFromConfig,
|
||||
defaultEmbeddingConfig
|
||||
} from '$lib/server/embeddings/factory.js';
|
||||
import { EmbeddingService } from '$lib/server/embeddings/embedding.service.js';
|
||||
import type { EmbeddingConfig } from '$lib/server/embeddings/factory.js';
|
||||
import type { Handle } from '@sveltejs/kit';
|
||||
|
||||
@@ -115,7 +115,12 @@
|
||||
/>
|
||||
{:else}
|
||||
<div class="mt-1">
|
||||
<FolderPicker bind:value={sourceUrl} onselect={(p) => { if (!title) title = p.split('/').at(-1) ?? ''; }} />
|
||||
<FolderPicker
|
||||
bind:value={sourceUrl}
|
||||
onselect={(p) => {
|
||||
if (!title) title = p.split('/').at(-1) ?? '';
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
@@ -133,7 +138,8 @@
|
||||
{#if source === 'github'}
|
||||
<label class="block">
|
||||
<span class="text-sm font-medium text-gray-700"
|
||||
>GitHub Token <span class="font-normal text-gray-500">(optional, for private repos)</span
|
||||
>GitHub Token <span class="font-normal text-gray-500"
|
||||
>(optional, for private repos)</span
|
||||
></span
|
||||
>
|
||||
<input
|
||||
|
||||
@@ -78,9 +78,7 @@
|
||||
title="Browse folders"
|
||||
>
|
||||
<svg class="h-4 w-4" viewBox="0 0 20 20" fill="currentColor">
|
||||
<path
|
||||
d="M2 6a2 2 0 012-2h5l2 2h5a2 2 0 012 2v6a2 2 0 01-2 2H4a2 2 0 01-2-2V6z"
|
||||
/>
|
||||
<path d="M2 6a2 2 0 012-2h5l2 2h5a2 2 0 012 2v6a2 2 0 01-2 2H4a2 2 0 01-2-2V6z" />
|
||||
</svg>
|
||||
Browse
|
||||
</button>
|
||||
@@ -94,7 +92,10 @@
|
||||
class="fixed inset-0 z-[60] flex items-center justify-center bg-black/50 p-4"
|
||||
onclick={handleBackdropClick}
|
||||
>
|
||||
<div class="flex w-full max-w-lg flex-col rounded-xl bg-white shadow-xl" style="max-height: 70vh">
|
||||
<div
|
||||
class="flex w-full max-w-lg flex-col rounded-xl bg-white shadow-xl"
|
||||
style="max-height: 70vh"
|
||||
>
|
||||
<!-- Header -->
|
||||
<div class="flex items-center justify-between border-b border-gray-200 px-4 py-3">
|
||||
<h3 class="text-sm font-semibold text-gray-900">Select Folder</h3>
|
||||
@@ -133,11 +134,7 @@
|
||||
{browsePath}
|
||||
</span>
|
||||
{#if loading}
|
||||
<svg
|
||||
class="h-4 w-4 animate-spin text-gray-400"
|
||||
fill="none"
|
||||
viewBox="0 0 24 24"
|
||||
>
|
||||
<svg class="h-4 w-4 animate-spin text-gray-400" fill="none" viewBox="0 0 24 24">
|
||||
<circle class="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" stroke-width="4"
|
||||
></circle>
|
||||
<path
|
||||
@@ -166,7 +163,9 @@
|
||||
title="Click to navigate, double-click to select"
|
||||
>
|
||||
<svg
|
||||
class="h-4 w-4 shrink-0 {entry.isGitRepo ? 'text-orange-400' : 'text-yellow-400'}"
|
||||
class="h-4 w-4 shrink-0 {entry.isGitRepo
|
||||
? 'text-orange-400'
|
||||
: 'text-yellow-400'}"
|
||||
viewBox="0 0 20 20"
|
||||
fill="currentColor"
|
||||
>
|
||||
@@ -176,7 +175,9 @@
|
||||
</svg>
|
||||
<span class="flex-1 truncate text-gray-800">{entry.name}</span>
|
||||
{#if entry.isGitRepo}
|
||||
<span class="shrink-0 rounded bg-orange-100 px-1.5 py-0.5 text-xs text-orange-700">
|
||||
<span
|
||||
class="shrink-0 rounded bg-orange-100 px-1.5 py-0.5 text-xs text-orange-700"
|
||||
>
|
||||
git
|
||||
</span>
|
||||
{/if}
|
||||
|
||||
@@ -36,8 +36,9 @@
|
||||
<p class="mt-0.5 truncate font-mono text-sm text-gray-500">{repo.id}</p>
|
||||
</div>
|
||||
<span
|
||||
class="ml-3 shrink-0 rounded-full px-2.5 py-0.5 text-xs font-medium {stateColors[repo.state] ??
|
||||
'bg-gray-100 text-gray-600'}"
|
||||
class="ml-3 shrink-0 rounded-full px-2.5 py-0.5 text-xs font-medium {stateColors[
|
||||
repo.state
|
||||
] ?? 'bg-gray-100 text-gray-600'}"
|
||||
>
|
||||
{stateLabels[repo.state] ?? repo.state}
|
||||
</span>
|
||||
|
||||
@@ -17,7 +17,10 @@
|
||||
};
|
||||
</script>
|
||||
|
||||
<div class="flex flex-col items-center rounded-lg p-3 {variantClasses[variant] ?? variantClasses.default}">
|
||||
<div
|
||||
class="flex flex-col items-center rounded-lg p-3 {variantClasses[variant] ??
|
||||
variantClasses.default}"
|
||||
>
|
||||
<span class="text-lg font-bold">{value}</span>
|
||||
<span class="mt-0.5 text-xs">{label}</span>
|
||||
</div>
|
||||
|
||||
@@ -17,6 +17,8 @@
|
||||
const config = $derived(statusConfig[status]);
|
||||
</script>
|
||||
|
||||
<span class="inline-flex items-center rounded-full px-2.5 py-0.5 text-xs font-medium {config.bg} {config.text}">
|
||||
<span
|
||||
class="inline-flex items-center rounded-full px-2.5 py-0.5 text-xs font-medium {config.bg} {config.text}"
|
||||
>
|
||||
{config.label}
|
||||
</span>
|
||||
|
||||
@@ -25,7 +25,7 @@
|
||||
{placeholder}
|
||||
onkeydown={handleKeydown}
|
||||
disabled={loading}
|
||||
class="flex-1 rounded-lg border border-gray-200 bg-white px-4 py-2.5 text-sm text-gray-900 placeholder-gray-400 shadow-sm outline-none transition-all focus:border-blue-400 focus:ring-2 focus:ring-blue-100 disabled:cursor-not-allowed disabled:opacity-60"
|
||||
class="flex-1 rounded-lg border border-gray-200 bg-white px-4 py-2.5 text-sm text-gray-900 placeholder-gray-400 shadow-sm transition-all outline-none focus:border-blue-400 focus:ring-2 focus:ring-blue-100 disabled:cursor-not-allowed disabled:opacity-60"
|
||||
/>
|
||||
<button
|
||||
onclick={onsubmit}
|
||||
|
||||
@@ -5,21 +5,15 @@
|
||||
|
||||
const isCode = $derived(snippet.type === 'code');
|
||||
|
||||
const title = $derived(
|
||||
snippet.type === 'code' ? snippet.title : null
|
||||
);
|
||||
const title = $derived(snippet.type === 'code' ? snippet.title : null);
|
||||
|
||||
const breadcrumb = $derived(
|
||||
snippet.type === 'code' ? snippet.description : snippet.breadcrumb
|
||||
);
|
||||
const breadcrumb = $derived(snippet.type === 'code' ? snippet.description : snippet.breadcrumb);
|
||||
|
||||
const content = $derived(
|
||||
snippet.type === 'code' ? snippet.codeList[0]?.code ?? '' : snippet.text
|
||||
snippet.type === 'code' ? (snippet.codeList[0]?.code ?? '') : snippet.text
|
||||
);
|
||||
|
||||
const language = $derived(
|
||||
snippet.type === 'code' ? (snippet.codeList[0]?.language ?? '') : null
|
||||
);
|
||||
const language = $derived(snippet.type === 'code' ? (snippet.codeList[0]?.language ?? '') : null);
|
||||
|
||||
const tokenCount = $derived(snippet.tokenCount ?? 0);
|
||||
</script>
|
||||
@@ -40,14 +34,14 @@
|
||||
</div>
|
||||
|
||||
{#if breadcrumb}
|
||||
<p class="bg-gray-50 px-4 py-1.5 text-xs italic text-gray-500">{breadcrumb}</p>
|
||||
<p class="bg-gray-50 px-4 py-1.5 text-xs text-gray-500 italic">{breadcrumb}</p>
|
||||
{/if}
|
||||
|
||||
<div class="p-4">
|
||||
{#if isCode}
|
||||
<pre
|
||||
class="overflow-x-auto rounded bg-gray-950 p-4 text-sm text-gray-100"
|
||||
><code>{content}</code></pre>
|
||||
<pre class="overflow-x-auto rounded bg-gray-950 p-4 text-sm text-gray-100"><code
|
||||
>{content}</code
|
||||
></pre>
|
||||
{:else}
|
||||
<div class="prose prose-sm max-w-none whitespace-pre-wrap text-gray-700">{content}</div>
|
||||
{/if}
|
||||
|
||||
@@ -83,9 +83,7 @@ function makeSnippetResult(snippet: Snippet): SnippetSearchResult {
|
||||
});
|
||||
}
|
||||
|
||||
function makeMetadata(
|
||||
overrides: Partial<ContextResponseMetadata> = {}
|
||||
): ContextResponseMetadata {
|
||||
function makeMetadata(overrides: Partial<ContextResponseMetadata> = {}): ContextResponseMetadata {
|
||||
return {
|
||||
localSource: false,
|
||||
resultCount: 1,
|
||||
@@ -160,7 +158,11 @@ describe('formatLibrarySearchJson', () => {
|
||||
|
||||
it('maps non-indexed state to initial', () => {
|
||||
const results: LibrarySearchResult[] = [
|
||||
new LibrarySearchResult({ repository: makeRepo({ state: 'pending' }), versions: [], score: 0 })
|
||||
new LibrarySearchResult({
|
||||
repository: makeRepo({ state: 'pending' }),
|
||||
versions: [],
|
||||
score: 0
|
||||
})
|
||||
];
|
||||
const response = formatLibrarySearchJson(results);
|
||||
expect(response.results[0].state).toBe('initial');
|
||||
@@ -168,7 +170,11 @@ describe('formatLibrarySearchJson', () => {
|
||||
|
||||
it('handles null lastIndexedAt', () => {
|
||||
const results: LibrarySearchResult[] = [
|
||||
new LibrarySearchResult({ repository: makeRepo({ lastIndexedAt: null }), versions: [], score: 0 })
|
||||
new LibrarySearchResult({
|
||||
repository: makeRepo({ lastIndexedAt: null }),
|
||||
versions: [],
|
||||
score: 0
|
||||
})
|
||||
];
|
||||
const response = formatLibrarySearchJson(results);
|
||||
expect(response.results[0].lastUpdateDate).toBeNull();
|
||||
|
||||
@@ -66,7 +66,9 @@ export const CORS_HEADERS = {
|
||||
/**
|
||||
* Convert internal LibrarySearchResult[] to the context7-compatible JSON body.
|
||||
*/
|
||||
export function formatLibrarySearchJson(results: LibrarySearchResult[]): LibrarySearchJsonResponseDto {
|
||||
export function formatLibrarySearchJson(
|
||||
results: LibrarySearchResult[]
|
||||
): LibrarySearchJsonResponseDto {
|
||||
return ContextResponseMapper.toLibrarySearchJson(results);
|
||||
}
|
||||
|
||||
@@ -94,7 +96,10 @@ export function formatContextJson(
|
||||
* @param snippets - Ranked snippet search results (already token-budget trimmed).
|
||||
* @param rules - Rules from `trueref.json` / `repository_configs`.
|
||||
*/
|
||||
function formatOriginLine(result: SnippetSearchResult, metadata?: ContextResponseMetadata): string | null {
|
||||
function formatOriginLine(
|
||||
result: SnippetSearchResult,
|
||||
metadata?: ContextResponseMetadata
|
||||
): string | null {
|
||||
if (!metadata?.repository) return null;
|
||||
|
||||
const parts = [
|
||||
|
||||
@@ -115,10 +115,7 @@ describe('parseConfigFile — description', () => {
|
||||
|
||||
describe('parseConfigFile — array path fields', () => {
|
||||
it('accepts valid folders', () => {
|
||||
const result = parseConfigFile(
|
||||
JSON.stringify({ folders: ['src/', 'docs/'] }),
|
||||
'trueref.json'
|
||||
);
|
||||
const result = parseConfigFile(JSON.stringify({ folders: ['src/', 'docs/'] }), 'trueref.json');
|
||||
expect(result.config.folders).toEqual(['src/', 'docs/']);
|
||||
expect(result.warnings).toHaveLength(0);
|
||||
});
|
||||
@@ -130,10 +127,7 @@ describe('parseConfigFile — array path fields', () => {
|
||||
});
|
||||
|
||||
it('skips non-string entries in folders with a warning', () => {
|
||||
const result = parseConfigFile(
|
||||
JSON.stringify({ folders: ['src/', 42, true] }),
|
||||
'trueref.json'
|
||||
);
|
||||
const result = parseConfigFile(JSON.stringify({ folders: ['src/', 42, true] }), 'trueref.json');
|
||||
expect(result.config.folders).toEqual(['src/']);
|
||||
expect(result.warnings.length).toBeGreaterThan(0);
|
||||
});
|
||||
@@ -174,7 +168,9 @@ describe('parseConfigFile — array path fields', () => {
|
||||
describe('parseConfigFile — rules', () => {
|
||||
it('accepts valid rules', () => {
|
||||
const result = parseConfigFile(
|
||||
JSON.stringify({ rules: ['Always use named imports.', 'Prefer async/await over callbacks.'] }),
|
||||
JSON.stringify({
|
||||
rules: ['Always use named imports.', 'Prefer async/await over callbacks.']
|
||||
}),
|
||||
'trueref.json'
|
||||
);
|
||||
expect(result.config.rules).toHaveLength(2);
|
||||
@@ -204,10 +200,7 @@ describe('parseConfigFile — rules', () => {
|
||||
});
|
||||
|
||||
it('ignores non-array rules with a warning', () => {
|
||||
const result = parseConfigFile(
|
||||
JSON.stringify({ rules: 'use named imports' }),
|
||||
'trueref.json'
|
||||
);
|
||||
const result = parseConfigFile(JSON.stringify({ rules: 'use named imports' }), 'trueref.json');
|
||||
expect(result.config.rules).toBeUndefined();
|
||||
expect(result.warnings.some((w) => /rules must be an array/.test(w))).toBe(true);
|
||||
});
|
||||
@@ -243,10 +236,7 @@ describe('parseConfigFile — previousVersions', () => {
|
||||
it('skips entries missing tag', () => {
|
||||
const result = parseConfigFile(
|
||||
JSON.stringify({
|
||||
previousVersions: [
|
||||
{ title: 'No tag here' },
|
||||
{ tag: 'v1.0.0', title: 'Valid' }
|
||||
]
|
||||
previousVersions: [{ title: 'No tag here' }, { tag: 'v1.0.0', title: 'Valid' }]
|
||||
}),
|
||||
'trueref.json'
|
||||
);
|
||||
@@ -275,10 +265,7 @@ describe('parseConfigFile — previousVersions', () => {
|
||||
});
|
||||
|
||||
it('ignores non-array previousVersions with a warning', () => {
|
||||
const result = parseConfigFile(
|
||||
JSON.stringify({ previousVersions: 'v1.0.0' }),
|
||||
'trueref.json'
|
||||
);
|
||||
const result = parseConfigFile(JSON.stringify({ previousVersions: 'v1.0.0' }), 'trueref.json');
|
||||
expect(result.config.previousVersions).toBeUndefined();
|
||||
expect(result.warnings.some((w) => /previousVersions must be an array/.test(w))).toBe(true);
|
||||
});
|
||||
@@ -294,9 +281,7 @@ describe('resolveConfig', () => {
|
||||
});
|
||||
|
||||
it('returns null when no matching filenames', () => {
|
||||
expect(
|
||||
resolveConfig([{ filename: 'package.json', content: '{"name":"x"}' }])
|
||||
).toBeNull();
|
||||
expect(resolveConfig([{ filename: 'package.json', content: '{"name":"x"}' }])).toBeNull();
|
||||
});
|
||||
|
||||
it('prefers trueref.json over context7.json', () => {
|
||||
|
||||
@@ -65,7 +65,9 @@ export function parseConfigFile(content: string, filename: string): ParsedConfig
|
||||
|
||||
// ---- 2. Root must be an object ------------------------------------------
|
||||
if (typeof raw !== 'object' || raw === null || Array.isArray(raw)) {
|
||||
throw new ConfigParseError(`${filename} must be a JSON object, got ${Array.isArray(raw) ? 'array' : typeof raw}`);
|
||||
throw new ConfigParseError(
|
||||
`${filename} must be a JSON object, got ${Array.isArray(raw) ? 'array' : typeof raw}`
|
||||
);
|
||||
}
|
||||
|
||||
const input = raw as Record<string, unknown>;
|
||||
@@ -131,7 +133,9 @@ export function parseConfigFile(content: string, filename: string): ParsedConfig
|
||||
})
|
||||
.map((item) => {
|
||||
if (item.length > maxLength) {
|
||||
warnings.push(`${field} entry truncated to ${maxLength} characters: "${item.slice(0, 40)}..."`);
|
||||
warnings.push(
|
||||
`${field} entry truncated to ${maxLength} characters: "${item.slice(0, 40)}..."`
|
||||
);
|
||||
return item.slice(0, maxLength);
|
||||
}
|
||||
return item;
|
||||
@@ -160,9 +164,7 @@ export function parseConfigFile(content: string, filename: string): ParsedConfig
|
||||
return false;
|
||||
}
|
||||
if (r.length < minLength) {
|
||||
warnings.push(
|
||||
`rules entry too short (< ${minLength} chars) — skipping: "${r}"`
|
||||
);
|
||||
warnings.push(`rules entry too short (< ${minLength} chars) — skipping: "${r}"`);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
|
||||
@@ -74,27 +74,46 @@ export const MAX_FILE_SIZE_BYTES = 500_000;
|
||||
*/
|
||||
export const IGNORED_DIR_NAMES = new Set([
|
||||
// ── Version control ────────────────────────────────────────────────────
|
||||
'.git', '.hg', '.svn',
|
||||
'.git',
|
||||
'.hg',
|
||||
'.svn',
|
||||
|
||||
// ── JavaScript / TypeScript ─────────────────────────────────────────────
|
||||
'node_modules',
|
||||
'.npm', '.yarn', '.pnpm-store', '.pnp',
|
||||
'.npm',
|
||||
'.yarn',
|
||||
'.pnpm-store',
|
||||
'.pnp',
|
||||
// Build outputs and framework caches
|
||||
'dist', 'build', 'out',
|
||||
'.next', '.nuxt', '.svelte-kit', '.vite',
|
||||
'.turbo', '.parcel-cache', '.webpack',
|
||||
'dist',
|
||||
'build',
|
||||
'out',
|
||||
'.next',
|
||||
'.nuxt',
|
||||
'.svelte-kit',
|
||||
'.vite',
|
||||
'.turbo',
|
||||
'.parcel-cache',
|
||||
'.webpack',
|
||||
|
||||
// ── Python ──────────────────────────────────────────────────────────────
|
||||
'__pycache__',
|
||||
'.venv', 'venv', 'env',
|
||||
'site-packages', '.eggs',
|
||||
'.pytest_cache', '.mypy_cache', '.ruff_cache',
|
||||
'.tox', '.nox',
|
||||
'.venv',
|
||||
'venv',
|
||||
'env',
|
||||
'site-packages',
|
||||
'.eggs',
|
||||
'.pytest_cache',
|
||||
'.mypy_cache',
|
||||
'.ruff_cache',
|
||||
'.tox',
|
||||
'.nox',
|
||||
'htmlcov',
|
||||
|
||||
// ── Java / Kotlin / Scala ───────────────────────────────────────────────
|
||||
'target', // Maven + sbt
|
||||
'.gradle', '.mvn',
|
||||
'.gradle',
|
||||
'.mvn',
|
||||
|
||||
// ── Ruby ────────────────────────────────────────────────────────────────
|
||||
'.bundle',
|
||||
@@ -103,19 +122,24 @@ export const IGNORED_DIR_NAMES = new Set([
|
||||
// 'vendor' below covers PHP Composer
|
||||
|
||||
// ── .NET ────────────────────────────────────────────────────────────────
|
||||
'bin', 'obj', 'packages',
|
||||
'bin',
|
||||
'obj',
|
||||
'packages',
|
||||
|
||||
// ── Haskell ─────────────────────────────────────────────────────────────
|
||||
'.stack-work', 'dist-newstyle',
|
||||
'.stack-work',
|
||||
'dist-newstyle',
|
||||
|
||||
// ── Dart / Flutter ──────────────────────────────────────────────────────
|
||||
'.dart_tool',
|
||||
|
||||
// ── Swift / iOS ─────────────────────────────────────────────────────────
|
||||
'Pods', 'DerivedData',
|
||||
'Pods',
|
||||
'DerivedData',
|
||||
|
||||
// ── Elixir / Erlang ─────────────────────────────────────────────────────
|
||||
'_build', 'deps',
|
||||
'_build',
|
||||
'deps',
|
||||
|
||||
// ── Clojure ─────────────────────────────────────────────────────────────
|
||||
'.cpcache',
|
||||
@@ -125,16 +149,25 @@ export const IGNORED_DIR_NAMES = new Set([
|
||||
'vendor',
|
||||
|
||||
// ── Generic caches / temp ───────────────────────────────────────────────
|
||||
'.cache', '.tmp', 'tmp', 'temp', '.temp', '.sass-cache',
|
||||
'.cache',
|
||||
'.tmp',
|
||||
'tmp',
|
||||
'temp',
|
||||
'.temp',
|
||||
'.sass-cache',
|
||||
|
||||
// ── Test coverage ───────────────────────────────────────────────────────
|
||||
'coverage', '.nyc_output',
|
||||
'coverage',
|
||||
'.nyc_output',
|
||||
|
||||
// ── IDE / editor artefacts ──────────────────────────────────────────────
|
||||
'.idea', '.vs',
|
||||
'.idea',
|
||||
'.vs',
|
||||
|
||||
// ── Generated code ──────────────────────────────────────────────────────
|
||||
'generated', '__generated__', '_generated',
|
||||
'generated',
|
||||
'__generated__',
|
||||
'_generated',
|
||||
|
||||
// ── Logs ────────────────────────────────────────────────────────────────
|
||||
'logs'
|
||||
@@ -264,11 +297,7 @@ export function detectLanguage(filePath: string): string {
|
||||
* 7. Must not be under a config.excludeFolders path / regex.
|
||||
* 8. Must be under a config.folders allowlist path / regex (if specified).
|
||||
*/
|
||||
export function shouldIndexFile(
|
||||
filePath: string,
|
||||
fileSize: number,
|
||||
config?: RepoConfig
|
||||
): boolean {
|
||||
export function shouldIndexFile(filePath: string, fileSize: number, config?: RepoConfig): boolean {
|
||||
const ext = extname(filePath).toLowerCase();
|
||||
const base = basename(filePath);
|
||||
|
||||
|
||||
@@ -35,10 +35,9 @@ export async function listGitHubTags(
|
||||
};
|
||||
if (token) headers['Authorization'] = `Bearer ${token}`;
|
||||
|
||||
const response = await fetch(
|
||||
`https://api.github.com/repos/${owner}/${repo}/tags?per_page=100`,
|
||||
{ headers }
|
||||
);
|
||||
const response = await fetch(`https://api.github.com/repos/${owner}/${repo}/tags?per_page=100`, {
|
||||
headers
|
||||
});
|
||||
|
||||
if (!response.ok) throw new GitHubApiError(response.status);
|
||||
return response.json() as Promise<GitHubTag[]>;
|
||||
|
||||
@@ -8,13 +8,14 @@
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||
|
||||
import { crawl } from './github.crawler.js';
|
||||
import { shouldIndexFile, detectLanguage, INDEXABLE_EXTENSIONS, MAX_FILE_SIZE_BYTES } from './file-filter.js';
|
||||
import { GitHubRateLimiter, Semaphore, withRetry } from './rate-limiter.js';
|
||||
import {
|
||||
AuthenticationError,
|
||||
PermissionError,
|
||||
RepositoryNotFoundError
|
||||
} from './types.js';
|
||||
shouldIndexFile,
|
||||
detectLanguage,
|
||||
INDEXABLE_EXTENSIONS,
|
||||
MAX_FILE_SIZE_BYTES
|
||||
} from './file-filter.js';
|
||||
import { GitHubRateLimiter, Semaphore, withRetry } from './rate-limiter.js';
|
||||
import { AuthenticationError, PermissionError, RepositoryNotFoundError } from './types.js';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Mock fetch helpers
|
||||
@@ -112,7 +113,9 @@ describe('shouldIndexFile()', () => {
|
||||
});
|
||||
|
||||
it('respects config.excludeFolders prefix', () => {
|
||||
expect(shouldIndexFile('internal/config.ts', 100, { excludeFolders: ['internal/'] })).toBe(false);
|
||||
expect(shouldIndexFile('internal/config.ts', 100, { excludeFolders: ['internal/'] })).toBe(
|
||||
false
|
||||
);
|
||||
});
|
||||
|
||||
it('allows files outside of config.excludeFolders', () => {
|
||||
@@ -169,8 +172,10 @@ describe('detectLanguage()', () => {
|
||||
it('detects markdown', () => expect(detectLanguage('README.md')).toBe('markdown'));
|
||||
it('detects svelte', () => expect(detectLanguage('App.svelte')).toBe('svelte'));
|
||||
it('detects yaml', () => expect(detectLanguage('config.yaml')).toBe('yaml'));
|
||||
it('returns empty string for unknown extension', () => expect(detectLanguage('file.xyz')).toBe(''));
|
||||
it('is case-insensitive for extensions', () => expect(detectLanguage('FILE.TS')).toBe('typescript'));
|
||||
it('returns empty string for unknown extension', () =>
|
||||
expect(detectLanguage('file.xyz')).toBe(''));
|
||||
it('is case-insensitive for extensions', () =>
|
||||
expect(detectLanguage('FILE.TS')).toBe('typescript'));
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -267,9 +272,9 @@ describe('withRetry()', () => {
|
||||
});
|
||||
|
||||
it('throws after exhausting all attempts', async () => {
|
||||
await expect(
|
||||
withRetry(() => Promise.reject(new Error('always fails')), 3)
|
||||
).rejects.toThrow('always fails');
|
||||
await expect(withRetry(() => Promise.reject(new Error('always fails')), 3)).rejects.toThrow(
|
||||
'always fails'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -425,7 +430,8 @@ describe('crawl()', () => {
|
||||
});
|
||||
|
||||
it('throws AuthenticationError on 401', async () => {
|
||||
stubFetch(() =>
|
||||
stubFetch(
|
||||
() =>
|
||||
new Response('Unauthorized', {
|
||||
status: 401,
|
||||
headers: {
|
||||
@@ -441,7 +447,8 @@ describe('crawl()', () => {
|
||||
});
|
||||
|
||||
it('throws PermissionError on 403 without rate-limit exhaustion', async () => {
|
||||
stubFetch(() =>
|
||||
stubFetch(
|
||||
() =>
|
||||
new Response('Forbidden', {
|
||||
status: 403,
|
||||
headers: {
|
||||
|
||||
@@ -106,9 +106,7 @@ async function throwForStatus(response: Response, rateLimiter: GitHubRateLimiter
|
||||
);
|
||||
}
|
||||
case 404:
|
||||
throw new RepositoryNotFoundError(
|
||||
`Repository not found or not accessible: ${response.url}`
|
||||
);
|
||||
throw new RepositoryNotFoundError(`Repository not found or not accessible: ${response.url}`);
|
||||
default: {
|
||||
const body = await response.text().catch(() => '');
|
||||
throw new Error(`GitHub API error ${response.status}: ${body}`);
|
||||
@@ -129,7 +127,8 @@ async function fetchRepoInfo(
|
||||
token: string | undefined,
|
||||
rateLimiter: GitHubRateLimiter
|
||||
): Promise<GitHubRepoResponse> {
|
||||
return withRetry(async () => {
|
||||
return withRetry(
|
||||
async () => {
|
||||
await rateLimiter.waitIfNeeded();
|
||||
|
||||
const response = await fetch(`${GITHUB_API}/repos/${owner}/${repo}`, {
|
||||
@@ -140,7 +139,10 @@ async function fetchRepoInfo(
|
||||
await throwForStatus(response, rateLimiter);
|
||||
|
||||
return (await response.json()) as GitHubRepoResponse;
|
||||
}, 3, isRetryable);
|
||||
},
|
||||
3,
|
||||
isRetryable
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -155,7 +157,8 @@ async function fetchTree(
|
||||
token: string | undefined,
|
||||
rateLimiter: GitHubRateLimiter
|
||||
): Promise<GitHubTreeResponse | null> {
|
||||
return withRetry(async () => {
|
||||
return withRetry(
|
||||
async () => {
|
||||
await rateLimiter.waitIfNeeded();
|
||||
|
||||
const url = `${GITHUB_API}/repos/${owner}/${repo}/git/trees/${ref}?recursive=1`;
|
||||
@@ -169,7 +172,10 @@ async function fetchTree(
|
||||
await throwForStatus(response, rateLimiter);
|
||||
|
||||
return (await response.json()) as GitHubTreeResponse;
|
||||
}, 3, isRetryable);
|
||||
},
|
||||
3,
|
||||
isRetryable
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -184,7 +190,8 @@ async function fetchSubTree(
|
||||
token: string | undefined,
|
||||
rateLimiter: GitHubRateLimiter
|
||||
): Promise<GitHubTreeResponse> {
|
||||
return withRetry(async () => {
|
||||
return withRetry(
|
||||
async () => {
|
||||
await rateLimiter.waitIfNeeded();
|
||||
|
||||
const url = `${GITHUB_API}/repos/${owner}/${repo}/git/trees/${treeSha}`;
|
||||
@@ -194,7 +201,10 @@ async function fetchSubTree(
|
||||
await throwForStatus(response, rateLimiter);
|
||||
|
||||
return (await response.json()) as GitHubTreeResponse;
|
||||
}, 3, isRetryable);
|
||||
},
|
||||
3,
|
||||
isRetryable
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -208,7 +218,8 @@ async function fetchCommitSha(
|
||||
token: string | undefined,
|
||||
rateLimiter: GitHubRateLimiter
|
||||
): Promise<string> {
|
||||
return withRetry(async () => {
|
||||
return withRetry(
|
||||
async () => {
|
||||
await rateLimiter.waitIfNeeded();
|
||||
|
||||
const url = `${GITHUB_API}/repos/${owner}/${repo}/commits/${ref}`;
|
||||
@@ -222,7 +233,10 @@ async function fetchCommitSha(
|
||||
// When Accept is 'application/vnd.github.sha', the response body is the
|
||||
// bare SHA string.
|
||||
return (await response.text()).trim();
|
||||
}, 3, isRetryable);
|
||||
},
|
||||
3,
|
||||
isRetryable
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -347,14 +361,7 @@ async function fetchRepoConfig(
|
||||
|
||||
const content =
|
||||
(await downloadRawFile(owner, repo, ref, configItem.path, token)) ??
|
||||
(await downloadViaContentsApi(
|
||||
owner,
|
||||
repo,
|
||||
ref,
|
||||
configItem.path,
|
||||
token,
|
||||
rateLimiter
|
||||
));
|
||||
(await downloadViaContentsApi(owner, repo, ref, configItem.path, token, rateLimiter));
|
||||
|
||||
if (!content) return undefined;
|
||||
|
||||
@@ -435,14 +442,7 @@ export async function crawl(options: CrawlOptions): Promise<CrawlResult> {
|
||||
// Prefer raw download (cheaper on rate limit); fall back to API.
|
||||
const content =
|
||||
(await downloadRawFile(owner, repo, ref!, item.path, token)) ??
|
||||
(await downloadViaContentsApi(
|
||||
owner,
|
||||
repo,
|
||||
ref!,
|
||||
item.path,
|
||||
token,
|
||||
rateLimiter
|
||||
));
|
||||
(await downloadViaContentsApi(owner, repo, ref!, item.path, token, rateLimiter));
|
||||
|
||||
if (content === null) {
|
||||
console.warn(`[GitHubCrawler] Could not download: ${item.path} — skipping.`);
|
||||
|
||||
@@ -52,7 +52,9 @@ async function cleanupTempRepo(root: string): Promise<void> {
|
||||
let root: string = '';
|
||||
const crawler = new LocalCrawler();
|
||||
|
||||
async function crawlRoot(opts: Partial<LocalCrawlOptions> = {}): Promise<ReturnType<LocalCrawler['crawl']>> {
|
||||
async function crawlRoot(
|
||||
opts: Partial<LocalCrawlOptions> = {}
|
||||
): Promise<ReturnType<LocalCrawler['crawl']>> {
|
||||
return crawler.crawl({ rootPath: root, ...opts });
|
||||
}
|
||||
|
||||
|
||||
@@ -141,7 +141,12 @@ export class LocalCrawler {
|
||||
});
|
||||
|
||||
// Crawl the worktree and stamp the result with the git-resolved metadata.
|
||||
const result = await this.crawlDirectory(worktreePath, options.config, options.onProgress, ref);
|
||||
const result = await this.crawlDirectory(
|
||||
worktreePath,
|
||||
options.config,
|
||||
options.onProgress,
|
||||
ref
|
||||
);
|
||||
|
||||
return { ...result, commitSha };
|
||||
} finally {
|
||||
|
||||
@@ -78,12 +78,8 @@
|
||||
"name": "documents_repository_id_repositories_id_fk",
|
||||
"tableFrom": "documents",
|
||||
"tableTo": "repositories",
|
||||
"columnsFrom": [
|
||||
"repository_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["repository_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
@@ -91,12 +87,8 @@
|
||||
"name": "documents_version_id_repository_versions_id_fk",
|
||||
"tableFrom": "documents",
|
||||
"tableTo": "repository_versions",
|
||||
"columnsFrom": [
|
||||
"version_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["version_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
@@ -196,12 +188,8 @@
|
||||
"name": "indexing_jobs_repository_id_repositories_id_fk",
|
||||
"tableFrom": "indexing_jobs",
|
||||
"tableTo": "repositories",
|
||||
"columnsFrom": [
|
||||
"repository_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["repository_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
@@ -411,12 +399,8 @@
|
||||
"name": "repository_configs_repository_id_repositories_id_fk",
|
||||
"tableFrom": "repository_configs",
|
||||
"tableTo": "repositories",
|
||||
"columnsFrom": [
|
||||
"repository_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["repository_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
@@ -493,12 +477,8 @@
|
||||
"name": "repository_versions_repository_id_repositories_id_fk",
|
||||
"tableFrom": "repository_versions",
|
||||
"tableTo": "repositories",
|
||||
"columnsFrom": [
|
||||
"repository_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["repository_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
@@ -583,12 +563,8 @@
|
||||
"name": "snippet_embeddings_snippet_id_snippets_id_fk",
|
||||
"tableFrom": "snippet_embeddings",
|
||||
"tableTo": "snippets",
|
||||
"columnsFrom": [
|
||||
"snippet_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["snippet_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
@@ -685,12 +661,8 @@
|
||||
"name": "snippets_document_id_documents_id_fk",
|
||||
"tableFrom": "snippets",
|
||||
"tableTo": "documents",
|
||||
"columnsFrom": [
|
||||
"document_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["document_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
@@ -698,12 +670,8 @@
|
||||
"name": "snippets_repository_id_repositories_id_fk",
|
||||
"tableFrom": "snippets",
|
||||
"tableTo": "repositories",
|
||||
"columnsFrom": [
|
||||
"repository_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["repository_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
@@ -711,12 +679,8 @@
|
||||
"name": "snippets_version_id_repository_versions_id_fk",
|
||||
"tableFrom": "snippets",
|
||||
"tableTo": "repository_versions",
|
||||
"columnsFrom": [
|
||||
"version_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["version_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
|
||||
@@ -78,12 +78,8 @@
|
||||
"name": "documents_repository_id_repositories_id_fk",
|
||||
"tableFrom": "documents",
|
||||
"tableTo": "repositories",
|
||||
"columnsFrom": [
|
||||
"repository_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["repository_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
@@ -91,12 +87,8 @@
|
||||
"name": "documents_version_id_repository_versions_id_fk",
|
||||
"tableFrom": "documents",
|
||||
"tableTo": "repository_versions",
|
||||
"columnsFrom": [
|
||||
"version_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["version_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
@@ -196,12 +188,8 @@
|
||||
"name": "indexing_jobs_repository_id_repositories_id_fk",
|
||||
"tableFrom": "indexing_jobs",
|
||||
"tableTo": "repositories",
|
||||
"columnsFrom": [
|
||||
"repository_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["repository_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
@@ -411,12 +399,8 @@
|
||||
"name": "repository_configs_repository_id_repositories_id_fk",
|
||||
"tableFrom": "repository_configs",
|
||||
"tableTo": "repositories",
|
||||
"columnsFrom": [
|
||||
"repository_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["repository_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
@@ -500,12 +484,8 @@
|
||||
"name": "repository_versions_repository_id_repositories_id_fk",
|
||||
"tableFrom": "repository_versions",
|
||||
"tableTo": "repositories",
|
||||
"columnsFrom": [
|
||||
"repository_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["repository_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
@@ -590,12 +570,8 @@
|
||||
"name": "snippet_embeddings_snippet_id_snippets_id_fk",
|
||||
"tableFrom": "snippet_embeddings",
|
||||
"tableTo": "snippets",
|
||||
"columnsFrom": [
|
||||
"snippet_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["snippet_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
@@ -692,12 +668,8 @@
|
||||
"name": "snippets_document_id_documents_id_fk",
|
||||
"tableFrom": "snippets",
|
||||
"tableTo": "documents",
|
||||
"columnsFrom": [
|
||||
"document_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["document_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
@@ -705,12 +677,8 @@
|
||||
"name": "snippets_repository_id_repositories_id_fk",
|
||||
"tableFrom": "snippets",
|
||||
"tableTo": "repositories",
|
||||
"columnsFrom": [
|
||||
"repository_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["repository_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
@@ -718,12 +686,8 @@
|
||||
"name": "snippets_version_id_repository_versions_id_fk",
|
||||
"tableFrom": "snippets",
|
||||
"tableTo": "repository_versions",
|
||||
"columnsFrom": [
|
||||
"version_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["version_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
|
||||
@@ -78,12 +78,8 @@
|
||||
"name": "documents_repository_id_repositories_id_fk",
|
||||
"tableFrom": "documents",
|
||||
"tableTo": "repositories",
|
||||
"columnsFrom": [
|
||||
"repository_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["repository_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
@@ -91,12 +87,8 @@
|
||||
"name": "documents_version_id_repository_versions_id_fk",
|
||||
"tableFrom": "documents",
|
||||
"tableTo": "repository_versions",
|
||||
"columnsFrom": [
|
||||
"version_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["version_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
@@ -278,12 +270,8 @@
|
||||
"name": "indexing_jobs_repository_id_repositories_id_fk",
|
||||
"tableFrom": "indexing_jobs",
|
||||
"tableTo": "repositories",
|
||||
"columnsFrom": [
|
||||
"repository_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["repository_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
@@ -493,12 +481,8 @@
|
||||
"name": "repository_configs_repository_id_repositories_id_fk",
|
||||
"tableFrom": "repository_configs",
|
||||
"tableTo": "repositories",
|
||||
"columnsFrom": [
|
||||
"repository_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["repository_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
@@ -582,12 +566,8 @@
|
||||
"name": "repository_versions_repository_id_repositories_id_fk",
|
||||
"tableFrom": "repository_versions",
|
||||
"tableTo": "repositories",
|
||||
"columnsFrom": [
|
||||
"repository_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["repository_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
@@ -679,12 +659,8 @@
|
||||
"name": "snippet_embeddings_snippet_id_snippets_id_fk",
|
||||
"tableFrom": "snippet_embeddings",
|
||||
"tableTo": "snippets",
|
||||
"columnsFrom": [
|
||||
"snippet_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["snippet_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
@@ -692,22 +668,15 @@
|
||||
"name": "snippet_embeddings_profile_id_embedding_profiles_id_fk",
|
||||
"tableFrom": "snippet_embeddings",
|
||||
"tableTo": "embedding_profiles",
|
||||
"columnsFrom": [
|
||||
"profile_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["profile_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {
|
||||
"snippet_embeddings_snippet_id_profile_id_pk": {
|
||||
"columns": [
|
||||
"snippet_id",
|
||||
"profile_id"
|
||||
],
|
||||
"columns": ["snippet_id", "profile_id"],
|
||||
"name": "snippet_embeddings_snippet_id_profile_id_pk"
|
||||
}
|
||||
},
|
||||
@@ -802,12 +771,8 @@
|
||||
"name": "snippets_document_id_documents_id_fk",
|
||||
"tableFrom": "snippets",
|
||||
"tableTo": "documents",
|
||||
"columnsFrom": [
|
||||
"document_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["document_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
@@ -815,12 +780,8 @@
|
||||
"name": "snippets_repository_id_repositories_id_fk",
|
||||
"tableFrom": "snippets",
|
||||
"tableTo": "repositories",
|
||||
"columnsFrom": [
|
||||
"repository_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["repository_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
@@ -828,12 +789,8 @@
|
||||
"name": "snippets_version_id_repository_versions_id_fk",
|
||||
"tableFrom": "snippets",
|
||||
"tableTo": "repository_versions",
|
||||
"columnsFrom": [
|
||||
"version_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"columnsFrom": ["version_id"],
|
||||
"columnsTo": ["id"],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
|
||||
@@ -381,7 +381,12 @@ describe('EmbeddingService', () => {
|
||||
.all(snippetId, 'local-default');
|
||||
expect(rows).toHaveLength(1);
|
||||
|
||||
const row = rows[0] as { model: string; dimensions: number; embedding: Buffer; profile_id: string };
|
||||
const row = rows[0] as {
|
||||
model: string;
|
||||
dimensions: number;
|
||||
embedding: Buffer;
|
||||
profile_id: string;
|
||||
};
|
||||
expect(row.model).toBe('test-model');
|
||||
expect(row.dimensions).toBe(4);
|
||||
expect(row.profile_id).toBe('local-default');
|
||||
@@ -494,9 +499,7 @@ describe('createProviderFromConfig', () => {
|
||||
});
|
||||
|
||||
it('throws when openai provider is selected without config', () => {
|
||||
expect(() =>
|
||||
createProviderFromConfig({ provider: 'openai' } as EmbeddingConfig)
|
||||
).toThrow();
|
||||
expect(() => createProviderFromConfig({ provider: 'openai' } as EmbeddingConfig)).toThrow();
|
||||
});
|
||||
|
||||
it('defaultEmbeddingConfig returns provider=none', () => {
|
||||
|
||||
@@ -41,18 +41,16 @@ export class EmbeddingService {
|
||||
|
||||
const placeholders = snippetIds.map(() => '?').join(',');
|
||||
const snippets = this.db
|
||||
.prepare<string[], SnippetRow>(
|
||||
`SELECT id, title, breadcrumb, content FROM snippets WHERE id IN (${placeholders})`
|
||||
)
|
||||
.prepare<
|
||||
string[],
|
||||
SnippetRow
|
||||
>(`SELECT id, title, breadcrumb, content FROM snippets WHERE id IN (${placeholders})`)
|
||||
.all(...snippetIds);
|
||||
|
||||
if (snippets.length === 0) return;
|
||||
|
||||
const texts = snippets.map((s) =>
|
||||
[s.title, s.breadcrumb, s.content]
|
||||
.filter(Boolean)
|
||||
.join('\n')
|
||||
.slice(0, TEXT_MAX_CHARS)
|
||||
[s.title, s.breadcrumb, s.content].filter(Boolean).join('\n').slice(0, TEXT_MAX_CHARS)
|
||||
);
|
||||
|
||||
const insert = this.db.prepare<[string, string, string, number, Buffer]>(`
|
||||
@@ -94,9 +92,10 @@ export class EmbeddingService {
|
||||
*/
|
||||
getEmbedding(snippetId: string, profileId: string = 'local-default'): Float32Array | null {
|
||||
const row = this.db
|
||||
.prepare<[string, string], { embedding: Buffer; dimensions: number }>(
|
||||
`SELECT embedding, dimensions FROM snippet_embeddings WHERE snippet_id = ? AND profile_id = ?`
|
||||
)
|
||||
.prepare<
|
||||
[string, string],
|
||||
{ embedding: Buffer; dimensions: number }
|
||||
>(`SELECT embedding, dimensions FROM snippet_embeddings WHERE snippet_id = ? AND profile_id = ?`)
|
||||
.get(snippetId, profileId);
|
||||
|
||||
if (!row) return null;
|
||||
|
||||
@@ -12,7 +12,11 @@ import { OpenAIEmbeddingProvider } from './openai.provider.js';
|
||||
import { LocalEmbeddingProvider } from './local.provider.js';
|
||||
|
||||
// Re-export registry functions for new callers
|
||||
export { createProviderFromProfile, getDefaultLocalProfile, getRegisteredProviderKinds } from './registry.js';
|
||||
export {
|
||||
createProviderFromProfile,
|
||||
getDefaultLocalProfile,
|
||||
getRegisteredProviderKinds
|
||||
} from './registry.js';
|
||||
|
||||
export interface EmbeddingConfig {
|
||||
provider: 'openai' | 'local' | 'none';
|
||||
|
||||
@@ -43,7 +43,12 @@ export class ContextResponseMapper {
|
||||
lastUpdateDate: repository.lastIndexedAt
|
||||
? repository.lastIndexedAt.toISOString()
|
||||
: null,
|
||||
state: repository.state === 'indexed' ? 'finalized' : repository.state === 'error' ? 'error' : 'initial',
|
||||
state:
|
||||
repository.state === 'indexed'
|
||||
? 'finalized'
|
||||
: repository.state === 'error'
|
||||
? 'error'
|
||||
: 'initial',
|
||||
totalTokens: repository.totalTokens ?? null,
|
||||
totalSnippets: repository.totalSnippets ?? null,
|
||||
stars: repository.stars ?? null,
|
||||
@@ -68,7 +73,9 @@ export class ContextResponseMapper {
|
||||
repositoryTitle: metadata.repository.title,
|
||||
source: metadata.repository.source,
|
||||
sourceUrl: metadata.repository.sourceUrl,
|
||||
version: snippet.versionId ? metadata.snippetVersions[snippet.versionId] ?? null : null,
|
||||
version: snippet.versionId
|
||||
? (metadata.snippetVersions[snippet.versionId] ?? null)
|
||||
: null,
|
||||
versionId: snippet.versionId,
|
||||
isLocal: metadata.localSource
|
||||
})
|
||||
|
||||
@@ -12,8 +12,7 @@ export class IndexingJobMapper {
|
||||
processedFiles: entity.processed_files,
|
||||
error: entity.error,
|
||||
startedAt: entity.started_at != null ? new Date(entity.started_at * 1000) : null,
|
||||
completedAt:
|
||||
entity.completed_at != null ? new Date(entity.completed_at * 1000) : null,
|
||||
completedAt: entity.completed_at != null ? new Date(entity.completed_at * 1000) : null,
|
||||
createdAt: new Date(entity.created_at * 1000)
|
||||
});
|
||||
}
|
||||
|
||||
@@ -1,4 +1,8 @@
|
||||
import { LibrarySearchResult, SnippetRepositoryRef, SnippetSearchResult } from '$lib/server/models/search-result.js';
|
||||
import {
|
||||
LibrarySearchResult,
|
||||
SnippetRepositoryRef,
|
||||
SnippetSearchResult
|
||||
} from '$lib/server/models/search-result.js';
|
||||
import { RepositoryMapper } from '$lib/server/mappers/repository.mapper.js';
|
||||
import { RepositoryVersionMapper } from '$lib/server/mappers/repository-version.mapper.js';
|
||||
import { SnippetMapper } from '$lib/server/mappers/snippet.mapper.js';
|
||||
@@ -26,9 +30,7 @@ export class SearchResultMapper {
|
||||
): LibrarySearchResult {
|
||||
return new LibrarySearchResult({
|
||||
repository: RepositoryMapper.fromEntity(repositoryEntity),
|
||||
versions: versionEntities.map((version) =>
|
||||
RepositoryVersionMapper.fromEntity(version)
|
||||
),
|
||||
versions: versionEntities.map((version) => RepositoryVersionMapper.fromEntity(version)),
|
||||
score
|
||||
});
|
||||
}
|
||||
|
||||
@@ -286,7 +286,8 @@ This is the second paragraph that also has enough content to be included here.
|
||||
});
|
||||
|
||||
it('skips paragraphs shorter than 20 characters', () => {
|
||||
const content = 'Short.\n\nThis is a much longer paragraph that definitely passes the minimum length filter.';
|
||||
const content =
|
||||
'Short.\n\nThis is a much longer paragraph that definitely passes the minimum length filter.';
|
||||
const snippets = parseCodeFile(content, 'notes.txt', 'text');
|
||||
expect(snippets.length).toBe(1);
|
||||
});
|
||||
@@ -331,7 +332,10 @@ export function realFunction(): string {
|
||||
|
||||
describe('parseCodeFile — token count', () => {
|
||||
it('all snippets have tokenCount within MAX_TOKENS', () => {
|
||||
const lines = Array.from({ length: 300 }, (_, i) => `// comment line number ${i} here\nconst x${i} = ${i};`);
|
||||
const lines = Array.from(
|
||||
{ length: 300 },
|
||||
(_, i) => `// comment line number ${i} here\nconst x${i} = ${i};`
|
||||
);
|
||||
const content = lines.join('\n');
|
||||
|
||||
const snippets = parseCodeFile(content, 'large.ts', 'typescript');
|
||||
|
||||
@@ -26,15 +26,19 @@ import {
|
||||
* The regex is tested line-by-line (multiline flag not needed).
|
||||
*/
|
||||
export const BOUNDARY_PATTERNS: Record<string, RegExp> = {
|
||||
typescript: /^(export\s+)?(declare\s+)?(async\s+)?(function|class|interface|type|enum|const|let|var)\s+\w+/,
|
||||
typescript:
|
||||
/^(export\s+)?(declare\s+)?(async\s+)?(function|class|interface|type|enum|const|let|var)\s+\w+/,
|
||||
javascript: /^(export\s+)?(async\s+)?(function|class|const|let|var)\s+\w+/,
|
||||
python: /^(async\s+)?(def|class)\s+\w+/,
|
||||
go: /^(func|type|var|const)\s+\w+/,
|
||||
rust: /^(pub(\s*\(crate\))?\s+)?(async\s+)?(fn|impl|struct|enum|trait|type|const|static)\s+\w+/,
|
||||
java: /^(\s*(public|private|protected|static|final|abstract|synchronized)\s+)+[\w<>\[\]]+\s+\w+\s*[({]/,
|
||||
csharp: /^(\s*(public|private|protected|internal|static|override|virtual|abstract|sealed)\s+)+[\w<>\[\]]+\s+\w+\s*[({]/,
|
||||
kotlin: /^(\s*(public|private|protected|internal|override|suspend|inline|open|abstract|sealed)\s+)*(fun|class|object|interface|data class|sealed class|enum class)\s+\w+/,
|
||||
swift: /^(\s*(public|private|internal|fileprivate|open|override|static|final|class)\s+)*(func|class|struct|enum|protocol|extension)\s+\w+/,
|
||||
csharp:
|
||||
/^(\s*(public|private|protected|internal|static|override|virtual|abstract|sealed)\s+)+[\w<>\[\]]+\s+\w+\s*[({]/,
|
||||
kotlin:
|
||||
/^(\s*(public|private|protected|internal|override|suspend|inline|open|abstract|sealed)\s+)*(fun|class|object|interface|data class|sealed class|enum class)\s+\w+/,
|
||||
swift:
|
||||
/^(\s*(public|private|internal|fileprivate|open|override|static|final|class)\s+)*(func|class|struct|enum|protocol|extension)\s+\w+/,
|
||||
ruby: /^(def|class|module)\s+\w+/
|
||||
};
|
||||
|
||||
@@ -42,7 +46,10 @@ export const BOUNDARY_PATTERNS: Record<string, RegExp> = {
|
||||
// Internal types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
type RawSnippet = Omit<NewSnippet, 'id' | 'repositoryId' | 'documentId' | 'versionId' | 'createdAt'>;
|
||||
type RawSnippet = Omit<
|
||||
NewSnippet,
|
||||
'id' | 'repositoryId' | 'documentId' | 'versionId' | 'createdAt'
|
||||
>;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Helpers
|
||||
@@ -161,7 +168,10 @@ function parseHtmlLikeFile(content: string, filePath: string, language: string):
|
||||
|
||||
while ((match = scriptPattern.exec(content)) !== null) {
|
||||
// Strip the outer tags, keep just the code
|
||||
const inner = match[0].replace(/^<script[^>]*>/, '').replace(/<\/script>$/, '').trim();
|
||||
const inner = match[0]
|
||||
.replace(/^<script[^>]*>/, '')
|
||||
.replace(/<\/script>$/, '')
|
||||
.trim();
|
||||
if (inner.length >= MIN_CONTENT_LENGTH) {
|
||||
scriptBlocks.push(inner);
|
||||
}
|
||||
|
||||
@@ -48,6 +48,13 @@ export function parseFile(file: CrawledFile, options: ParseOptions): NewSnippet[
|
||||
|
||||
// Re-export helpers for consumers that need them individually
|
||||
export { detectLanguage } from './language.js';
|
||||
export { estimateTokens, chunkText, chunkLines, MAX_TOKENS, OVERLAP_TOKENS, MIN_CONTENT_LENGTH } from './chunker.js';
|
||||
export {
|
||||
estimateTokens,
|
||||
chunkText,
|
||||
chunkLines,
|
||||
MAX_TOKENS,
|
||||
OVERLAP_TOKENS,
|
||||
MIN_CONTENT_LENGTH
|
||||
} from './chunker.js';
|
||||
export { parseMarkdown } from './markdown.parser.js';
|
||||
export { parseCodeFile, BOUNDARY_PATTERNS } from './code.parser.js';
|
||||
|
||||
@@ -99,7 +99,10 @@ describe('parseMarkdown — section splitting', () => {
|
||||
|
||||
describe('parseMarkdown — code block extraction', () => {
|
||||
it('extracts a fenced code block as a code snippet', () => {
|
||||
const codeBlock = fence('typescript', 'function hello(name: string): string {\n return `Hello, ${name}!`;\n}');
|
||||
const codeBlock = fence(
|
||||
'typescript',
|
||||
'function hello(name: string): string {\n return `Hello, ${name}!`;\n}'
|
||||
);
|
||||
const source = [
|
||||
'# Example',
|
||||
'',
|
||||
@@ -232,7 +235,10 @@ describe('parseMarkdown — large content chunking', () => {
|
||||
describe('parseMarkdown — real-world sample', () => {
|
||||
it('correctly parses a realistic README excerpt', () => {
|
||||
const bashInstall = fence('bash', 'npm install my-library');
|
||||
const tsUsage = fence('typescript', "import { doTheThing } from 'my-library';\n\ndoTheThing({ verbose: true });");
|
||||
const tsUsage = fence(
|
||||
'typescript',
|
||||
"import { doTheThing } from 'my-library';\n\ndoTheThing({ verbose: true });"
|
||||
);
|
||||
|
||||
const source = [
|
||||
'# My Library',
|
||||
|
||||
@@ -7,7 +7,13 @@
|
||||
|
||||
import { basename } from 'node:path';
|
||||
import type { NewSnippet } from '$lib/server/db/schema.js';
|
||||
import { estimateTokens, chunkText, MAX_TOKENS, OVERLAP_TOKENS, MIN_CONTENT_LENGTH } from './chunker.js';
|
||||
import {
|
||||
estimateTokens,
|
||||
chunkText,
|
||||
MAX_TOKENS,
|
||||
OVERLAP_TOKENS,
|
||||
MIN_CONTENT_LENGTH
|
||||
} from './chunker.js';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Internal types
|
||||
@@ -121,7 +127,10 @@ function splitIntoSections(source: string): MarkdownSection[] {
|
||||
// Public parser
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
type RawSnippet = Omit<NewSnippet, 'id' | 'repositoryId' | 'documentId' | 'versionId' | 'createdAt'>;
|
||||
type RawSnippet = Omit<
|
||||
NewSnippet,
|
||||
'id' | 'repositoryId' | 'documentId' | 'versionId' | 'createdAt'
|
||||
>;
|
||||
|
||||
/**
|
||||
* Parse a Markdown/MDX file into raw snippets (before IDs and DB fields are
|
||||
|
||||
@@ -22,10 +22,7 @@ function createTestDb(): Database.Database {
|
||||
client.pragma('foreign_keys = ON');
|
||||
|
||||
const migrationsFolder = join(import.meta.dirname, '../db/migrations');
|
||||
const migrationSql = readFileSync(
|
||||
join(migrationsFolder, '0000_large_master_chief.sql'),
|
||||
'utf-8'
|
||||
);
|
||||
const migrationSql = readFileSync(join(migrationsFolder, '0000_large_master_chief.sql'), 'utf-8');
|
||||
|
||||
const statements = migrationSql
|
||||
.split('--> statement-breakpoint')
|
||||
@@ -45,10 +42,7 @@ function createTestDb(): Database.Database {
|
||||
|
||||
const now = Math.floor(Date.now() / 1000);
|
||||
|
||||
function insertRepo(
|
||||
db: Database.Database,
|
||||
overrides: Partial<Record<string, unknown>> = {}
|
||||
): void {
|
||||
function insertRepo(db: Database.Database, overrides: Partial<Record<string, unknown>> = {}): void {
|
||||
db.prepare(
|
||||
`INSERT INTO repositories
|
||||
(id, title, source, source_url, branch, state,
|
||||
@@ -62,7 +56,15 @@ function insertRepo(
|
||||
overrides.source_url ?? '/tmp/test-repo',
|
||||
overrides.branch ?? 'main',
|
||||
overrides.state ?? 'pending',
|
||||
0, 0, 0, 0, null, null, null, now, now
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
now,
|
||||
now
|
||||
);
|
||||
}
|
||||
|
||||
@@ -108,9 +110,10 @@ describe('recoverStaleJobs', () => {
|
||||
insertJob(db, { status: 'running' });
|
||||
recoverStaleJobs(db);
|
||||
|
||||
const row = db
|
||||
.prepare(`SELECT status, error FROM indexing_jobs LIMIT 1`)
|
||||
.get() as { status: string; error: string };
|
||||
const row = db.prepare(`SELECT status, error FROM indexing_jobs LIMIT 1`).get() as {
|
||||
status: string;
|
||||
error: string;
|
||||
};
|
||||
expect(row.status).toBe('failed');
|
||||
expect(row.error).toMatch(/restarted/i);
|
||||
});
|
||||
@@ -119,9 +122,9 @@ describe('recoverStaleJobs', () => {
|
||||
db.prepare(`UPDATE repositories SET state = 'indexing' WHERE id = '/test/repo'`).run();
|
||||
recoverStaleJobs(db);
|
||||
|
||||
const row = db
|
||||
.prepare(`SELECT state FROM repositories WHERE id = '/test/repo'`)
|
||||
.get() as { state: string };
|
||||
const row = db.prepare(`SELECT state FROM repositories WHERE id = '/test/repo'`).get() as {
|
||||
state: string;
|
||||
};
|
||||
expect(row.state).toBe('error');
|
||||
});
|
||||
|
||||
@@ -164,9 +167,7 @@ describe('JobQueue', () => {
|
||||
const job2 = queue.enqueue('/test/repo');
|
||||
expect(job1.id).toBe(job2.id);
|
||||
|
||||
const count = (
|
||||
db.prepare(`SELECT COUNT(*) as n FROM indexing_jobs`).get() as { n: number }
|
||||
).n;
|
||||
const count = (db.prepare(`SELECT COUNT(*) as n FROM indexing_jobs`).get() as { n: number }).n;
|
||||
expect(count).toBe(1);
|
||||
});
|
||||
|
||||
@@ -255,19 +256,19 @@ describe('IndexingPipeline', () => {
|
||||
})
|
||||
};
|
||||
|
||||
return new IndexingPipeline(
|
||||
db,
|
||||
mockGithubCrawl as never,
|
||||
mockLocalCrawler as never,
|
||||
null
|
||||
);
|
||||
return new IndexingPipeline(db, mockGithubCrawl as never, mockLocalCrawler as never, null);
|
||||
}
|
||||
|
||||
function makeJob(repositoryId = '/test/repo') {
|
||||
const jobId = insertJob(db, { repository_id: repositoryId, status: 'queued' });
|
||||
return db
|
||||
.prepare(`SELECT * FROM indexing_jobs WHERE id = ?`)
|
||||
.get(jobId) as { id: string; repositoryId?: string; repository_id?: string; status: string; versionId?: string; version_id?: string };
|
||||
return db.prepare(`SELECT * FROM indexing_jobs WHERE id = ?`).get(jobId) as {
|
||||
id: string;
|
||||
repositoryId?: string;
|
||||
repository_id?: string;
|
||||
status: string;
|
||||
versionId?: string;
|
||||
version_id?: string;
|
||||
};
|
||||
}
|
||||
|
||||
it('marks job as done when there are no files to index', async () => {
|
||||
@@ -289,9 +290,9 @@ describe('IndexingPipeline', () => {
|
||||
|
||||
await pipeline.run(job as never);
|
||||
|
||||
const updated = db
|
||||
.prepare(`SELECT status FROM indexing_jobs WHERE id = ?`)
|
||||
.get(job.id) as { status: string };
|
||||
const updated = db.prepare(`SELECT status FROM indexing_jobs WHERE id = ?`).get(job.id) as {
|
||||
status: string;
|
||||
};
|
||||
// The job should end in 'done' — the running→done transition is covered
|
||||
// by the pipeline's internal updateJob calls.
|
||||
expect(updated.status).toBe('done');
|
||||
@@ -363,27 +364,24 @@ describe('IndexingPipeline', () => {
|
||||
const job1 = makeJob();
|
||||
await pipeline.run(job1 as never);
|
||||
|
||||
const firstDocCount = (
|
||||
db.prepare(`SELECT COUNT(*) as n FROM documents`).get() as { n: number }
|
||||
).n;
|
||||
const firstSnippetIds = (
|
||||
db.prepare(`SELECT id FROM snippets`).all() as { id: string }[]
|
||||
).map((r) => r.id);
|
||||
const firstDocCount = (db.prepare(`SELECT COUNT(*) as n FROM documents`).get() as { n: number })
|
||||
.n;
|
||||
const firstSnippetIds = (db.prepare(`SELECT id FROM snippets`).all() as { id: string }[]).map(
|
||||
(r) => r.id
|
||||
);
|
||||
|
||||
// Second run with identical files.
|
||||
const job2Id = insertJob(db, { repository_id: '/test/repo', status: 'queued' });
|
||||
const job2 = db
|
||||
.prepare(`SELECT * FROM indexing_jobs WHERE id = ?`)
|
||||
.get(job2Id) as never;
|
||||
const job2 = db.prepare(`SELECT * FROM indexing_jobs WHERE id = ?`).get(job2Id) as never;
|
||||
|
||||
await pipeline.run(job2);
|
||||
|
||||
const secondDocCount = (
|
||||
db.prepare(`SELECT COUNT(*) as n FROM documents`).get() as { n: number }
|
||||
).n;
|
||||
const secondSnippetIds = (
|
||||
db.prepare(`SELECT id FROM snippets`).all() as { id: string }[]
|
||||
).map((r) => r.id);
|
||||
const secondSnippetIds = (db.prepare(`SELECT id FROM snippets`).all() as { id: string }[]).map(
|
||||
(r) => r.id
|
||||
);
|
||||
|
||||
// Document count stays the same and snippet IDs are unchanged.
|
||||
expect(secondDocCount).toBe(firstDocCount);
|
||||
@@ -395,7 +393,8 @@ describe('IndexingPipeline', () => {
|
||||
files: [
|
||||
{
|
||||
path: 'README.md',
|
||||
content: '# Original\n\nThis is the original version of the documentation with sufficient content.',
|
||||
content:
|
||||
'# Original\n\nThis is the original version of the documentation with sufficient content.',
|
||||
sha: 'sha-v1',
|
||||
language: 'markdown'
|
||||
}
|
||||
@@ -415,7 +414,8 @@ describe('IndexingPipeline', () => {
|
||||
files: [
|
||||
{
|
||||
path: 'README.md',
|
||||
content: '# Updated\n\nThis is a completely different version of the documentation with new content.',
|
||||
content:
|
||||
'# Updated\n\nThis is a completely different version of the documentation with new content.',
|
||||
sha: 'sha-v2',
|
||||
language: 'markdown'
|
||||
}
|
||||
@@ -423,14 +423,11 @@ describe('IndexingPipeline', () => {
|
||||
totalFiles: 1
|
||||
});
|
||||
const job2Id = insertJob(db, { repository_id: '/test/repo', status: 'queued' });
|
||||
const job2 = db
|
||||
.prepare(`SELECT * FROM indexing_jobs WHERE id = ?`)
|
||||
.get(job2Id) as never;
|
||||
const job2 = db.prepare(`SELECT * FROM indexing_jobs WHERE id = ?`).get(job2Id) as never;
|
||||
await pipeline2.run(job2);
|
||||
|
||||
const finalDocCount = (
|
||||
db.prepare(`SELECT COUNT(*) as n FROM documents`).get() as { n: number }
|
||||
).n;
|
||||
const finalDocCount = (db.prepare(`SELECT COUNT(*) as n FROM documents`).get() as { n: number })
|
||||
.n;
|
||||
// Only one document should exist (the updated one).
|
||||
expect(finalDocCount).toBe(1);
|
||||
|
||||
@@ -452,9 +449,9 @@ describe('IndexingPipeline', () => {
|
||||
const job = makeJob();
|
||||
await pipeline.run(job as never);
|
||||
|
||||
const updated = db
|
||||
.prepare(`SELECT progress FROM indexing_jobs WHERE id = ?`)
|
||||
.get(job.id) as { progress: number };
|
||||
const updated = db.prepare(`SELECT progress FROM indexing_jobs WHERE id = ?`).get(job.id) as {
|
||||
progress: number;
|
||||
};
|
||||
expect(updated.progress).toBe(100);
|
||||
});
|
||||
|
||||
@@ -467,12 +464,7 @@ describe('IndexingPipeline', () => {
|
||||
commitSha: 'abc'
|
||||
});
|
||||
|
||||
const pipeline = new IndexingPipeline(
|
||||
db,
|
||||
vi.fn() as never,
|
||||
{ crawl } as never,
|
||||
null
|
||||
);
|
||||
const pipeline = new IndexingPipeline(db, vi.fn() as never, { crawl } as never, null);
|
||||
|
||||
const job = makeJob();
|
||||
await pipeline.run(job as never);
|
||||
@@ -511,7 +503,10 @@ describe('IndexingPipeline', () => {
|
||||
await pipeline1.run(job1 as never);
|
||||
|
||||
const afterFirstRun = {
|
||||
docs: db.prepare(`SELECT file_path, checksum FROM documents ORDER BY file_path`).all() as { file_path: string; checksum: string }[],
|
||||
docs: db.prepare(`SELECT file_path, checksum FROM documents ORDER BY file_path`).all() as {
|
||||
file_path: string;
|
||||
checksum: string;
|
||||
}[],
|
||||
snippetCount: (db.prepare(`SELECT COUNT(*) as n FROM snippets`).get() as { n: number }).n
|
||||
};
|
||||
expect(afterFirstRun.docs).toHaveLength(3);
|
||||
|
||||
@@ -250,7 +250,10 @@ export class IndexingPipeline {
|
||||
private async crawl(
|
||||
repo: Repository,
|
||||
job: IndexingJob
|
||||
): Promise<{ files: Array<{ path: string; content: string; sha: string; size: number; language: string }>; totalFiles: number }> {
|
||||
): Promise<{
|
||||
files: Array<{ path: string; content: string; sha: string; size: number; language: string }>;
|
||||
totalFiles: number;
|
||||
}> {
|
||||
if (repo.source === 'github') {
|
||||
// Parse owner/repo from the canonical ID: "/owner/repo"
|
||||
const parts = repo.id.replace(/^\//, '').split('/');
|
||||
|
||||
@@ -133,9 +133,7 @@ export class JobQueue {
|
||||
|
||||
// Check whether another job was queued while this one ran.
|
||||
const next = this.db
|
||||
.prepare<[], { id: string }>(
|
||||
`SELECT id FROM indexing_jobs WHERE status = 'queued' LIMIT 1`
|
||||
)
|
||||
.prepare<[], { id: string }>(`SELECT id FROM indexing_jobs WHERE status = 'queued' LIMIT 1`)
|
||||
.get();
|
||||
if (next) {
|
||||
setImmediate(() => this.processNext());
|
||||
@@ -147,9 +145,7 @@ export class JobQueue {
|
||||
* Retrieve a single job by ID.
|
||||
*/
|
||||
getJob(id: string): IndexingJob | null {
|
||||
const raw = this.db
|
||||
.prepare<[string], IndexingJobEntity>(`${JOB_SELECT} WHERE id = ?`)
|
||||
.get(id);
|
||||
const raw = this.db.prepare<[string], IndexingJobEntity>(`${JOB_SELECT} WHERE id = ?`).get(id);
|
||||
return raw ? IndexingJobMapper.fromEntity(new IndexingJobEntity(raw)) : null;
|
||||
}
|
||||
|
||||
@@ -178,9 +174,9 @@ export class JobQueue {
|
||||
const sql = `${JOB_SELECT} ${where} ORDER BY created_at DESC LIMIT ?`;
|
||||
params.push(limit);
|
||||
|
||||
return (this.db.prepare<unknown[], IndexingJobEntity>(sql).all(...params) as IndexingJobEntity[]).map(
|
||||
(row) => IndexingJobMapper.fromEntity(new IndexingJobEntity(row))
|
||||
);
|
||||
return (
|
||||
this.db.prepare<unknown[], IndexingJobEntity>(sql).all(...params) as IndexingJobEntity[]
|
||||
).map((row) => IndexingJobMapper.fromEntity(new IndexingJobEntity(row)));
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -228,9 +224,7 @@ export class JobQueue {
|
||||
return false;
|
||||
}
|
||||
|
||||
this.db
|
||||
.prepare(`UPDATE indexing_jobs SET status = 'paused' WHERE id = ?`)
|
||||
.run(id);
|
||||
this.db.prepare(`UPDATE indexing_jobs SET status = 'paused' WHERE id = ?`).run(id);
|
||||
|
||||
return true;
|
||||
}
|
||||
@@ -249,9 +243,7 @@ export class JobQueue {
|
||||
return false;
|
||||
}
|
||||
|
||||
this.db
|
||||
.prepare(`UPDATE indexing_jobs SET status = 'queued' WHERE id = ?`)
|
||||
.run(id);
|
||||
this.db.prepare(`UPDATE indexing_jobs SET status = 'queued' WHERE id = ?`).run(id);
|
||||
|
||||
// Trigger queue processing in case the queue was idle
|
||||
this.drainQueued();
|
||||
|
||||
@@ -27,7 +27,11 @@ function createTestDb(): Database.Database {
|
||||
const migrationsFolder = join(import.meta.dirname, '../db/migrations');
|
||||
|
||||
// Run all migrations in order
|
||||
const migrations = ['0000_large_master_chief.sql', '0001_quick_nighthawk.sql', '0002_silky_stellaris.sql'];
|
||||
const migrations = [
|
||||
'0000_large_master_chief.sql',
|
||||
'0001_quick_nighthawk.sql',
|
||||
'0002_silky_stellaris.sql'
|
||||
];
|
||||
for (const migrationFile of migrations) {
|
||||
const migrationSql = readFileSync(join(migrationsFolder, migrationFile), 'utf-8');
|
||||
const statements = migrationSql
|
||||
@@ -123,9 +127,7 @@ function seedEmbedding(
|
||||
// Mock EmbeddingProvider
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function makeMockProvider(
|
||||
returnValues: number[][] = [[1, 0, 0, 0]]
|
||||
): EmbeddingProvider {
|
||||
function makeMockProvider(returnValues: number[][] = [[1, 0, 0, 0]]): EmbeddingProvider {
|
||||
return {
|
||||
name: 'mock',
|
||||
dimensions: returnValues[0]?.length ?? 4,
|
||||
@@ -254,9 +256,18 @@ describe('reciprocalRankFusion', () => {
|
||||
});
|
||||
|
||||
it('handles three lists correctly', () => {
|
||||
const r1 = [{ id: 'a', score: 1 }, { id: 'b', score: 0 }];
|
||||
const r2 = [{ id: 'b', score: 1 }, { id: 'c', score: 0 }];
|
||||
const r3 = [{ id: 'a', score: 1 }, { id: 'c', score: 0 }];
|
||||
const r1 = [
|
||||
{ id: 'a', score: 1 },
|
||||
{ id: 'b', score: 0 }
|
||||
];
|
||||
const r2 = [
|
||||
{ id: 'b', score: 1 },
|
||||
{ id: 'c', score: 0 }
|
||||
];
|
||||
const r3 = [
|
||||
{ id: 'a', score: 1 },
|
||||
{ id: 'c', score: 0 }
|
||||
];
|
||||
const result = reciprocalRankFusion(r1, r2, r3);
|
||||
// 'a' appears first in r1 and r3 → higher combined score than 'b' or 'c'.
|
||||
expect(result[0].id).toBe('a');
|
||||
|
||||
@@ -103,10 +103,7 @@ export class HybridSearchService {
|
||||
* @param options - Search parameters including repositoryId and alpha blend.
|
||||
* @returns Ranked array of SnippetSearchResult, deduplicated by snippet ID.
|
||||
*/
|
||||
async search(
|
||||
query: string,
|
||||
options: HybridSearchOptions
|
||||
): Promise<SnippetSearchResult[]> {
|
||||
async search(query: string, options: HybridSearchOptions): Promise<SnippetSearchResult[]> {
|
||||
const limit = options.limit ?? 20;
|
||||
const mode = options.searchMode ?? 'auto';
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
*/
|
||||
export function preprocessQuery(raw: string): string {
|
||||
// 1. Trim and collapse whitespace.
|
||||
let q = raw.trim().replace(/\s+/g, ' ');
|
||||
const q = raw.trim().replace(/\s+/g, ' ');
|
||||
|
||||
if (!q) return q;
|
||||
|
||||
@@ -91,7 +91,10 @@ export function preprocessQuery(raw: string): string {
|
||||
}
|
||||
|
||||
// Remove trailing operators
|
||||
while (finalTokens.length > 0 && ['AND', 'OR', 'NOT'].includes(finalTokens[finalTokens.length - 1])) {
|
||||
while (
|
||||
finalTokens.length > 0 &&
|
||||
['AND', 'OR', 'NOT'].includes(finalTokens[finalTokens.length - 1])
|
||||
) {
|
||||
finalTokens.pop();
|
||||
}
|
||||
|
||||
|
||||
@@ -32,9 +32,7 @@ export interface FusedItem {
|
||||
* descending relevance (index 0 = most relevant).
|
||||
* @returns Fused array sorted by descending rrfScore, deduplicated by id.
|
||||
*/
|
||||
export function reciprocalRankFusion(
|
||||
...rankings: Array<Array<RankedItem>>
|
||||
): Array<FusedItem> {
|
||||
export function reciprocalRankFusion(...rankings: Array<Array<RankedItem>>): Array<FusedItem> {
|
||||
const K = 60; // Standard RRF constant.
|
||||
const scores = new Map<string, number>();
|
||||
|
||||
|
||||
@@ -674,7 +674,9 @@ describe('formatLibraryResults', () => {
|
||||
id: '/facebook/react/v18',
|
||||
repositoryId: '/facebook/react',
|
||||
tag: 'v18',
|
||||
title: 'React 18', commitHash: null, state: 'indexed',
|
||||
title: 'React 18',
|
||||
commitHash: null,
|
||||
state: 'indexed',
|
||||
totalSnippets: 1000,
|
||||
indexedAt: null,
|
||||
createdAt: now
|
||||
@@ -731,7 +733,9 @@ describe('formatLibraryResults', () => {
|
||||
describe('formatSnippetResults', () => {
|
||||
const now = new Date();
|
||||
|
||||
function makeSnippetResult(overrides: Partial<Parameters<typeof formatSnippetResults>[0][number]> = {}): Parameters<typeof formatSnippetResults>[0][number] {
|
||||
function makeSnippetResult(
|
||||
overrides: Partial<Parameters<typeof formatSnippetResults>[0][number]> = {}
|
||||
): Parameters<typeof formatSnippetResults>[0][number] {
|
||||
return {
|
||||
snippet: {
|
||||
id: crypto.randomUUID(),
|
||||
|
||||
@@ -87,10 +87,7 @@ export class SearchService {
|
||||
if (!processedQuery) return [];
|
||||
|
||||
// Build the WHERE clause dynamically based on optional filters.
|
||||
const conditions: string[] = [
|
||||
'snippets_fts MATCH ?',
|
||||
's.repository_id = ?'
|
||||
];
|
||||
const conditions: string[] = ['snippets_fts MATCH ?', 's.repository_id = ?'];
|
||||
const params: unknown[] = [processedQuery, repositoryId];
|
||||
|
||||
if (versionId !== undefined) {
|
||||
@@ -132,10 +129,14 @@ export class SearchService {
|
||||
const rows = this.db.prepare(sql).all(...params) as RawSnippetRow[];
|
||||
|
||||
return rows.map((row) =>
|
||||
SearchResultMapper.snippetFromEntity(new SnippetEntity(row), {
|
||||
SearchResultMapper.snippetFromEntity(
|
||||
new SnippetEntity(row),
|
||||
{
|
||||
id: row.repo_id,
|
||||
title: row.repo_title
|
||||
}, row.score)
|
||||
},
|
||||
row.score
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
@@ -188,7 +189,11 @@ export class SearchService {
|
||||
|
||||
return rows.map((row) => {
|
||||
const compositeScore =
|
||||
row.exact_match + row.prefix_match + row.desc_match + row.snippet_score + row.trust_component;
|
||||
row.exact_match +
|
||||
row.prefix_match +
|
||||
row.desc_match +
|
||||
row.snippet_score +
|
||||
row.trust_component;
|
||||
return SearchResultMapper.libraryFromEntity(
|
||||
new RepositoryEntity(row),
|
||||
this.getVersionEntities(row.id),
|
||||
@@ -203,9 +208,7 @@ export class SearchService {
|
||||
|
||||
private getVersionEntities(repositoryId: string): RepositoryVersionEntity[] {
|
||||
return this.db
|
||||
.prepare(
|
||||
`SELECT * FROM repository_versions WHERE repository_id = ? ORDER BY created_at DESC`
|
||||
)
|
||||
.prepare(`SELECT * FROM repository_versions WHERE repository_id = ? ORDER BY created_at DESC`)
|
||||
.all(repositoryId) as RawVersionRow[];
|
||||
}
|
||||
}
|
||||
|
||||
@@ -46,9 +46,7 @@ interface RawEmbeddingRow {
|
||||
*/
|
||||
export function cosineSimilarity(a: Float32Array, b: Float32Array): number {
|
||||
if (a.length !== b.length) {
|
||||
throw new Error(
|
||||
`Embedding dimension mismatch: ${a.length} vs ${b.length}`
|
||||
);
|
||||
throw new Error(`Embedding dimension mismatch: ${a.length} vs ${b.length}`);
|
||||
}
|
||||
|
||||
let dot = 0;
|
||||
|
||||
@@ -27,10 +27,7 @@ function createTestDb(): Database.Database {
|
||||
client.pragma('foreign_keys = ON');
|
||||
|
||||
const migrationsFolder = join(import.meta.dirname, '../db/migrations');
|
||||
const migrationSql = readFileSync(
|
||||
join(migrationsFolder, '0000_large_master_chief.sql'),
|
||||
'utf-8'
|
||||
);
|
||||
const migrationSql = readFileSync(join(migrationsFolder, '0000_large_master_chief.sql'), 'utf-8');
|
||||
|
||||
// Drizzle migration files use `--> statement-breakpoint` as separator.
|
||||
const statements = migrationSql
|
||||
@@ -261,9 +258,7 @@ describe('RepositoryService.add()', () => {
|
||||
});
|
||||
|
||||
it('throws InvalidInputError when sourceUrl is empty', () => {
|
||||
expect(() =>
|
||||
service.add({ source: 'github', sourceUrl: '' })
|
||||
).toThrow(InvalidInputError);
|
||||
expect(() => service.add({ source: 'github', sourceUrl: '' })).toThrow(InvalidInputError);
|
||||
});
|
||||
|
||||
it('stores description and branch when provided', () => {
|
||||
@@ -321,9 +316,7 @@ describe('RepositoryService.update()', () => {
|
||||
});
|
||||
|
||||
it('throws NotFoundError for a non-existent repository', () => {
|
||||
expect(() =>
|
||||
service.update('/not/found', { title: 'New Title' })
|
||||
).toThrow(NotFoundError);
|
||||
expect(() => service.update('/not/found', { title: 'New Title' })).toThrow(NotFoundError);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -74,9 +74,7 @@ export class RepositoryService {
|
||||
.get(state) as { n: number };
|
||||
return row.n;
|
||||
}
|
||||
const row = this.db
|
||||
.prepare(`SELECT COUNT(*) as n FROM repositories`)
|
||||
.get() as { n: number };
|
||||
const row = this.db.prepare(`SELECT COUNT(*) as n FROM repositories`).get() as { n: number };
|
||||
return row.n;
|
||||
}
|
||||
|
||||
@@ -115,13 +113,13 @@ export class RepositoryService {
|
||||
}
|
||||
// Default title from owner/repo
|
||||
const parts = id.split('/').filter(Boolean);
|
||||
title = input.title ?? (parts[1] ?? id);
|
||||
title = input.title ?? parts[1] ?? id;
|
||||
} else {
|
||||
// local
|
||||
const existing = this.list({ limit: 9999 }).map((r) => r.id);
|
||||
id = resolveLocalId(input.sourceUrl, existing);
|
||||
const parts = input.sourceUrl.split('/');
|
||||
title = input.title ?? (parts.at(-1) ?? 'local-repo');
|
||||
title = input.title ?? parts.at(-1) ?? 'local-repo';
|
||||
}
|
||||
|
||||
// Check for collision
|
||||
|
||||
@@ -25,14 +25,8 @@ function createTestDb(): Database.Database {
|
||||
const migrationsFolder = join(import.meta.dirname, '../db/migrations');
|
||||
|
||||
// Apply all migration files in order
|
||||
const migration0 = readFileSync(
|
||||
join(migrationsFolder, '0000_large_master_chief.sql'),
|
||||
'utf-8'
|
||||
);
|
||||
const migration1 = readFileSync(
|
||||
join(migrationsFolder, '0001_quick_nighthawk.sql'),
|
||||
'utf-8'
|
||||
);
|
||||
const migration0 = readFileSync(join(migrationsFolder, '0000_large_master_chief.sql'), 'utf-8');
|
||||
const migration1 = readFileSync(join(migrationsFolder, '0001_quick_nighthawk.sql'), 'utf-8');
|
||||
|
||||
// Apply first migration
|
||||
const statements0 = migration0
|
||||
@@ -201,9 +195,7 @@ describe('VersionService.remove()', () => {
|
||||
|
||||
versionService.remove('/facebook/react', 'v18.3.0');
|
||||
|
||||
const doc = client
|
||||
.prepare(`SELECT id FROM documents WHERE id = ?`)
|
||||
.get(docId);
|
||||
const doc = client.prepare(`SELECT id FROM documents WHERE id = ?`).get(docId);
|
||||
expect(doc).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -40,12 +40,7 @@ export class VersionService {
|
||||
* @throws NotFoundError when the parent repository does not exist
|
||||
* @throws AlreadyExistsError when the tag is already registered
|
||||
*/
|
||||
add(
|
||||
repositoryId: string,
|
||||
tag: string,
|
||||
title?: string,
|
||||
commitHash?: string
|
||||
): RepositoryVersion {
|
||||
add(repositoryId: string, tag: string, title?: string, commitHash?: string): RepositoryVersion {
|
||||
// Verify parent repository exists.
|
||||
const repo = this.db
|
||||
.prepare(`SELECT id, source, source_url FROM repositories WHERE id = ?`)
|
||||
@@ -115,9 +110,7 @@ export class VersionService {
|
||||
*/
|
||||
getByTag(repositoryId: string, tag: string): RepositoryVersion | null {
|
||||
const row = this.db
|
||||
.prepare(
|
||||
`SELECT * FROM repository_versions WHERE repository_id = ? AND tag = ?`
|
||||
)
|
||||
.prepare(`SELECT * FROM repository_versions WHERE repository_id = ? AND tag = ?`)
|
||||
.get(repositoryId, tag) as RepositoryVersionEntity | undefined;
|
||||
return row ? RepositoryVersionMapper.fromEntity(new RepositoryVersionEntity(row)) : null;
|
||||
}
|
||||
@@ -137,9 +130,9 @@ export class VersionService {
|
||||
previousVersions: { tag: string; title: string; commitHash?: string }[]
|
||||
): RepositoryVersion[] {
|
||||
// Verify parent repository exists.
|
||||
const repo = this.db
|
||||
.prepare(`SELECT id FROM repositories WHERE id = ?`)
|
||||
.get(repositoryId) as { id: string } | undefined;
|
||||
const repo = this.db.prepare(`SELECT id FROM repositories WHERE id = ?`).get(repositoryId) as
|
||||
| { id: string }
|
||||
| undefined;
|
||||
|
||||
if (!repo) {
|
||||
throw new NotFoundError(`Repository ${repositoryId} not found`);
|
||||
|
||||
@@ -65,13 +65,10 @@ export function discoverVersionTags(options: DiscoverTagsOptions): string[] {
|
||||
|
||||
try {
|
||||
// List all tags, sorted by commit date (newest first)
|
||||
const output = execSync(
|
||||
`git -C "${repoPath}" tag -l --sort=-creatordate`,
|
||||
{
|
||||
const output = execSync(`git -C "${repoPath}" tag -l --sort=-creatordate`, {
|
||||
encoding: 'utf-8',
|
||||
stdio: ['ignore', 'pipe', 'pipe']
|
||||
}
|
||||
).trim();
|
||||
}).trim();
|
||||
|
||||
if (!output) return [];
|
||||
|
||||
|
||||
@@ -33,10 +33,11 @@ export function resolveLocalId(path: string, existingIds: string[]): string {
|
||||
* Slugify a string to be safe for use in IDs.
|
||||
*/
|
||||
function slugify(str: string): string {
|
||||
return str
|
||||
return (
|
||||
str
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9-_]/g, '-')
|
||||
.replace(/-+/g, '-')
|
||||
.replace(/^-|-$/g, '')
|
||||
|| 'repo';
|
||||
.replace(/^-|-$/g, '') || 'repo'
|
||||
);
|
||||
}
|
||||
|
||||
@@ -59,13 +59,10 @@ export function errorResponse(
|
||||
status: number,
|
||||
details?: Record<string, unknown>
|
||||
): Response {
|
||||
return new Response(
|
||||
JSON.stringify({ error, code, ...(details ? { details } : {}) }),
|
||||
{
|
||||
return new Response(JSON.stringify({ error, code, ...(details ? { details } : {}) }), {
|
||||
status,
|
||||
headers: { 'Content-Type': 'application/json' }
|
||||
}
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -20,15 +20,9 @@ import { createServer } from 'node:http';
|
||||
import { Server } from '@modelcontextprotocol/sdk/server/index.js';
|
||||
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
|
||||
import { StreamableHTTPServerTransport } from '@modelcontextprotocol/sdk/server/streamableHttp.js';
|
||||
import {
|
||||
CallToolRequestSchema,
|
||||
ListToolsRequestSchema
|
||||
} from '@modelcontextprotocol/sdk/types.js';
|
||||
import { CallToolRequestSchema, ListToolsRequestSchema } from '@modelcontextprotocol/sdk/types.js';
|
||||
|
||||
import {
|
||||
RESOLVE_LIBRARY_ID_TOOL,
|
||||
handleResolveLibraryId
|
||||
} from './tools/resolve-library-id.js';
|
||||
import { RESOLVE_LIBRARY_ID_TOOL, handleResolveLibraryId } from './tools/resolve-library-id.js';
|
||||
import { QUERY_DOCS_TOOL, handleQueryDocs } from './tools/query-docs.js';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -9,7 +9,9 @@ import { z } from 'zod';
|
||||
import { searchLibraries } from '../client.js';
|
||||
|
||||
export const ResolveLibraryIdSchema = z.object({
|
||||
libraryName: z.string().describe('Library name to search for and resolve to a TrueRef library ID'),
|
||||
libraryName: z
|
||||
.string()
|
||||
.describe('Library name to search for and resolve to a TrueRef library ID'),
|
||||
query: z.string().describe("The user's question or context to help rank results")
|
||||
});
|
||||
|
||||
|
||||
@@ -150,7 +150,9 @@
|
||||
{#if loading && jobs.length === 0}
|
||||
<div class="flex items-center justify-center py-12">
|
||||
<div class="text-center">
|
||||
<div class="inline-block h-8 w-8 animate-spin rounded-full border-4 border-solid border-blue-600 border-r-transparent"></div>
|
||||
<div
|
||||
class="inline-block h-8 w-8 animate-spin rounded-full border-4 border-solid border-blue-600 border-r-transparent"
|
||||
></div>
|
||||
<p class="mt-2 text-gray-600">Loading jobs...</p>
|
||||
</div>
|
||||
</div>
|
||||
@@ -160,26 +162,38 @@
|
||||
</div>
|
||||
{:else if jobs.length === 0}
|
||||
<div class="rounded-md bg-gray-50 p-8 text-center">
|
||||
<p class="text-gray-600">No jobs found. Jobs will appear here when repositories are indexed.</p>
|
||||
<p class="text-gray-600">
|
||||
No jobs found. Jobs will appear here when repositories are indexed.
|
||||
</p>
|
||||
</div>
|
||||
{:else}
|
||||
<div class="overflow-x-auto rounded-lg border border-gray-200 bg-white shadow">
|
||||
<table class="min-w-full divide-y divide-gray-200">
|
||||
<thead class="bg-gray-50">
|
||||
<tr>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider text-gray-500">
|
||||
<th
|
||||
class="px-6 py-3 text-left text-xs font-medium tracking-wider text-gray-500 uppercase"
|
||||
>
|
||||
Repository
|
||||
</th>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider text-gray-500">
|
||||
<th
|
||||
class="px-6 py-3 text-left text-xs font-medium tracking-wider text-gray-500 uppercase"
|
||||
>
|
||||
Status
|
||||
</th>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider text-gray-500">
|
||||
<th
|
||||
class="px-6 py-3 text-left text-xs font-medium tracking-wider text-gray-500 uppercase"
|
||||
>
|
||||
Progress
|
||||
</th>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider text-gray-500">
|
||||
<th
|
||||
class="px-6 py-3 text-left text-xs font-medium tracking-wider text-gray-500 uppercase"
|
||||
>
|
||||
Created
|
||||
</th>
|
||||
<th class="px-6 py-3 text-right text-xs font-medium uppercase tracking-wider text-gray-500">
|
||||
<th
|
||||
class="px-6 py-3 text-right text-xs font-medium tracking-wider text-gray-500 uppercase"
|
||||
>
|
||||
Actions
|
||||
</th>
|
||||
</tr>
|
||||
@@ -187,16 +201,16 @@
|
||||
<tbody class="divide-y divide-gray-200 bg-white">
|
||||
{#each jobs as job (job.id)}
|
||||
<tr class="hover:bg-gray-50">
|
||||
<td class="whitespace-nowrap px-6 py-4 text-sm font-medium text-gray-900">
|
||||
<td class="px-6 py-4 text-sm font-medium whitespace-nowrap text-gray-900">
|
||||
{job.repositoryId}
|
||||
{#if job.versionId}
|
||||
<span class="ml-1 text-xs text-gray-500">@{job.versionId}</span>
|
||||
{/if}
|
||||
</td>
|
||||
<td class="whitespace-nowrap px-6 py-4 text-sm text-gray-500">
|
||||
<td class="px-6 py-4 text-sm whitespace-nowrap text-gray-500">
|
||||
<JobStatusBadge status={job.status} />
|
||||
</td>
|
||||
<td class="whitespace-nowrap px-6 py-4 text-sm text-gray-500">
|
||||
<td class="px-6 py-4 text-sm whitespace-nowrap text-gray-500">
|
||||
<div class="flex items-center">
|
||||
<span class="mr-2">{job.progress}%</span>
|
||||
<div class="h-2 w-32 rounded-full bg-gray-200">
|
||||
@@ -212,10 +226,10 @@
|
||||
{/if}
|
||||
</div>
|
||||
</td>
|
||||
<td class="whitespace-nowrap px-6 py-4 text-sm text-gray-500">
|
||||
<td class="px-6 py-4 text-sm whitespace-nowrap text-gray-500">
|
||||
{formatDate(job.createdAt)}
|
||||
</td>
|
||||
<td class="whitespace-nowrap px-6 py-4 text-right text-sm font-medium">
|
||||
<td class="px-6 py-4 text-right text-sm font-medium whitespace-nowrap">
|
||||
<div class="flex justify-end gap-2">
|
||||
{#if canPause(job.status)}
|
||||
<button
|
||||
@@ -256,9 +270,7 @@
|
||||
</div>
|
||||
|
||||
{#if loading}
|
||||
<div class="mt-4 text-center text-sm text-gray-500">
|
||||
Refreshing...
|
||||
</div>
|
||||
<div class="mt-4 text-center text-sm text-gray-500">Refreshing...</div>
|
||||
{/if}
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
@@ -20,11 +20,7 @@ import { createProviderFromProfile } from '$lib/server/embeddings/registry';
|
||||
import type { EmbeddingProfile } from '$lib/server/db/schema';
|
||||
import { parseLibraryId } from '$lib/server/api/library-id';
|
||||
import { selectSnippetsWithinBudget, DEFAULT_TOKEN_BUDGET } from '$lib/server/api/token-budget';
|
||||
import {
|
||||
formatContextJson,
|
||||
formatContextTxt,
|
||||
CORS_HEADERS
|
||||
} from '$lib/server/api/formatters';
|
||||
import { formatContextJson, formatContextTxt, CORS_HEADERS } from '$lib/server/api/formatters';
|
||||
import type { ContextResponseMetadata } from '$lib/server/mappers/context-response.mapper';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -36,9 +32,10 @@ function getServices(db: ReturnType<typeof getClient>) {
|
||||
|
||||
// Load the active embedding profile from the database
|
||||
const profileRow = db
|
||||
.prepare<[], EmbeddingProfile>(
|
||||
'SELECT * FROM embedding_profiles WHERE is_default = 1 AND enabled = 1 LIMIT 1'
|
||||
)
|
||||
.prepare<
|
||||
[],
|
||||
EmbeddingProfile
|
||||
>('SELECT * FROM embedding_profiles WHERE is_default = 1 AND enabled = 1 LIMIT 1')
|
||||
.get();
|
||||
|
||||
const provider = profileRow ? createProviderFromProfile(profileRow) : null;
|
||||
@@ -53,7 +50,10 @@ interface RawRepoConfig {
|
||||
|
||||
function getRules(db: ReturnType<typeof getClient>, repositoryId: string): string[] {
|
||||
const row = db
|
||||
.prepare<[string], RawRepoConfig>(`SELECT rules FROM repository_configs WHERE repository_id = ?`)
|
||||
.prepare<
|
||||
[string],
|
||||
RawRepoConfig
|
||||
>(`SELECT rules FROM repository_configs WHERE repository_id = ?`)
|
||||
.get(repositoryId);
|
||||
|
||||
if (!row?.rules) return [];
|
||||
@@ -88,9 +88,10 @@ function getSnippetVersionTags(
|
||||
|
||||
const placeholders = versionIds.map(() => '?').join(', ');
|
||||
const rows = db
|
||||
.prepare<string[], RawVersionRow>(
|
||||
`SELECT id, tag FROM repository_versions WHERE id IN (${placeholders})`
|
||||
)
|
||||
.prepare<
|
||||
string[],
|
||||
RawVersionRow
|
||||
>(`SELECT id, tag FROM repository_versions WHERE id IN (${placeholders})`)
|
||||
.all(...versionIds);
|
||||
|
||||
return Object.fromEntries(rows.map((row) => [row.id, row.tag]));
|
||||
@@ -116,13 +117,10 @@ export const GET: RequestHandler = async ({ url }) => {
|
||||
const query = url.searchParams.get('query');
|
||||
|
||||
if (!query || !query.trim()) {
|
||||
return new Response(
|
||||
JSON.stringify({ error: 'query is required', code: 'MISSING_PARAMETER' }),
|
||||
{
|
||||
return new Response(JSON.stringify({ error: 'query is required', code: 'MISSING_PARAMETER' }), {
|
||||
status: 400,
|
||||
headers: { 'Content-Type': 'application/json', ...CORS_HEADERS }
|
||||
}
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
const responseType = url.searchParams.get('type') ?? 'json';
|
||||
@@ -157,9 +155,10 @@ export const GET: RequestHandler = async ({ url }) => {
|
||||
|
||||
// Verify the repository exists and check its state.
|
||||
const repo = db
|
||||
.prepare<[string], RawRepoState>(
|
||||
`SELECT id, state, title, source, source_url, branch FROM repositories WHERE id = ?`
|
||||
)
|
||||
.prepare<
|
||||
[string],
|
||||
RawRepoState
|
||||
>(`SELECT id, state, title, source, source_url, branch FROM repositories WHERE id = ?`)
|
||||
.get(parsed.repositoryId);
|
||||
|
||||
if (!repo) {
|
||||
@@ -193,9 +192,10 @@ export const GET: RequestHandler = async ({ url }) => {
|
||||
let resolvedVersion: RawVersionRow | undefined;
|
||||
if (parsed.version) {
|
||||
resolvedVersion = db
|
||||
.prepare<[string, string], RawVersionRow>(
|
||||
`SELECT id, tag FROM repository_versions WHERE repository_id = ? AND tag = ?`
|
||||
)
|
||||
.prepare<
|
||||
[string, string],
|
||||
RawVersionRow
|
||||
>(`SELECT id, tag FROM repository_versions WHERE repository_id = ? AND tag = ?`)
|
||||
.get(parsed.repositoryId, parsed.version);
|
||||
|
||||
// Version not found is not fatal — fall back to default branch.
|
||||
@@ -240,7 +240,8 @@ export const GET: RequestHandler = async ({ url }) => {
|
||||
sourceUrl: repo.source_url,
|
||||
branch: repo.branch
|
||||
},
|
||||
version: parsed.version || resolvedVersion
|
||||
version:
|
||||
parsed.version || resolvedVersion
|
||||
? {
|
||||
requested: parsed.version ?? null,
|
||||
resolved: resolvedVersion?.tag ?? null,
|
||||
|
||||
@@ -10,7 +10,7 @@ export const GET: RequestHandler = ({ url }) => {
|
||||
|
||||
let entries: { name: string; path: string; isGitRepo: boolean }[] = [];
|
||||
let error: string | null = null;
|
||||
let resolved = target;
|
||||
const resolved = target;
|
||||
|
||||
try {
|
||||
const items = fs.readdirSync(target, { withFileTypes: true });
|
||||
|
||||
@@ -7,7 +7,11 @@ import type { RequestHandler } from './$types';
|
||||
import { getClient } from '$lib/server/db/client.js';
|
||||
import { IndexingJobMapper } from '$lib/server/mappers/indexing-job.mapper.js';
|
||||
import { JobQueue } from '$lib/server/pipeline/job-queue.js';
|
||||
import { handleServiceError, NotFoundError, InvalidInputError } from '$lib/server/utils/validation.js';
|
||||
import {
|
||||
handleServiceError,
|
||||
NotFoundError,
|
||||
InvalidInputError
|
||||
} from '$lib/server/utils/validation.js';
|
||||
|
||||
export const POST: RequestHandler = ({ params }) => {
|
||||
try {
|
||||
@@ -19,9 +23,7 @@ export const POST: RequestHandler = ({ params }) => {
|
||||
|
||||
const success = queue.cancelJob(params.id);
|
||||
if (!success) {
|
||||
throw new InvalidInputError(
|
||||
`Cannot cancel job ${params.id} - job is already done or failed`
|
||||
);
|
||||
throw new InvalidInputError(`Cannot cancel job ${params.id} - job is already done or failed`);
|
||||
}
|
||||
|
||||
// Fetch updated job
|
||||
|
||||
@@ -7,7 +7,11 @@ import type { RequestHandler } from './$types';
|
||||
import { getClient } from '$lib/server/db/client.js';
|
||||
import { IndexingJobMapper } from '$lib/server/mappers/indexing-job.mapper.js';
|
||||
import { JobQueue } from '$lib/server/pipeline/job-queue.js';
|
||||
import { handleServiceError, NotFoundError, InvalidInputError } from '$lib/server/utils/validation.js';
|
||||
import {
|
||||
handleServiceError,
|
||||
NotFoundError,
|
||||
InvalidInputError
|
||||
} from '$lib/server/utils/validation.js';
|
||||
|
||||
export const POST: RequestHandler = ({ params }) => {
|
||||
try {
|
||||
|
||||
@@ -7,7 +7,11 @@ import type { RequestHandler } from './$types';
|
||||
import { getClient } from '$lib/server/db/client.js';
|
||||
import { IndexingJobMapper } from '$lib/server/mappers/indexing-job.mapper.js';
|
||||
import { JobQueue } from '$lib/server/pipeline/job-queue.js';
|
||||
import { handleServiceError, NotFoundError, InvalidInputError } from '$lib/server/utils/validation.js';
|
||||
import {
|
||||
handleServiceError,
|
||||
NotFoundError,
|
||||
InvalidInputError
|
||||
} from '$lib/server/utils/validation.js';
|
||||
|
||||
export const POST: RequestHandler = ({ params }) => {
|
||||
try {
|
||||
@@ -19,7 +23,9 @@ export const POST: RequestHandler = ({ params }) => {
|
||||
|
||||
const success = queue.resumeJob(params.id);
|
||||
if (!success) {
|
||||
throw new InvalidInputError(`Cannot resume job ${params.id} - only paused jobs can be resumed`);
|
||||
throw new InvalidInputError(
|
||||
`Cannot resume job ${params.id} - only paused jobs can be resumed`
|
||||
);
|
||||
}
|
||||
|
||||
// Fetch updated job
|
||||
|
||||
@@ -58,9 +58,7 @@ export const POST: RequestHandler = async ({ request }) => {
|
||||
let jobResponse: ReturnType<typeof IndexingJobMapper.toDto> | null = null;
|
||||
if (body.autoIndex !== false) {
|
||||
const queue = getQueue();
|
||||
const job = queue
|
||||
? queue.enqueue(repo.id)
|
||||
: service.createIndexingJob(repo.id);
|
||||
const job = queue ? queue.enqueue(repo.id) : service.createIndexingJob(repo.id);
|
||||
jobResponse = IndexingJobMapper.toDto(job);
|
||||
}
|
||||
|
||||
|
||||
@@ -28,9 +28,7 @@ export const POST: RequestHandler = async ({ params, request }) => {
|
||||
// Use the queue so processNext() is triggered immediately.
|
||||
// Falls back to direct DB insert if the queue isn't initialised yet.
|
||||
const queue = getQueue();
|
||||
const job = queue
|
||||
? queue.enqueue(id, versionId)
|
||||
: service.createIndexingJob(id, versionId);
|
||||
const job = queue ? queue.enqueue(id, versionId) : service.createIndexingJob(id, versionId);
|
||||
|
||||
return json({ job: IndexingJobMapper.toDto(job) }, { status: 202 });
|
||||
} catch (err) {
|
||||
|
||||
@@ -146,4 +146,3 @@ function sanitizeProfile(profile: EmbeddingProfile): EmbeddingProfile {
|
||||
}
|
||||
return profile;
|
||||
}
|
||||
|
||||
|
||||
@@ -16,9 +16,10 @@ export const GET: RequestHandler = async () => {
|
||||
try {
|
||||
const db = getClient();
|
||||
const profile = db
|
||||
.prepare<[], EmbeddingProfile>(
|
||||
'SELECT * FROM embedding_profiles WHERE is_default = 1 AND enabled = 1 LIMIT 1'
|
||||
)
|
||||
.prepare<
|
||||
[],
|
||||
EmbeddingProfile
|
||||
>('SELECT * FROM embedding_profiles WHERE is_default = 1 AND enabled = 1 LIMIT 1')
|
||||
.get();
|
||||
|
||||
if (!profile) {
|
||||
@@ -42,7 +43,6 @@ export const GET: RequestHandler = async () => {
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
export const POST: RequestHandler = async ({ request }) => {
|
||||
try {
|
||||
const body = await request.json();
|
||||
|
||||
@@ -8,7 +8,9 @@
|
||||
let { data }: { data: PageData } = $props();
|
||||
|
||||
// Initialized empty; $effect syncs from data prop on every navigation/reload.
|
||||
let repo = $state<Repository & { versions?: RepositoryVersion[] }>({} as Repository & { versions?: RepositoryVersion[] });
|
||||
let repo = $state<Repository & { versions?: RepositoryVersion[] }>(
|
||||
{} as Repository & { versions?: RepositoryVersion[] }
|
||||
);
|
||||
let recentJobs = $state<IndexingJob[]>([]);
|
||||
$effect(() => {
|
||||
if (data.repo) repo = data.repo;
|
||||
@@ -189,7 +191,7 @@
|
||||
<dl class="grid grid-cols-1 gap-y-2 text-sm sm:grid-cols-2">
|
||||
<div class="flex gap-2">
|
||||
<dt class="text-gray-500">Source</dt>
|
||||
<dd class="font-medium capitalize text-gray-900">{repo.source}</dd>
|
||||
<dd class="font-medium text-gray-900 capitalize">{repo.source}</dd>
|
||||
</div>
|
||||
<div class="flex gap-2">
|
||||
<dt class="text-gray-500">Branch</dt>
|
||||
|
||||
@@ -227,7 +227,9 @@
|
||||
<div class="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
|
||||
<div class="mb-4 flex items-center gap-3">
|
||||
<div class="flex min-w-0 flex-1 items-center gap-2">
|
||||
<span class="shrink-0 rounded bg-green-100 px-2 py-0.5 text-xs font-medium text-green-700">
|
||||
<span
|
||||
class="shrink-0 rounded bg-green-100 px-2 py-0.5 text-xs font-medium text-green-700"
|
||||
>
|
||||
Selected
|
||||
</span>
|
||||
<span class="truncate font-mono text-sm text-gray-700">{selectedLibraryTitle}</span>
|
||||
@@ -285,7 +287,9 @@
|
||||
{:else if query && !loadingSnippets && snippets.length === 0 && !snippetError}
|
||||
<div class="flex flex-col items-center py-16 text-center">
|
||||
<p class="text-sm text-gray-500">No snippets found for that query.</p>
|
||||
<p class="mt-1 text-xs text-gray-400">Try a different question or select another library.</p>
|
||||
<p class="mt-1 text-xs text-gray-400">
|
||||
Try a different question or select another library.
|
||||
</p>
|
||||
</div>
|
||||
{/if}
|
||||
{/if}
|
||||
|
||||
@@ -203,7 +203,11 @@
|
||||
: 'border border-gray-200 text-gray-700 hover:bg-gray-50'
|
||||
].join(' ')}
|
||||
>
|
||||
{p === 'none' ? 'None (FTS5 only)' : p === 'openai' ? 'OpenAI-compatible' : 'Local Model'}
|
||||
{p === 'none'
|
||||
? 'None (FTS5 only)'
|
||||
: p === 'openai'
|
||||
? 'OpenAI-compatible'
|
||||
: 'Local Model'}
|
||||
</button>
|
||||
{/each}
|
||||
</div>
|
||||
|
||||
Reference in New Issue
Block a user