diff --git a/.github/agents b/.github/agents deleted file mode 120000 index e9ce9b9..0000000 --- a/.github/agents +++ /dev/null @@ -1 +0,0 @@ -/home/moze/Sources/copilot-agents/.github/agents \ No newline at end of file diff --git a/.github/schemas b/.github/schemas deleted file mode 120000 index f200112..0000000 --- a/.github/schemas +++ /dev/null @@ -1 +0,0 @@ -/home/moze/Sources/copilot-agents/.github/schemas \ No newline at end of file diff --git a/.github/skills b/.github/skills deleted file mode 120000 index 74398ae..0000000 --- a/.github/skills +++ /dev/null @@ -1 +0,0 @@ -/home/moze/Sources/copilot-agents/.github/skills \ No newline at end of file diff --git a/.gitignore b/.gitignore index 71925a2..4aa4427 100644 --- a/.gitignore +++ b/.gitignore @@ -36,3 +36,8 @@ docs/docs_cache_state.yaml # Claude Code — ignore local/machine-specific settings, keep project rules .claude/ !.claude/rules/ + +# Github Copilot +.github/agents +.github/schemas +.github/skills diff --git a/README.md b/README.md index e855d93..d2a0e17 100644 --- a/README.md +++ b/README.md @@ -16,9 +16,12 @@ The goal is straightforward: give your assistants accurate, current, version-awa - Stores metadata in SQLite. - Supports keyword search out of the box with SQLite FTS5. - Supports semantic and hybrid search when an embedding provider is configured. -- Exposes REST endpoints for library discovery and documentation retrieval. +- Supports multi-version indexing: index specific git tags independently, query a version by appending it to the library ID. +- Discovers available git tags from local repositories automatically. +- Stores per-version rules from `trueref.json` and prepends them to every `query-docs` response. +- Exposes REST endpoints for library discovery, documentation retrieval, and version management. - Exposes an MCP server over stdio and HTTP for AI clients. -- Provides a SvelteKit web UI for repository management, search, indexing jobs, and embedding settings. +- Provides a SvelteKit web UI for repository management, version management, search, indexing jobs, and embedding settings. - Supports repository-level configuration through `trueref.json` or `context7.json`. ## Project status @@ -28,10 +31,12 @@ TrueRef is under active development. The current codebase already includes: - repository management - indexing jobs and recovery on restart - local and GitHub crawling -- version registration support +- multi-version indexing with git tag isolation +- automatic tag discovery for local git repositories +- per-version rules from `trueref.json` prepended to context responses - context7-compatible API endpoints - MCP stdio and HTTP transports -- configurable embedding providers +- configurable embedding providers (none / OpenAI-compatible / local ONNX) ## Architecture @@ -66,7 +71,15 @@ Each indexed repository becomes a library with an ID such as `/facebook/react`. ### Versions -Libraries can register version tags. Queries can target a specific version by using a library ID such as `/facebook/react/v18.3.0`. +Libraries can register version tags. Each version is indexed independently so snippets from different releases never mix. + +Query a specific version by appending the tag to the library ID: + +``` +/facebook/react/v18.3.0 +``` + +For local repositories, TrueRef can discover all available git tags automatically via the versions/discover endpoint. Tags can be added through the UI on the repository detail page or via the REST API. ### Snippets @@ -76,6 +89,8 @@ Documents are split into code and info snippets. These snippets are what search Repository rules defined in `trueref.json` are prepended to `query-docs` responses so assistants get usage constraints along with the retrieved content. +Rules are stored per version when a version-specific config is found during indexing, so different releases can carry different usage guidance. + ## Requirements - Node.js 20+ @@ -153,6 +168,12 @@ Use the main page to: - delete an indexed repository - monitor active indexing jobs +Open a repository's detail page to: + +- view registered version tags +- discover available git tags (local repositories) +- trigger version-specific indexing jobs + ### Search Use the Search page to: @@ -175,21 +196,40 @@ If no embedding provider is configured, TrueRef still works with FTS5-only searc ## Repository configuration -TrueRef supports a repository-local config file named `trueref.json`. +You can place a `trueref.json` file at the **root** of any repository you index. TrueRef reads it during every indexing run to control what gets indexed and what gets shown to AI assistants. -For compatibility with existing context7-style repositories, `context7.json` is also supported. +For backward compatibility with repositories that already have a `context7.json`, that file is also supported. When both files are present, `trueref.json` takes precedence. -### What the config controls +### Where to place it -- project display title -- project description -- included folders -- excluded folders -- excluded file names -- assistant-facing usage rules -- previously released versions +``` +my-library/ +├── trueref.json ← here, at the repository root +├── src/ +├── docs/ +└── ... +``` -### Example `trueref.json` +For GitHub repositories, TrueRef fetches the file from the default branch root. For local repositories, it reads it from the filesystem root of the indexed folder. + +### Fields + +| Field | Type | Required | Description | +|---|---|---|---| +| `$schema` | string | No | URL to the live JSON Schema for editor validation | +| `projectTitle` | string | No | Display name override (max 100 chars) | +| `description` | string | No | Library description used for search ranking (10–500 chars) | +| `folders` | string[] | No | Path prefixes or regex strings to **include** (max 50 items). If absent, all folders are included | +| `excludeFolders` | string[] | No | Path prefixes or regex strings to **exclude** after the `folders` allowlist (max 50 items) | +| `excludeFiles` | string[] | No | Exact filenames to skip — no path, no glob (max 100 items) | +| `rules` | string[] | No | Best-practice rules prepended to every `query-docs` response (max 20 rules, 5–500 chars each) | +| `previousVersions` | object[] | No | Version tags to register when the repository is indexed (max 50 entries) | + +`previousVersions` entries each require a `tag` (e.g. `"v1.2.3"`) and a `title` (e.g. `"Version 1.2.3"`). + +The parser is intentionally lenient: unknown keys are silently ignored, mistyped values are skipped with a warning, and oversized strings or arrays are truncated. Only invalid JSON or a non-object root is a hard error. + +### Full example ```json { @@ -197,30 +237,76 @@ For compatibility with existing context7-style repositories, `context7.json` is "projectTitle": "My Internal SDK", "description": "Internal SDK for billing, auth, and event ingestion.", "folders": ["src/", "docs/"], - "excludeFolders": ["tests/", "fixtures/", "node_modules/"], - "excludeFiles": ["CHANGELOG.md"], + "excludeFolders": ["tests/", "fixtures/", "node_modules/", "__mocks__/"], + "excludeFiles": ["CHANGELOG.md", "jest.config.ts"], "rules": [ "Prefer named imports over wildcard imports.", - "Use the async client API for all network calls." + "Use the async client API for all network calls.", + "Never import from internal sub-paths — use the package root only." ], "previousVersions": [ - { - "tag": "v1.2.3", - "title": "Version 1.2.3" - } + { "tag": "v2.0.0", "title": "Version 2.0.0" }, + { "tag": "v1.2.3", "title": "Version 1.2.3 (legacy)" } ] } ``` -### JSON schema +### How `folders` and `excludeFolders` are matched -You can point your editor to the live schema served by TrueRef: +Both fields accept strings that are matched against the full relative file path within the repository. A string is treated as a path prefix unless it starts with `^`, in which case it is compiled as a regex: -```text +```json +{ + "folders": ["src/", "docs/", "^packages/core"], + "excludeFolders": ["src/internal/", "__tests__"] +} +``` + +- `"src/"` — includes any file whose path starts with `src/` +- `"^packages/core"` — regex, includes only `packages/core` not `packages/core-utils` + +`excludeFolders` is applied **after** the `folders` allowlist, so you can narrow a broad include with a targeted exclude. + +### How `rules` are used + +Rules are stored in the database at index time and automatically prepended to every `query-docs` response for that library (and version). This means AI assistants receive them alongside the retrieved snippets without any extra configuration. + +When a version is indexed, the rules from the config found at that version's checkout are stored separately. Different version tags can therefore carry different rules. + +Example context response with rules prepended: + +``` +RULES: +- Prefer named imports over wildcard imports. +- Use the async client API for all network calls. + +LIBRARY DOCUMENTATION: +... +``` + +### How `previousVersions` works + +When TrueRef indexes a repository and finds `previousVersions`, it registers those tags in the versions table. The tags are then available for version-specific indexing and queries without any further manual registration. + +This is useful when you want all historical releases available from a fresh TrueRef setup without manually triggering one indexing job per version. + +### JSON Schema for editor support + +TrueRef serves a live JSON Schema at: + +``` http://localhost:5173/api/v1/schema/trueref-config.json ``` -That enables validation and autocomplete in editors that support JSON Schema references. +Add it to your `trueref.json` via the `$schema` field to get inline validation and autocomplete in VS Code, IntelliJ, and any other editor that supports JSON Schema Draft 07: + +```json +{ + "$schema": "http://localhost:5173/api/v1/schema/trueref-config.json" +} +``` + +If you are running TrueRef on a server, replace `localhost:5173` with your actual host and port. The schema endpoint always reflects the version of TrueRef you are running. ## REST API @@ -299,6 +385,36 @@ curl "http://localhost:5173/api/v1/jobs" curl "http://localhost:5173/api/v1/jobs/" ``` +### Version management + +List registered versions for a library: + +```sh +curl "http://localhost:5173/api/v1/libs/%2Ffacebook%2Freact/versions" +``` + +Index a specific version tag: + +```sh +curl -X POST "http://localhost:5173/api/v1/libs/%2Ffacebook%2Freact/versions/v18.3.0/index" +``` + +Discover available git tags (local repositories only): + +```sh +curl -X POST "http://localhost:5173/api/v1/libs/%2Fpath%2Fto%2Fmy-lib/versions/discover" +``` + +Returns `{ "tags": [{ "tag": "v1.0.0", "commitHash": "abc123" }, ...] }`. Returns an empty array for GitHub repositories. + +### Version-targeted context retrieval + +Append the version tag to the library ID to retrieve snippets from a specific indexed version: + +```sh +curl "http://localhost:5173/api/v1/context?libraryId=/facebook/react/v18.3.0&query=how%20to%20use%20useEffect&type=txt" +``` + ### Response formats The two search endpoints support: