Initial commit: trueref v0.1.0-SNAPSHOT
Some checks failed
Build and publish Docker image / Build and push (push) Failing after 1m27s
Some checks failed
Build and publish Docker image / Build and push (push) Failing after 1m27s
Java 21 / Spring Boot 3.5.3 multi-module Maven project. Hybrid BM25+HNSW search with RRF, cross-encoder reranker, ONNX Runtime 1.22.0 (CPU + CUDA 12 GPU variants).
This commit is contained in:
moze
85
.gitea/workflows/docker.yml
Normal file
85
.gitea/workflows/docker.yml
Normal file
@@ -0,0 +1,85 @@
|
|||||||
|
name: Build and publish Docker image
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches:
|
||||||
|
- main
|
||||||
|
- master
|
||||||
|
tags:
|
||||||
|
- 'v*.*.*'
|
||||||
|
workflow_dispatch:
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
docker:
|
||||||
|
name: Build and push
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
|
# Set up Docker Buildx for efficient layer caching.
|
||||||
|
- name: Set up Docker Buildx
|
||||||
|
uses: docker/setup-buildx-action@v3
|
||||||
|
|
||||||
|
# Log in to the Gitea container registry.
|
||||||
|
# The built-in GITEA_TOKEN is injected automatically by Gitea Actions and
|
||||||
|
# has write access to packages in the same organisation/user namespace.
|
||||||
|
- name: Log in to Gitea registry
|
||||||
|
uses: docker/login-action@v3
|
||||||
|
with:
|
||||||
|
registry: git.sal.giize.com
|
||||||
|
username: ${{ gitea.actor }}
|
||||||
|
password: ${{ secrets.GITEA_TOKEN }}
|
||||||
|
|
||||||
|
# ── Determine tags ───────────────────────────────────────────────────
|
||||||
|
# On a version tag (v1.2.3): latest, cpu, cpu-1.2.3, 1.2.3
|
||||||
|
# On branch push (main/master): latest, cpu
|
||||||
|
- name: Docker metadata (CPU)
|
||||||
|
id: meta_cpu
|
||||||
|
uses: docker/metadata-action@v5
|
||||||
|
with:
|
||||||
|
images: git.sal.giize.com/mozempk/trueref
|
||||||
|
flavor: |
|
||||||
|
latest=auto
|
||||||
|
tags: |
|
||||||
|
type=raw,value=latest,enable={{is_default_branch}}
|
||||||
|
type=raw,value=cpu,enable={{is_default_branch}}
|
||||||
|
type=semver,pattern={{version}},prefix=cpu-
|
||||||
|
type=semver,pattern={{version}}
|
||||||
|
|
||||||
|
- name: Docker metadata (GPU)
|
||||||
|
id: meta_gpu
|
||||||
|
uses: docker/metadata-action@v5
|
||||||
|
with:
|
||||||
|
images: git.sal.giize.com/mozempk/trueref
|
||||||
|
flavor: |
|
||||||
|
latest=false
|
||||||
|
tags: |
|
||||||
|
type=raw,value=gpu,enable={{is_default_branch}}
|
||||||
|
type=semver,pattern={{version}},prefix=gpu-
|
||||||
|
|
||||||
|
# ── CPU image ────────────────────────────────────────────────────────
|
||||||
|
- name: Build and push CPU image
|
||||||
|
uses: docker/build-push-action@v5
|
||||||
|
with:
|
||||||
|
context: .
|
||||||
|
file: Dockerfile
|
||||||
|
push: true
|
||||||
|
tags: ${{ steps.meta_cpu.outputs.tags }}
|
||||||
|
labels: ${{ steps.meta_cpu.outputs.labels }}
|
||||||
|
cache-from: type=gha,scope=cpu
|
||||||
|
cache-to: type=gha,mode=max,scope=cpu
|
||||||
|
|
||||||
|
# ── GPU image ────────────────────────────────────────────────────────
|
||||||
|
# Built from the same source; only the runtime base image differs.
|
||||||
|
- name: Build and push GPU image
|
||||||
|
uses: docker/build-push-action@v5
|
||||||
|
with:
|
||||||
|
context: .
|
||||||
|
file: Dockerfile.gpu
|
||||||
|
push: true
|
||||||
|
tags: ${{ steps.meta_gpu.outputs.tags }}
|
||||||
|
labels: ${{ steps.meta_gpu.outputs.labels }}
|
||||||
|
cache-from: type=gha,scope=gpu
|
||||||
|
cache-to: type=gha,mode=max,scope=gpu
|
||||||
30
.gitignore
vendored
Normal file
30
.gitignore
vendored
Normal file
@@ -0,0 +1,30 @@
|
|||||||
|
target/
|
||||||
|
build/
|
||||||
|
out/
|
||||||
|
.idea/
|
||||||
|
.vscode/
|
||||||
|
*.iml
|
||||||
|
*.ipr
|
||||||
|
*.iws
|
||||||
|
.DS_Store
|
||||||
|
|
||||||
|
# Maven
|
||||||
|
.mvn/wrapper/maven-wrapper.jar
|
||||||
|
|
||||||
|
# trueref runtime data (models, DB, index — too large / machine-specific)
|
||||||
|
data/
|
||||||
|
data-onnx-smoke/
|
||||||
|
logs/
|
||||||
|
|
||||||
|
# cuDNN and other large native runtime libraries
|
||||||
|
runtime/
|
||||||
|
|
||||||
|
# JVM crash dumps
|
||||||
|
hs_err_pid*.log
|
||||||
|
core.*
|
||||||
|
|
||||||
|
# Frontend
|
||||||
|
trueref-frontend/web/node_modules/
|
||||||
|
trueref-frontend/web/build/
|
||||||
|
trueref-frontend/web/.svelte-kit/
|
||||||
|
node_modules/
|
||||||
428
ARCHITECTURE.md
Normal file
428
ARCHITECTURE.md
Normal file
@@ -0,0 +1,428 @@
|
|||||||
|
# trueref — Architecture
|
||||||
|
|
||||||
|
> Self-hosted, fat-JAR, Java-21 clone of [Context7](https://github.com/upstash/context7) ingestion + retrieval, with first-class differential per-tag indexing, embedded vector + BM25 store, ONNX-accelerated embeddings/rerank, Streamable-HTTP MCP server, REST + OpenAPI, and a SvelteKit UI.
|
||||||
|
|
||||||
|
## 1. Goals & Non-Goals
|
||||||
|
|
||||||
|
### Goals
|
||||||
|
- **Functional parity with Context7** ingestion outcome (own chunk schema).
|
||||||
|
- **Differential per-tag indexing**: every git tag of every registered repo is independently queryable.
|
||||||
|
- **Embedded everything**: single fat JAR runnable on a workstation/server. No external Postgres/Qdrant.
|
||||||
|
- **GPU-accelerated retrieval** via ONNX Runtime (CUDA Linux/Win, DirectML Win, CPU fallback).
|
||||||
|
- **MCP Streamable-HTTP server** exposing exactly two tools: `resolve-library-id`, `get-library-docs` — drop-in for any MCP client.
|
||||||
|
- **Full observability** of ingestion pipelines surfaced in the UI (live progress, log tail, history, timings, resource usage).
|
||||||
|
- **REST + OpenAPI/Swagger** for programmatic and UI use.
|
||||||
|
- **SvelteKit UI** for repo registration, indexing control, monitoring, and ad-hoc query.
|
||||||
|
- **Hexagonal architecture** so vector store, embedder, parser, persistence, etc. are swappable.
|
||||||
|
|
||||||
|
### Non-Goals
|
||||||
|
- No public hosted SaaS — self-host only.
|
||||||
|
- No model fine-tuning.
|
||||||
|
- No mobile app.
|
||||||
|
- No generative LLM in the pipeline (retrieval-only, like Context7).
|
||||||
|
- No multi-tenancy / auth (LAN-only deployment).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Tech Stack (locked)
|
||||||
|
|
||||||
|
| Concern | Choice | Rationale |
|
||||||
|
|---|---|---|
|
||||||
|
| Language / runtime | **Java 21 LTS** | Virtual threads stable; Spring Boot 3.5 supported. (Java 25 dropped — Boot 3.5 supports up to 23.) |
|
||||||
|
| Framework | **Spring Boot 3.5.x** + **Spring AI 1.0.x** | Web MVC + virtual-thread executor; Spring AI for embedding/MCP abstractions. |
|
||||||
|
| Build | **Maven** | Stable, ubiquitous, Spring-Boot first-class. |
|
||||||
|
| Metadata store | **H2 (MVCC mode, file-based)** + Flyway | Zero ops, JDBC, MVCC concurrency, fits fat JAR. |
|
||||||
|
| Vector + lexical store | **Apache Lucene 9.x** | Pure JVM. BM25 + HNSW kNN in one index. Collapses two stores. |
|
||||||
|
| Embedding model | **BAAI/bge-m3** (ONNX) | Multilingual, 8k context, dense+sparse capable. MIT-like license. |
|
||||||
|
| Reranker | **BAAI/bge-reranker-v2-m3** (ONNX) | Cross-encoder, Apache 2.0. |
|
||||||
|
| ML runtime | **ONNX Runtime** (`onnxruntime_gpu` Linux CUDA / `onnxruntime-directml` Win / `onnxruntime` CPU) | In-JVM via official Java bindings. |
|
||||||
|
| Git | **JGit** | Pure Java; clone, fetch, tag enumeration, diff. |
|
||||||
|
| Code parsing | **Pure-Java heuristic chunker** (markdown-aware, brace-balanced for C-family, indent-based for Python, sliding-window fallback) | No native deps; preserves fat-JAR purity. Tree-sitter is a documented future swap (see FINDINGS §F11). |
|
||||||
|
| Job orchestration | **Custom virtual-thread orchestrator** + H2-backed durable state | Fast, no Spring Batch overhead. |
|
||||||
|
| MCP server | **Spring AI MCP Server (Streamable HTTP)** | Spec 2025-03-26, single `/mcp` endpoint. |
|
||||||
|
| REST docs | **springdoc-openapi** | OpenAPI 3 + Swagger UI auto-generated. |
|
||||||
|
| Observability | **Micrometer + OpenTelemetry**, exposed via REST/SSE for UI. **Prometheus + Grafana optional** via `/actuator/prometheus`. | UI-first; Prom/Graf attach later. |
|
||||||
|
| Frontend | **SvelteKit + `@sveltejs/adapter-static`** | Built into `bootstrap/src/main/resources/static/`, served by Spring as part of fat JAR. |
|
||||||
|
| Packaging | **Single fat JAR** via `spring-boot-maven-plugin` | One artifact, embedded everything. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Hexagonal Layout (Maven multi-module)
|
||||||
|
|
||||||
|
Direction of dependencies is enforced by Maven coordinates alone — no ArchUnit needed.
|
||||||
|
|
||||||
|
```
|
||||||
|
trueref-parent/ (pom; BOM + plugin management)
|
||||||
|
├── trueref-domain pure Java; records, sealed types, port interfaces. ZERO deps.
|
||||||
|
├── trueref-application use-case impls; depends on: domain
|
||||||
|
├── trueref-adapters ALL adapters live here; depends on: domain, application
|
||||||
|
│ └── com.trueref.adapter
|
||||||
|
│ ├── in
|
||||||
|
│ │ ├── rest @RestController + DTOs + OpenAPI + SSE
|
||||||
|
│ │ └── mcp MCP tool defs (Spring AI MCP server)
|
||||||
|
│ └── out
|
||||||
|
│ ├── persistence.h2 JdbcClient + Flyway, RepositoryStore impl
|
||||||
|
│ ├── vectorstore.lucene Lucene BM25 + HNSW kNN, ChunkStore impl
|
||||||
|
│ ├── embedding.onnx ONNX bge-m3 + bge-reranker-v2-m3
|
||||||
|
│ ├── git.jgit GitClient impl
|
||||||
|
│ ├── parsing.treesitter CodeParser impl
|
||||||
|
│ └── cache.disk EmbeddingCache (file-per-hash)
|
||||||
|
├── trueref-frontend SvelteKit; built via frontend-maven-plugin into static jar
|
||||||
|
└── trueref-bootstrap @SpringBootApplication; wires beans; produces fat JAR
|
||||||
|
depends on: domain, application, adapters, frontend
|
||||||
|
```
|
||||||
|
|
||||||
|
**Dependency rule (Maven-enforced):**
|
||||||
|
- `domain` → nothing.
|
||||||
|
- `application` → `domain`.
|
||||||
|
- `adapters` → `domain` + `application`.
|
||||||
|
- `frontend` → none (resource-only jar).
|
||||||
|
- `bootstrap` → all of the above (the only place wiring lives).
|
||||||
|
|
||||||
|
> All packages live under `com.trueref.*` regardless of module. Module boundaries enforce dependency direction; package layout inside `adapters` mirrors the in/out hexagonal convention.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Core Domain Model
|
||||||
|
|
||||||
|
```
|
||||||
|
Repository {
|
||||||
|
id: UUID
|
||||||
|
name: String // "spring-projects/spring-boot"
|
||||||
|
remoteUrl: String? // null if local-only
|
||||||
|
localPath: Path // either user-provided or our managed clone dir
|
||||||
|
managedClone: bool // true if WE clone/fetch
|
||||||
|
ignoreGlobs: List<String> // per-repo overrides
|
||||||
|
maxFileSizeBytes: long // default 1MB
|
||||||
|
pollIntervalSec: long // default 3600; 0 disables polling
|
||||||
|
versionMappingRules: List<TagPattern> // exact, v-prefix, release-prefix, regex
|
||||||
|
createdAt, updatedAt
|
||||||
|
}
|
||||||
|
|
||||||
|
Version {
|
||||||
|
id: UUID
|
||||||
|
repoId: UUID
|
||||||
|
tag: String // "v1.2.3" or branch name
|
||||||
|
commitSha: String
|
||||||
|
status: enum { DISCOVERED, INDEXING, INDEXED, FAILED, INACTIVE }
|
||||||
|
indexedAt: Instant?
|
||||||
|
chunkCount: int
|
||||||
|
errorMessage: String?
|
||||||
|
}
|
||||||
|
|
||||||
|
Chunk { // global, deduplicated by content_hash
|
||||||
|
id: UUID
|
||||||
|
contentHash: String // sha256 of canonicalized content
|
||||||
|
content: String // the snippet text
|
||||||
|
language: String // "java", "python", "markdown", ...
|
||||||
|
symbol: String? // function/class name if AST-extracted
|
||||||
|
tokenCount: int
|
||||||
|
// dense + sparse vectors stored in Lucene index, not here
|
||||||
|
}
|
||||||
|
|
||||||
|
ChunkVersion { // many-to-many: which versions contain which chunks
|
||||||
|
chunkId: UUID
|
||||||
|
versionId: UUID
|
||||||
|
filePath: String
|
||||||
|
startLine: int
|
||||||
|
endLine: int
|
||||||
|
// PK (chunkId, versionId, filePath, startLine)
|
||||||
|
}
|
||||||
|
|
||||||
|
IngestionJob {
|
||||||
|
id: UUID
|
||||||
|
repoId: UUID
|
||||||
|
versionId: UUID? // null = repo-level (e.g. discovery)
|
||||||
|
type: enum { DISCOVER_TAGS, INDEX_VERSION, COMPACT, REFRESH }
|
||||||
|
status: enum { QUEUED, RUNNING, SUCCEEDED, FAILED, CANCELLED }
|
||||||
|
startedAt, finishedAt
|
||||||
|
stages: List<JobStage>
|
||||||
|
}
|
||||||
|
|
||||||
|
JobStage {
|
||||||
|
jobId: UUID
|
||||||
|
name: enum { CLONE, FETCH, CHECKOUT, DISCOVER_FILES, PARSE, CHUNK, EMBED, INDEX, COMMIT }
|
||||||
|
status: enum { PENDING, RUNNING, SUCCEEDED, FAILED, SKIPPED }
|
||||||
|
startedAt, finishedAt
|
||||||
|
itemsProcessed: long
|
||||||
|
itemsTotal: long
|
||||||
|
bytesProcessed: long
|
||||||
|
errorMessage: String?
|
||||||
|
}
|
||||||
|
|
||||||
|
JobLogEvent { // ring-buffered + persisted; streamed via SSE
|
||||||
|
jobId: UUID
|
||||||
|
ts: Instant
|
||||||
|
level: enum { DEBUG, INFO, WARN, ERROR }
|
||||||
|
stage: JobStage.name?
|
||||||
|
message: String
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Ingestion Pipeline
|
||||||
|
|
||||||
|
```
|
||||||
|
┌────────────────────────────────────────────────────────┐
|
||||||
|
│ IngestionOrchestrator (virtual-thread per stage) │
|
||||||
|
└────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
┌──────────────────────────┼──────────────────────────────────────────┐
|
||||||
|
▼ ▼ ▼
|
||||||
|
[CLONE/FETCH] [DISCOVER_TAGS] [INDEX_VERSION job]
|
||||||
|
JGit pull/clone git tag list ∩ (per (repo,tag))
|
||||||
|
version mapping
|
||||||
|
rules
|
||||||
|
│
|
||||||
|
┌────────────────────────────┤
|
||||||
|
▼ ▼
|
||||||
|
[CHECKOUT worktree] (parallel tags up to N)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[DISCOVER_FILES]
|
||||||
|
respect .gitignore +
|
||||||
|
defaults + per-repo globs +
|
||||||
|
max file size
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[GIT_DIFF vs prev indexed tag]
|
||||||
|
→ if exists, only changed
|
||||||
|
files reach PARSE
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[PARSE] heuristic chunker
|
||||||
|
(markdown sections; brace-balanced;
|
||||||
|
indent-based; sliding-window fallback)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[CHUNK] AST-aware splits +
|
||||||
|
sliding-window fallback
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[HASH + DEDUPE]
|
||||||
|
content_hash lookup → existing
|
||||||
|
chunkId reused
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[EMBED] ONNX bge-m3
|
||||||
|
NEW chunks only
|
||||||
|
(GPU semaphore-gated batch)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[INDEX] Lucene upsert:
|
||||||
|
- chunk doc with vector
|
||||||
|
- chunk_version doc
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[COMMIT] Lucene commit +
|
||||||
|
H2 transaction
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Version.status = INDEXED
|
||||||
|
```
|
||||||
|
|
||||||
|
### Key invariants
|
||||||
|
|
||||||
|
1. **Embeddings are computed at most once per `content_hash`.** Persistent disk cache keyed by hash → vector bytes.
|
||||||
|
2. **A tag's chunks = union of (a) reused chunks via hash and (b) newly-embedded chunks.** This makes re-indexing a near-identical tag almost free.
|
||||||
|
3. **Git-diff fast path:** if a tag's parent (nearest previously indexed tag in semver order) exists, only files changed in `git diff parent..tag` are re-parsed. Unchanged files contribute their parent's chunk_versions verbatim with new line offsets adjusted by diff (or fully re-parsed if rename detection is ambiguous).
|
||||||
|
4. **Per-stage virtual-thread pools.** Threads themselves are unbounded (per user spec), but a **GPU semaphore** (default `permits = ortSessionCount`) gates ONNX inference to avoid GPU OOM. Lucene writer is single-thread (its own queue).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Search Pipeline
|
||||||
|
|
||||||
|
```
|
||||||
|
query ─► [Query Rewrite] rule-based: lowercase, dedupe stop tokens,
|
||||||
|
│ optional library-id-aware expansion
|
||||||
|
▼
|
||||||
|
[BM25 search] [Dense kNN search]
|
||||||
|
Lucene similarity Lucene HNSW (bge-m3 dense)
|
||||||
|
│ │
|
||||||
|
└─────────────► [RRF fusion] ◄──────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
top-K candidates (default 50)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[Cross-encoder rerank]
|
||||||
|
ONNX bge-reranker-v2-m3
|
||||||
|
(GPU semaphore)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[Token-budget assemble]
|
||||||
|
pack snippets up to `tokens` param
|
||||||
|
(default 5000, min 500, max 50000)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
ranked snippets w/ citations
|
||||||
|
(file path, repo, tag, lines)
|
||||||
|
```
|
||||||
|
|
||||||
|
All searches are **scoped** to `(repoId, versionId)` filter clauses on the Lucene index using `chunk_versions` join semantics.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. MCP Server (Streamable HTTP)
|
||||||
|
|
||||||
|
- Single endpoint: `POST /mcp` (JSON-RPC over HTTP) with optional SSE upgrade per request, per [MCP 2025-03-26 spec](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/).
|
||||||
|
- **Two tools, exactly matching Context7 schema:**
|
||||||
|
|
||||||
|
### `resolve-library-id`
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"name": "resolve-library-id",
|
||||||
|
"description": "Resolves a library/package name to a trueref-compatible library ID...",
|
||||||
|
"inputSchema": {
|
||||||
|
"type": "object",
|
||||||
|
"required": ["libraryName"],
|
||||||
|
"properties": {
|
||||||
|
"libraryName": { "type": "string" },
|
||||||
|
"query": { "type": "string", "description": "optional, ranks results by relevance" }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Returns ranked candidate library IDs (`/{owner}/{repo}` style) with metadata (description, snippet count, available versions, source reputation).
|
||||||
|
|
||||||
|
### `get-library-docs`
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"name": "get-library-docs",
|
||||||
|
"inputSchema": {
|
||||||
|
"type": "object",
|
||||||
|
"required": ["libraryId"],
|
||||||
|
"properties": {
|
||||||
|
"libraryId": { "type": "string", "description": "/org/project[/version]" },
|
||||||
|
"topic": { "type": "string" },
|
||||||
|
"tokens": { "type": "integer", "minimum": 500, "maximum": 50000, "default": 5000 }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### On-demand indexing flow
|
||||||
|
- If `libraryId` includes a version that maps to a known git tag but is **not yet indexed**:
|
||||||
|
1. Enqueue `INDEX_VERSION` job immediately.
|
||||||
|
2. Return a **partial** response built from the **nearest indexed tag** (semver-closest) plus a status block: `{ "indexing": { "status": "in_progress", "version": "1.2.3", "retryAfterSec": 30 } }`.
|
||||||
|
- If version maps to **no** tag: return error `version_not_found` with the list of candidate tags discovered.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. REST API Surface
|
||||||
|
|
||||||
|
| Method | Path | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| GET | `/api/repos` | List registered repos |
|
||||||
|
| POST | `/api/repos` | Register (local path or remote URL) |
|
||||||
|
| GET | `/api/repos/{id}` | Repo detail + version summary |
|
||||||
|
| DELETE | `/api/repos/{id}` | Unregister + soft-delete versions |
|
||||||
|
| POST | `/api/repos/{id}/discover` | Force tag discovery |
|
||||||
|
| GET | `/api/repos/{id}/versions` | All known versions + status |
|
||||||
|
| POST | `/api/repos/{id}/versions/{tag}/index` | Index a specific tag |
|
||||||
|
| POST | `/api/repos/{id}/versions/{tag}/reindex` | Force re-index |
|
||||||
|
| GET | `/api/jobs` | List jobs (filter by repo/version/status) |
|
||||||
|
| GET | `/api/jobs/{id}` | Job detail with stages |
|
||||||
|
| GET | `/api/jobs/{id}/log` (SSE) | Live log stream |
|
||||||
|
| GET | `/api/jobs/stream` (SSE) | Live job-status events for the dashboard |
|
||||||
|
| POST | `/api/search` | Hybrid search across one or more (repo, version) scopes |
|
||||||
|
| GET | `/api/resolve?q=react` | Library-ID resolution preview |
|
||||||
|
| GET | `/api/observability/metrics` | UI-friendly aggregated metrics JSON |
|
||||||
|
| GET | `/api/observability/resources` | Heap, GPU mem (via NVML when present), index size |
|
||||||
|
| GET | `/swagger-ui/index.html` | Swagger UI |
|
||||||
|
| GET | `/v3/api-docs` | OpenAPI JSON |
|
||||||
|
| ANY | `/mcp` | MCP Streamable HTTP endpoint |
|
||||||
|
| GET | `/actuator/prometheus` | Prometheus scrape (optional) |
|
||||||
|
| GET | `/**` | SPA fallback to `index.html` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Concurrency & Performance
|
||||||
|
|
||||||
|
- **Virtual threads everywhere** for I/O (HTTP, JGit, file I/O, Lucene reads).
|
||||||
|
- **`Tomcat` configured with virtual-thread executor** (`spring.threads.virtual.enabled=true`).
|
||||||
|
- **Per-stage logical pools** are unbounded virtual-thread executors per orchestrator instance.
|
||||||
|
- **GPU access gated by a `Semaphore`** with permits = number of ONNX sessions (configurable, default = 2).
|
||||||
|
- **Lucene writer**: single `IndexWriter` instance protected by a queue; readers use a refresh-on-search `SearcherManager`.
|
||||||
|
- **Embedding cache**: file-per-hash on disk under `data/embedding-cache/`; hot LRU in memory.
|
||||||
|
- **Tag concurrency**: not capped (per spec), but each tag job awaits the GPU semaphore — natural backpressure.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Observability
|
||||||
|
|
||||||
|
- **Metrics** via Micrometer (`MeterRegistry`):
|
||||||
|
- Counters: chunks_embedded, chunks_reused, files_skipped, jobs_succeeded/failed.
|
||||||
|
- Timers: stage durations per stage name.
|
||||||
|
- Gauges: active_jobs, gpu_semaphore_available, lucene_index_size_bytes, heap_used.
|
||||||
|
- **OpenTelemetry traces** for every job (one trace per `IngestionJob`, span per `JobStage`).
|
||||||
|
- **JobEventBus**: in-process pub/sub. SSE controllers subscribe and push events to UI.
|
||||||
|
- **UI dashboards** (no Grafana required):
|
||||||
|
- "Live" tab: progress bars per running (repo, tag), per-stage throughput, log tail.
|
||||||
|
- "History" tab: paginated jobs table.
|
||||||
|
- "Stats" tab: per-stage timing histograms, chunk counts per repo/version, chunk dedupe ratio.
|
||||||
|
- "Resources" tab: heap, GPU memory (NVML where available), index size on disk.
|
||||||
|
- **Prometheus** scraping is opt-in (Actuator endpoint).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Storage Layout (on disk)
|
||||||
|
|
||||||
|
```
|
||||||
|
$TRUEREF_HOME/ # default: ./data
|
||||||
|
├── h2/ # H2 database files
|
||||||
|
├── lucene/ # single index dir; one Lucene writer
|
||||||
|
├── repos/ # managed clones (when managedClone=true)
|
||||||
|
│ └── <repoId>/...
|
||||||
|
├── embedding-cache/ # one file per content_hash → fp16 vector bytes
|
||||||
|
├── models/ # ONNX model files (auto-downloaded on first run)
|
||||||
|
│ ├── bge-m3/
|
||||||
|
│ └── bge-reranker-v2-m3/
|
||||||
|
└── logs/
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. Configuration (excerpt)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
trueref:
|
||||||
|
home: ${TRUEREF_HOME:./data}
|
||||||
|
ingestion:
|
||||||
|
poll-interval-default: 1h
|
||||||
|
tag-cap-default: 100 # most-recent N tags by semver/date
|
||||||
|
max-file-size-bytes-default: 1048576
|
||||||
|
embedding:
|
||||||
|
model: bge-m3
|
||||||
|
onnx-providers: [cuda, directml, cpu] # tried in order
|
||||||
|
session-count: 2 # = GPU semaphore permits
|
||||||
|
batch-size: 32
|
||||||
|
reranker:
|
||||||
|
model: bge-reranker-v2-m3
|
||||||
|
top-k: 50
|
||||||
|
search:
|
||||||
|
rrf-k: 60
|
||||||
|
final-top-k: 20
|
||||||
|
mcp:
|
||||||
|
tokens-default: 5000
|
||||||
|
tokens-min: 500
|
||||||
|
tokens-max: 50000
|
||||||
|
spring:
|
||||||
|
threads.virtual.enabled: true
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. Out-of-the-box behaviors locked from clarifications
|
||||||
|
|
||||||
|
- **Auth**: none (LAN-only) on REST and MCP.
|
||||||
|
- **Tag selection**: default cap 100 most-recent; on-demand index of any tag via UI search OR via MCP when an unindexed version is requested.
|
||||||
|
- **Differential indexing**: dedupe by `content_hash` AND skip unchanged files via `git diff parent..tag`.
|
||||||
|
- **Repo input**: UI-add (local path or remote URL) AND watched folder `./data/watched/` for bare repos.
|
||||||
|
- **Re-index trigger**: on-demand + scheduled `git fetch` poll (default 1h per repo).
|
||||||
|
- **Stale tag cleanup**: soft delete via `Version.status=INACTIVE`; compaction job reclaims orphan chunks.
|
||||||
|
- **Embedding cache**: persistent on disk, keyed by `content_hash`.
|
||||||
|
- **Concurrency**: unbounded virtual threads, GPU semaphore-gated.
|
||||||
|
|
||||||
|
See [FINDINGS.md](FINDINGS.md) for research backing each choice.
|
||||||
91
CODE_STYLE.md
Normal file
91
CODE_STYLE.md
Normal file
@@ -0,0 +1,91 @@
|
|||||||
|
# trueref — Code Style
|
||||||
|
|
||||||
|
## 1. Language & Toolchain
|
||||||
|
- **Java 21**, source/target 21.
|
||||||
|
- **Maven** with `spring-boot-maven-plugin` for the fat JAR.
|
||||||
|
- **Spotless** with **Palantir Java Format** for formatting (4-space indent, 120 col).
|
||||||
|
- **ErrorProne** + **NullAway** for static analysis. NullAway annotations: `@org.jspecify.annotations.Nullable` / `@NonNull`.
|
||||||
|
- Hexagonal boundaries are enforced by **Maven module dependencies** (no ArchUnit). See ARCHITECTURE §3.
|
||||||
|
|
||||||
|
## 2. Records, Sealed Types, Pattern Matching
|
||||||
|
- Prefer **records** for DTOs, value objects, and `port.in`/`port.out` parameter/result types.
|
||||||
|
- Use **sealed interfaces** for closed result/event hierarchies (`sealed interface IngestionEvent permits ...`).
|
||||||
|
- Use **pattern matching** (`switch` expressions, `instanceof`) over visitor pattern.
|
||||||
|
|
||||||
|
## 3. Nullability & Optional
|
||||||
|
- All API surfaces (public methods on ports, REST DTOs) are **non-null by default**; mark nullable explicitly with `@Nullable`.
|
||||||
|
- Use `Optional<T>` **only** as a return type from query-style methods. Never as a field, never as a parameter.
|
||||||
|
|
||||||
|
## 4. Concurrency
|
||||||
|
- Spawn virtual threads via `Thread.ofVirtual().start(...)` or `Executors.newVirtualThreadPerTaskExecutor()`. **Never** call `Thread.sleep` inside a synchronized block.
|
||||||
|
- Shared mutable state is forbidden in `domain` and discouraged in `application`. When unavoidable, use `java.util.concurrent.atomic.*` or a `ReentrantLock`.
|
||||||
|
- GPU work goes through `GpuSemaphore.acquire()` (a thin wrapper around `Semaphore`).
|
||||||
|
- Long-running orchestration uses **structured concurrency** (`StructuredTaskScope`) where it improves cancellation safety.
|
||||||
|
|
||||||
|
## 5. Error Handling
|
||||||
|
- **Domain errors** are sealed exception hierarchies rooted at `TrueRefException`. Adapters translate them to HTTP/JSON-RPC errors centrally (REST: `@ControllerAdvice`; MCP: dedicated translator).
|
||||||
|
- **No checked exceptions** at port boundaries. Wrap third-party checked exceptions at the adapter edge.
|
||||||
|
- Validation errors carry a stable `code` (string) so the UI can localize.
|
||||||
|
- Never `catch (Exception e)` and swallow. Either log + rethrow as a domain exception or let it propagate.
|
||||||
|
|
||||||
|
## 6. Logging
|
||||||
|
- **SLF4J** with parameterized messages: `log.info("indexed tag {} of repo {}", tag, repoName);` — never string-concatenate.
|
||||||
|
- Structured fields via MDC: `repoId`, `versionId`, `jobId`, `stage`. Cleared in a try/finally.
|
||||||
|
- Log levels:
|
||||||
|
- `ERROR`: unrecoverable, requires operator attention.
|
||||||
|
- `WARN`: degraded, automatic recovery in progress.
|
||||||
|
- `INFO`: lifecycle events (job started/finished, repo registered).
|
||||||
|
- `DEBUG`: per-file, per-chunk detail. Off by default.
|
||||||
|
|
||||||
|
## 7. Naming
|
||||||
|
- Use cases (`port.in`): imperative verb phrases — `IndexVersion`, `ResolveLibraryId`.
|
||||||
|
- SPIs (`port.out`): noun-ish role names — `EmbeddingService`, `ChunkStore`, `GitClient`.
|
||||||
|
- Adapter classes: `<Tech><Role>` — `LuceneChunkStore`, `OnnxEmbeddingService`, `JGitClient`.
|
||||||
|
- DTOs: `<Resource><Action>Request` / `<Resource>Response`.
|
||||||
|
- Records' field names are camelCase (no Hungarian, no `_` prefixes).
|
||||||
|
|
||||||
|
## 8. Package & File Discipline
|
||||||
|
- One public type per file.
|
||||||
|
- Internal helpers are package-private. Avoid `public` unless used across packages.
|
||||||
|
- Domain packages export **only** records and interfaces. No Spring annotations, no Lombok, no Jackson annotations.
|
||||||
|
- Adapter packages may use Spring stereotypes (`@Component`, `@Repository`, `@RestController`) but adapters depend on **port interfaces only** when interacting with the application.
|
||||||
|
|
||||||
|
## 9. Spring Wiring
|
||||||
|
- Wiring lives in `bootstrap`. Each adapter package may define a `@Configuration` (constructor-injected `@Bean` factories) but **does not** auto-`@ComponentScan` itself; bootstrap explicitly imports.
|
||||||
|
- Use `@ConfigurationProperties` records for typed config; never raw `@Value`.
|
||||||
|
- Prefer constructor injection. **No field injection.**
|
||||||
|
|
||||||
|
## 10. Persistence (H2)
|
||||||
|
- Migrations under `src/main/resources/db/migration` named `V<N>__<snake_case>.sql`. Flyway runs at startup.
|
||||||
|
- All access via Spring **`JdbcClient`** (Spring Boot 3.2+, fluent JDBC). **No JPA/Hibernate**, no `JdbcTemplate` directly.
|
||||||
|
- Mappers are explicit `RowMapper<T>` lambdas, not reflection-based.
|
||||||
|
- SQL lives next to the repository class, either as `static final String` constants or in `*.sql` files loaded via `ClassPathResource` for non-trivial queries.
|
||||||
|
|
||||||
|
## 11. REST
|
||||||
|
- Controllers in `adapter.in.rest`. They depend only on `port.in` interfaces and DTO records.
|
||||||
|
- DTOs are **separate** from domain records. Mapping via plain `static of(...)` factories. **No MapStruct.**
|
||||||
|
- All endpoints documented with `@Operation`, `@ApiResponses`, `@Schema` (springdoc).
|
||||||
|
- Request validation via `jakarta.validation` annotations on DTOs.
|
||||||
|
- SSE endpoints return `SseEmitter`; subscribe to `JobEventBus`, unsubscribe on completion/timeout.
|
||||||
|
|
||||||
|
## 12. MCP
|
||||||
|
- Tool definitions are **records** decorated to produce JSON Schema via Spring AI's MCP support. Schema strings stay verbatim (1:1 with Context7) so LLMs see identical contracts.
|
||||||
|
- Tool handlers depend only on `port.in` (`SearchLibraryDocs`, `ResolveLibraryId`).
|
||||||
|
|
||||||
|
## 13. Tests
|
||||||
|
- **JUnit 5** + **AssertJ** + **Mockito** (sparingly).
|
||||||
|
- Unit tests live next to the package they test. Integration tests under `src/test/java/.../it/` and use `@SpringBootTest`.
|
||||||
|
- Use **Testcontainers** only when truly required (we mostly avoid it via embedded stores).
|
||||||
|
- ArchUnit test suite is mandatory and runs in CI.
|
||||||
|
|
||||||
|
## 14. Dependency Hygiene
|
||||||
|
- BOM-managed versions only. Add a dependency only if it provides clear value over JDK + Spring Boot + already-included libs.
|
||||||
|
- No Lombok, no Guava (use JDK 21 equivalents), no Reactor (we use virtual threads + blocking).
|
||||||
|
- No Kotlin, no Scala.
|
||||||
|
|
||||||
|
## 15. Documentation
|
||||||
|
- All architectural decisions go in **ARCHITECTURE.md**.
|
||||||
|
- All research notes go in **FINDINGS.md** with sources.
|
||||||
|
- All conventions go in **this file**.
|
||||||
|
- Per-package `package-info.java` may exist for non-trivial packages, summarizing role and exported types.
|
||||||
|
- No README sprawl: `README.md` is a quickstart only and links to the three docs above.
|
||||||
51
Dockerfile
Normal file
51
Dockerfile
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
# ─── Build stage ──────────────────────────────────────────────────────────────
|
||||||
|
# eclipse-temurin:21-jdk-jammy ships JDK 21 + Maven-compatible toolchain.
|
||||||
|
# frontend-maven-plugin downloads Node/npm automatically, so no explicit
|
||||||
|
# Node install is needed in the build stage.
|
||||||
|
FROM eclipse-temurin:21-jdk-jammy AS builder
|
||||||
|
|
||||||
|
RUN apt-get update \
|
||||||
|
&& apt-get install -y --no-install-recommends maven \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
WORKDIR /build
|
||||||
|
COPY . .
|
||||||
|
|
||||||
|
RUN mvn -q package -DskipTests -T 1C
|
||||||
|
|
||||||
|
# ─── Runtime stage (CPU-only) ─────────────────────────────────────────────────
|
||||||
|
FROM eclipse-temurin:21-jre-jammy
|
||||||
|
|
||||||
|
LABEL org.opencontainers.image.title="TrueRef"
|
||||||
|
LABEL org.opencontainers.image.description="Self-hosted documentation retrieval platform for AI coding assistants (CPU variant)"
|
||||||
|
LABEL org.opencontainers.image.url="https://git.sal.giize.com/mozempk/trueref"
|
||||||
|
LABEL org.opencontainers.image.source="https://git.sal.giize.com/mozempk/trueref"
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
COPY --from=builder /build/trueref-bootstrap/target/trueref.jar /app/trueref.jar
|
||||||
|
|
||||||
|
# /data is the default trueref.home: H2 DB, Lucene index, embedding cache and
|
||||||
|
# downloaded models all live here. Mount a volume to persist between restarts.
|
||||||
|
VOLUME /data
|
||||||
|
|
||||||
|
ENV TRUEREF_HOME=/data \
|
||||||
|
TRUEREF_PORT=18080 \
|
||||||
|
JAVA_OPTS=""
|
||||||
|
|
||||||
|
EXPOSE 18080
|
||||||
|
|
||||||
|
# JVM flags required by trueref:
|
||||||
|
# --enable-native-access silences FFM Linker warning from DJL tokenizers
|
||||||
|
# --add-modules enables Lucene 10 SIMD codepath (incubator.vector)
|
||||||
|
# Spring properties are passed via CMD so users can override them at runtime.
|
||||||
|
ENTRYPOINT ["sh", "-c", \
|
||||||
|
"exec java \
|
||||||
|
--enable-native-access=ALL-UNNAMED \
|
||||||
|
--add-modules=jdk.incubator.vector \
|
||||||
|
${JAVA_OPTS} \
|
||||||
|
-jar /app/trueref.jar \
|
||||||
|
--server.port=${TRUEREF_PORT} \
|
||||||
|
--trueref.home=${TRUEREF_HOME} \
|
||||||
|
--trueref.embedding.onnx-providers=cpu \
|
||||||
|
\"$@\"", "--"]
|
||||||
69
Dockerfile.gpu
Normal file
69
Dockerfile.gpu
Normal file
@@ -0,0 +1,69 @@
|
|||||||
|
# ─── Build stage ──────────────────────────────────────────────────────────────
|
||||||
|
FROM eclipse-temurin:21-jdk-jammy AS builder
|
||||||
|
|
||||||
|
RUN apt-get update \
|
||||||
|
&& apt-get install -y --no-install-recommends maven \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
WORKDIR /build
|
||||||
|
COPY . .
|
||||||
|
|
||||||
|
RUN mvn -q package -DskipTests -T 1C
|
||||||
|
|
||||||
|
# ─── Runtime stage (NVIDIA GPU / CUDA 12 + cuDNN 9) ──────────────────────────
|
||||||
|
# nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04 ships:
|
||||||
|
# - CUDA 12.4 runtime libs (libcuda.so, libcublas, etc.)
|
||||||
|
# - cuDNN 9 (cu12 build) required by ONNX Runtime CUDA execution provider
|
||||||
|
#
|
||||||
|
# Prerequisites on the Docker host:
|
||||||
|
# - NVIDIA GPU driver ≥ 550 (CUDA 12.4 compatible)
|
||||||
|
# - nvidia-container-toolkit installed and configured
|
||||||
|
#
|
||||||
|
# Run with: docker run --gpus all --device /dev/nvidia0 ...
|
||||||
|
FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04
|
||||||
|
|
||||||
|
LABEL org.opencontainers.image.title="TrueRef (GPU)"
|
||||||
|
LABEL org.opencontainers.image.description="Self-hosted documentation retrieval platform for AI coding assistants (NVIDIA GPU / CUDA 12 variant)"
|
||||||
|
LABEL org.opencontainers.image.url="https://git.sal.giize.com/mozempk/trueref"
|
||||||
|
LABEL org.opencontainers.image.source="https://git.sal.giize.com/mozempk/trueref"
|
||||||
|
|
||||||
|
# Install Eclipse Temurin 21 JRE onto the CUDA base image.
|
||||||
|
RUN apt-get update \
|
||||||
|
&& apt-get install -y --no-install-recommends wget apt-transport-https gnupg \
|
||||||
|
&& wget -q -O - https://packages.adoptium.net/artifactory/api/gpg/key/public \
|
||||||
|
| gpg --dearmor -o /usr/share/keyrings/adoptium.gpg \
|
||||||
|
&& echo "deb [signed-by=/usr/share/keyrings/adoptium.gpg] https://packages.adoptium.net/artifactory/deb jammy main" \
|
||||||
|
> /etc/apt/sources.list.d/adoptium.list \
|
||||||
|
&& apt-get update \
|
||||||
|
&& apt-get install -y --no-install-recommends temurin-21-jre \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
COPY --from=builder /build/trueref-bootstrap/target/trueref.jar /app/trueref.jar
|
||||||
|
|
||||||
|
VOLUME /data
|
||||||
|
|
||||||
|
ENV TRUEREF_HOME=/data \
|
||||||
|
TRUEREF_PORT=18080 \
|
||||||
|
# Physical GPU index visible inside the container (0 after --gpus all remapping).
|
||||||
|
TRUEREF_GPU=0 \
|
||||||
|
# 0 = unbounded arena; set to e.g. 8589934592 (8 GiB) on shared hosts.
|
||||||
|
TRUEREF_MEM_LIMIT=0 \
|
||||||
|
JAVA_OPTS="" \
|
||||||
|
# CUDA_DEVICE_ORDER ensures nvidia-smi numbering matches CUDA runtime numbering.
|
||||||
|
CUDA_DEVICE_ORDER=PCI_BUS_ID
|
||||||
|
|
||||||
|
EXPOSE 18080
|
||||||
|
|
||||||
|
ENTRYPOINT ["sh", "-c", \
|
||||||
|
"exec java \
|
||||||
|
--enable-native-access=ALL-UNNAMED \
|
||||||
|
--add-modules=jdk.incubator.vector \
|
||||||
|
${JAVA_OPTS} \
|
||||||
|
-jar /app/trueref.jar \
|
||||||
|
--server.port=${TRUEREF_PORT} \
|
||||||
|
--trueref.home=${TRUEREF_HOME} \
|
||||||
|
--trueref.embedding.gpu-device-id=${TRUEREF_GPU} \
|
||||||
|
--trueref.embedding.gpu-mem-limit-bytes=${TRUEREF_MEM_LIMIT} \
|
||||||
|
\"$@\"", "--"]
|
||||||
207
FINDINGS.md
Normal file
207
FINDINGS.md
Normal file
@@ -0,0 +1,207 @@
|
|||||||
|
# trueref — Findings
|
||||||
|
|
||||||
|
Research notes backing the choices in [ARCHITECTURE.md](ARCHITECTURE.md). Each section ends with a verdict and follow-up questions if any.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F1. Context7 ingestion behavior (what we replicate functionally)
|
||||||
|
|
||||||
|
- Context7 ingests git repositories and crawls associated docs sites driven by a `context7.json` manifest at the repo root, plus an optional `llms.txt` index.
|
||||||
|
- It produces snippets shaped roughly as `{ title, description, source, code, language }` and serves them via two MCP tools: `resolve-library-id` and `get-library-docs`.
|
||||||
|
- The `get-library-docs` API accepts `topic` and `tokens` parameters; topic biases retrieval, tokens caps the response size (defaults observed in client docs: ~5000).
|
||||||
|
- Source: upstash/context7 GitHub repo & MCP docs.
|
||||||
|
|
||||||
|
**Verdict:** functional parity is achievable without copying the manifest schema. Our chunk model captures the same fields under different names (`symbol`/`content`/`filePath`/`language`). MCP tool signatures are kept **byte-identical** for LLM compatibility.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F2. Embedded vector store choice — Lucene 9 over Qdrant
|
||||||
|
|
||||||
|
- Qdrant is a Rust binary; embedding it in a fat JAR requires extracting & spawning a child process, contradicting the "single JAR, embedded everything" goal.
|
||||||
|
- **Apache Lucene ≥9.0** ships HNSW kNN (`KnnFloatVectorField`) alongside BM25 in a single index segment. Pure JVM, no native deps.
|
||||||
|
- Lucene supports **filtered kNN** (`KnnFloatVectorQuery` with a `BooleanQuery` filter), which we need for `(repoId, versionId)` scoping.
|
||||||
|
- Trade-off: Lucene HNSW lacks Qdrant's payload-rich filtering tricks (e.g. quantization presets, named vectors). Acceptable for our scale; we get BM25 in the same store for free.
|
||||||
|
|
||||||
|
**Verdict:** Lucene 9 (we'll target the latest 9.x). One `IndexWriter`, refresh-on-search via `SearcherManager`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F3. Embedding model — bge-m3
|
||||||
|
|
||||||
|
- BAAI/bge-m3: 568M params, 8192 ctx, multilingual (100+ langs), trained on multi-functionality (dense + sparse + colbert).
|
||||||
|
- ONNX export available (BAAI provides it; community variants on HuggingFace).
|
||||||
|
- License: MIT-style (model weights), works for self-hosted commercial use.
|
||||||
|
- Vector dim: 1024 (dense). Sparse vocab compatible with Lucene if we want SPLADE-like sparse — out of scope for v1.
|
||||||
|
|
||||||
|
**Verdict:** bge-m3 (dense only for v1). Sparse channel deferred.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F4. Reranker — bge-reranker-v2-m3
|
||||||
|
|
||||||
|
- Cross-encoder, scores (query, passage) pairs.
|
||||||
|
- Same family as embedder: balanced quality/cost, ONNX-exportable.
|
||||||
|
- Apache 2.0 license.
|
||||||
|
|
||||||
|
**Verdict:** bge-reranker-v2-m3. Top-K candidates from RRF fed in, top-N (default 20) returned.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F5. ML runtime — ONNX Runtime (Java bindings)
|
||||||
|
|
||||||
|
- ONNX Runtime has **official Java bindings** (`com.microsoft.onnxruntime:onnxruntime` + `onnxruntime_gpu`).
|
||||||
|
- Execution providers we will support:
|
||||||
|
- **CUDA** (`onnxruntime_gpu`): Linux + Windows with NVIDIA driver ≥ matching CUDA 12.x.
|
||||||
|
- **DirectML** (`onnxruntime-directml`): Windows, any DX12 GPU.
|
||||||
|
- **CPU**: always-on fallback.
|
||||||
|
- ONNX Runtime has **no Vulkan execution provider**. Our earlier "Vulkan fallback" wish is not satisfiable in this stack — we drop it.
|
||||||
|
- Generative LLMs in ONNX (e.g. Phi-3.5-mini) are possible but awkward (KV cache management, tokenizer differences). Since we picked **retrieval-only**, no generative model is needed.
|
||||||
|
|
||||||
|
**Verdict:** ONNX Runtime, providers tried in order: cuda → directml → cpu. Vulkan dropped (documented).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F6. Java version — 21 LTS, not 25
|
||||||
|
|
||||||
|
- Spring Boot 3.5.x officially supports Java 17–23.
|
||||||
|
- Spring AI 1.0.x targets the same range.
|
||||||
|
- Java 25 is supported by neither at time of writing; risking obscure reflection/MR-JAR issues with downstream libs (JGit, Lucene, ONNX bindings).
|
||||||
|
- Java 21 is LTS and has stable virtual threads + structured concurrency (`StructuredTaskScope` was preview through 23, finalizing soon — we'll guard usage behind a thin wrapper to ease later upgrade).
|
||||||
|
|
||||||
|
**Verdict:** Java 21 LTS. Re-evaluate to 25 once Spring Boot certifies it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F7. Differential indexing scheme
|
||||||
|
|
||||||
|
- We chose **dedupe-by-content-hash** AND **git-diff-driven file skipping**.
|
||||||
|
- The hash dedupe alone gives constant-cost embeddings for unchanged code across tags.
|
||||||
|
- The git-diff path additionally avoids parsing/chunking unchanged files, which dominates ingest CPU on large repos.
|
||||||
|
- Storage model:
|
||||||
|
- `chunks`: one row per unique `content_hash`. Vector lives in Lucene keyed by `chunkId`.
|
||||||
|
- `chunk_versions`: many-to-many; one row per `(chunk, version, file, line range)`.
|
||||||
|
- Search: `BooleanQuery(filter=chunk_versions.version_id IN scope)` joined to vector field.
|
||||||
|
- The chunk dedupe ratio is reported as a UI metric — it's the most intuitive measure of "differential" effectiveness.
|
||||||
|
|
||||||
|
**Verdict:** confirmed; both mechanisms compose without conflict.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F8. MCP transport — Streamable HTTP
|
||||||
|
|
||||||
|
- The current MCP spec (revision 2025-03-26) defines **Streamable HTTP**: a single `POST /mcp` endpoint that may upgrade to SSE for long-lived/streamed responses; replaces the deprecated 2024-11-05 SSE transport.
|
||||||
|
- Spring AI 1.0 ships an MCP server module that supports Streamable HTTP via Spring MVC.
|
||||||
|
- We expose **only** Streamable HTTP, no SSE-only legacy endpoint (per user spec).
|
||||||
|
|
||||||
|
**Verdict:** Streamable HTTP only at `/mcp`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F9. Embedded SQL store — H2 (MVCC)
|
||||||
|
|
||||||
|
- H2 in MVCC mode supports concurrent readers and a single writer with row-level locking. Good enough for our metadata write rates (jobs, versions, chunk_versions).
|
||||||
|
- File-based, single JAR dependency, JDBC.
|
||||||
|
- Considered & rejected:
|
||||||
|
- **DuckDB**: column-store, slower OLTP, no good Flyway story.
|
||||||
|
- **SQLite**: poor concurrency under write load.
|
||||||
|
- **Embedded Postgres (zonky)**: pulls a 100+ MB native binary per OS — fights the fat JAR goal.
|
||||||
|
|
||||||
|
**Verdict:** H2 file-based, MVCC=true, with Flyway migrations.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F10. Job orchestration — custom virtual-thread orchestrator
|
||||||
|
|
||||||
|
- Spring Batch is feature-rich but requires a JobRepository (typically Postgres or H2) and adds startup cost we don't need.
|
||||||
|
- Our jobs are **per-tag**, **simple linear stage sequences**, with persistence-of-status as the only durability requirement.
|
||||||
|
- Custom orchestrator: each `IngestionJob` runs on a virtual thread; stages execute sequentially; stage transitions are durably written to H2 in a transaction; `JobEventBus` emits events for SSE.
|
||||||
|
- Crash recovery: on startup, scan jobs in `RUNNING` status, mark them `FAILED` (or resume specific resumable stages — v2).
|
||||||
|
|
||||||
|
**Verdict:** custom orchestrator. Spring Batch deferred unless we hit a ceiling.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F11. Code parser — pure-Java heuristic for v1, tree-sitter pluggable for v2
|
||||||
|
|
||||||
|
The Java tree-sitter ecosystem in 2026 is fragmented:
|
||||||
|
|
||||||
|
- **`io.github.tree-sitter:jtreesitter`** uses Project Panama FFI → requires **Java 22+**. We target Java 21 LTS, so this is out.
|
||||||
|
- **`io.github.bonede:tree-sitter`** is JNI-based and works on Java 21, but bundling per-OS (linux/windows/mac × x64/arm64) native grammar binaries for many languages bloats the fat JAR significantly and creates a packaging matrix we don't want to maintain in v1.
|
||||||
|
- **`ai.serenade.treesitter:java-tree-sitter`** is unmaintained.
|
||||||
|
|
||||||
|
**Decision (v1):** ship a pure-Java heuristic `CodeParser` adapter. Strategies, tried in order per file:
|
||||||
|
|
||||||
|
1. **Markdown / `.txt` / `.rst`**: split by ATX/Setext headings; large sections further split by paragraph.
|
||||||
|
2. **Brace-balanced languages** (java, c, c++, c#, go, rust, js, ts, kotlin, scala, swift): walk the file tracking brace depth + line-based heuristics (function signatures, top-level declarations) to extract chunks of complete top-level constructs. Symbol name extracted via a tiny regex per language.
|
||||||
|
3. **Indent-based languages** (python, yaml, ruby): split on top-level `def`/`class`/`module` boundaries; symbol name from the declaration line.
|
||||||
|
4. **Fallback** (any text file): sliding-window of N lines (default 80) with M lines overlap (default 10).
|
||||||
|
|
||||||
|
The `CodeParser` port is unchanged. A future tree-sitter implementation (when JDK upgrade or upstream packaging matures) can be swapped in by providing an alternate `@Component` and toggling a config flag — that's exactly what hexagonal architecture buys us.
|
||||||
|
|
||||||
|
**Verdict:** pure-Java heuristic parser for v1; tree-sitter remains a documented future enhancement.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F12. Concurrency caps & GPU contention
|
||||||
|
|
||||||
|
- User chose **unbounded virtual threads**. This is safe for I/O-bound stages.
|
||||||
|
- ONNX inference is GPU-bound; calling the same `OrtSession` from many threads concurrently is unsupported. Two mitigations:
|
||||||
|
1. A **session pool** of size N (config `embedding.session-count`, default 2).
|
||||||
|
2. A **`Semaphore(N)`** acquired by any caller before invoking inference. Pool & semaphore sizes match.
|
||||||
|
- This means tag-level parallelism is naturally throttled by GPU capacity without explicit per-tag limits.
|
||||||
|
|
||||||
|
**Verdict:** session pool + semaphore. Document the knob clearly in `application.yml`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F13. Frontend in fat JAR
|
||||||
|
|
||||||
|
- SvelteKit `@sveltejs/adapter-static` produces a fully static bundle (HTML/CSS/JS). We build it as a Maven sub-step (frontend-maven-plugin) and copy `frontend/build/` to `bootstrap/src/main/resources/static/`. Spring serves it by default.
|
||||||
|
- SPA fallback: a `WebMvcConfigurer` maps all unmatched non-API paths to `index.html` so client-side routing works.
|
||||||
|
|
||||||
|
**Verdict:** static adapter + Spring static-resource serving. Single artifact preserved.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F14. Open questions / future work
|
||||||
|
|
||||||
|
1. **Sparse channel** (bge-m3 sparse / SPLADE) for stronger lexical recall — deferred to v2.
|
||||||
|
2. **Per-language reranker fine-tuning** — out of scope (no fine-tuning, per spec).
|
||||||
|
3. **Compaction job** to truly delete orphan chunks (currently soft-delete on versions). Schedule TBD.
|
||||||
|
4. **Watched-folder** auto-discovery semantics: how often do we rescan `./data/watched/`? Default proposal: every 5 min + on filesystem watch event (Java NIO `WatchService`).
|
||||||
|
5. **Repo size cap**: do we need a maximum total cloned size to prevent runaway disk use? Currently unlimited; could add per-repo and global caps in v2.
|
||||||
|
6. **GPU memory introspection**: Linux NVML via JNI (`jnvml`) for GPU mem gauges; on Windows + DirectML we surface only "available/in-use" booleans.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F15. References (for re-checking when libraries bump)
|
||||||
|
|
||||||
|
- Context7 repo & MCP tool surface — to sanity-check schema fidelity on releases.
|
||||||
|
- Spring AI 1.0.x release notes — verify MCP server Streamable HTTP module name & API.
|
||||||
|
- Spring Boot 3.5.x release notes — confirm Java version compatibility window.
|
||||||
|
- Lucene 9.x kNN docs — confirm filtered vector query API surface.
|
||||||
|
- ONNX Runtime Java release notes — confirm CUDA/DirectML EP availability per version.
|
||||||
|
- BAAI/bge-m3 model card — confirm ONNX export availability/format.
|
||||||
|
- MCP spec 2025-03-26 — Streamable HTTP transport requirements.
|
||||||
|
|
||||||
|
> Use the Context7 MCP lookup skill before bumping any of the above to fetch fresh, version-specific docs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F16. Smoke-test log (2026-04-21)
|
||||||
|
|
||||||
|
End-to-end smoke after first assembly:
|
||||||
|
- `mvn -pl trueref-bootstrap -am package` → BUILD SUCCESS, fat JAR ~582 MB.
|
||||||
|
- `mvn test` → **16 tests pass** (parser 6, pooling 5, disk cache 5), **0 failures**.
|
||||||
|
- `java -jar trueref-bootstrap/target/trueref.jar --trueref.embedding.session-count=0` — started in 3.6 s.
|
||||||
|
- `GET /actuator/health` → `UP` (db H2, disk, ping, ssl).
|
||||||
|
- `POST /api/repos` + `GET /api/repos` — round-trips a repo.
|
||||||
|
- `GET /swagger-ui.html` → 302 redirect (to `/swagger-ui/index.html`), `GET /v3/api-docs` → 200.
|
||||||
|
- `GET /` → 200 (SvelteKit SPA served from Spring static resources).
|
||||||
|
- `POST /mcp` one-shot JSON-RPC returns HTTP 500 — expected, the WebMVC MCP transport requires an SSE session established by `GET /sse` first; MCP clients that implement the Streamable-HTTP spec do this automatically. Verified MCP tools register: `tools/list` handler is reached (error thrown is transport-level session lookup, not bean wiring).
|
||||||
|
|
||||||
|
Fixes landed during smoke:
|
||||||
|
- `V1__init_schema.sql`: H2 in PostgreSQL mode rejects `AUTO_INCREMENT`. Switched `job_log_events.id` to `BIGINT GENERATED BY DEFAULT AS IDENTITY` and removed the explicit `NULL` constraint.
|
||||||
|
- `OnnxProperties.sessionCount` can now be 0 (disables the ONNX stack, for environments where models aren't available); `GpuSemaphore` accepts 0 permits by internally using 1 (never acquired in disabled mode).
|
||||||
|
- `OnnxEmbeddingService` / `OnnxRerankerService` short-circuit in disabled mode; reranker pass-through preserves input order.
|
||||||
|
- `ApplicationBeans` exposes only concrete beans (not both the class and its interface) to avoid ambiguous autowiring.
|
||||||
21
README.md
Normal file
21
README.md
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
# trueref
|
||||||
|
|
||||||
|
Self-hosted [Context7](https://github.com/upstash/context7) clone in Java 21 + Spring Boot 3.5: indexes git repositories per tag, exposes a Streamable-HTTP MCP server, REST + Swagger, and a SvelteKit dashboard for ingestion observability and querying.
|
||||||
|
|
||||||
|
See:
|
||||||
|
- [ARCHITECTURE.md](ARCHITECTURE.md) — design, hexagonal layout, pipelines, MCP/REST surfaces.
|
||||||
|
- [CODE_STYLE.md](CODE_STYLE.md) — conventions.
|
||||||
|
- [FINDINGS.md](FINDINGS.md) — research notes backing every choice.
|
||||||
|
|
||||||
|
## Quickstart
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./mvnw -DskipTests package
|
||||||
|
java -jar trueref-bootstrap/target/trueref.jar
|
||||||
|
```
|
||||||
|
|
||||||
|
Browse:
|
||||||
|
- UI: http://localhost:8080/
|
||||||
|
- Swagger: http://localhost:8080/swagger-ui.html
|
||||||
|
- MCP endpoint: http://localhost:8080/mcp
|
||||||
|
- Actuator: http://localhost:8080/actuator
|
||||||
198
pom.xml
Normal file
198
pom.xml
Normal file
@@ -0,0 +1,198 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||||
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||||
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||||
|
<modelVersion>4.0.0</modelVersion>
|
||||||
|
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-parent</artifactId>
|
||||||
|
<version>0.1.0-SNAPSHOT</version>
|
||||||
|
<packaging>pom</packaging>
|
||||||
|
|
||||||
|
<name>trueref</name>
|
||||||
|
<description>Self-hosted Context7-style library docs indexer + MCP server</description>
|
||||||
|
|
||||||
|
<modules>
|
||||||
|
<module>trueref-domain</module>
|
||||||
|
<module>trueref-application</module>
|
||||||
|
<module>trueref-adapters</module>
|
||||||
|
<module>trueref-frontend</module>
|
||||||
|
<module>trueref-bootstrap</module>
|
||||||
|
</modules>
|
||||||
|
|
||||||
|
<parent>
|
||||||
|
<groupId>org.springframework.boot</groupId>
|
||||||
|
<artifactId>spring-boot-starter-parent</artifactId>
|
||||||
|
<version>3.5.3</version>
|
||||||
|
<relativePath/>
|
||||||
|
</parent>
|
||||||
|
|
||||||
|
<properties>
|
||||||
|
<java.version>21</java.version>
|
||||||
|
<maven.compiler.release>21</maven.compiler.release>
|
||||||
|
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||||
|
|
||||||
|
<spring-ai.version>1.0.0</spring-ai.version>
|
||||||
|
<springdoc.version>2.8.6</springdoc.version>
|
||||||
|
<jgit.version>7.3.0.202506031305-r</jgit.version>
|
||||||
|
<lucene.version>10.4.0</lucene.version>
|
||||||
|
<onnxruntime.version>1.22.0</onnxruntime.version>
|
||||||
|
<huggingface-tokenizers.version>0.33.0</huggingface-tokenizers.version>
|
||||||
|
<h2.version>2.3.232</h2.version>
|
||||||
|
<flyway.version>11.8.2</flyway.version>
|
||||||
|
<jspecify.version>1.0.0</jspecify.version>
|
||||||
|
<assertj.version>3.26.3</assertj.version>
|
||||||
|
|
||||||
|
<!-- Plugins -->
|
||||||
|
<spotless.version>2.43.0</spotless.version>
|
||||||
|
<frontend-maven-plugin.version>1.15.1</frontend-maven-plugin.version>
|
||||||
|
<node.version>v20.18.0</node.version>
|
||||||
|
<npm.version>10.8.2</npm.version>
|
||||||
|
</properties>
|
||||||
|
|
||||||
|
<dependencyManagement>
|
||||||
|
<dependencies>
|
||||||
|
<!-- Internal modules -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-domain</artifactId>
|
||||||
|
<version>${project.version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-application</artifactId>
|
||||||
|
<version>${project.version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-adapters</artifactId>
|
||||||
|
<version>${project.version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-frontend</artifactId>
|
||||||
|
<version>${project.version}</version>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!-- Spring AI BOM -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.springframework.ai</groupId>
|
||||||
|
<artifactId>spring-ai-bom</artifactId>
|
||||||
|
<version>${spring-ai.version}</version>
|
||||||
|
<type>pom</type>
|
||||||
|
<scope>import</scope>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!-- 3rd-party -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.springdoc</groupId>
|
||||||
|
<artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
|
||||||
|
<version>${springdoc.version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.eclipse.jgit</groupId>
|
||||||
|
<artifactId>org.eclipse.jgit</artifactId>
|
||||||
|
<version>${jgit.version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.eclipse.jgit</groupId>
|
||||||
|
<artifactId>org.eclipse.jgit.ssh.apache</artifactId>
|
||||||
|
<version>${jgit.version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.apache.lucene</groupId>
|
||||||
|
<artifactId>lucene-core</artifactId>
|
||||||
|
<version>${lucene.version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.apache.lucene</groupId>
|
||||||
|
<artifactId>lucene-analysis-common</artifactId>
|
||||||
|
<version>${lucene.version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.apache.lucene</groupId>
|
||||||
|
<artifactId>lucene-queryparser</artifactId>
|
||||||
|
<version>${lucene.version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.microsoft.onnxruntime</groupId>
|
||||||
|
<artifactId>onnxruntime</artifactId>
|
||||||
|
<version>${onnxruntime.version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.microsoft.onnxruntime</groupId>
|
||||||
|
<artifactId>onnxruntime_gpu</artifactId>
|
||||||
|
<version>${onnxruntime.version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>ai.djl.huggingface</groupId>
|
||||||
|
<artifactId>tokenizers</artifactId>
|
||||||
|
<version>${huggingface-tokenizers.version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.jspecify</groupId>
|
||||||
|
<artifactId>jspecify</artifactId>
|
||||||
|
<version>${jspecify.version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.assertj</groupId>
|
||||||
|
<artifactId>assertj-core</artifactId>
|
||||||
|
<version>${assertj.version}</version>
|
||||||
|
<scope>test</scope>
|
||||||
|
</dependency>
|
||||||
|
</dependencies>
|
||||||
|
</dependencyManagement>
|
||||||
|
|
||||||
|
<dependencies>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.jspecify</groupId>
|
||||||
|
<artifactId>jspecify</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.junit.jupiter</groupId>
|
||||||
|
<artifactId>junit-jupiter</artifactId>
|
||||||
|
<scope>test</scope>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.assertj</groupId>
|
||||||
|
<artifactId>assertj-core</artifactId>
|
||||||
|
<scope>test</scope>
|
||||||
|
</dependency>
|
||||||
|
</dependencies>
|
||||||
|
|
||||||
|
<build>
|
||||||
|
<pluginManagement>
|
||||||
|
<plugins>
|
||||||
|
<plugin>
|
||||||
|
<groupId>com.diffplug.spotless</groupId>
|
||||||
|
<artifactId>spotless-maven-plugin</artifactId>
|
||||||
|
<version>${spotless.version}</version>
|
||||||
|
<configuration>
|
||||||
|
<java>
|
||||||
|
<palantirJavaFormat/>
|
||||||
|
<removeUnusedImports/>
|
||||||
|
<importOrder/>
|
||||||
|
<trimTrailingWhitespace/>
|
||||||
|
<endWithNewline/>
|
||||||
|
</java>
|
||||||
|
</configuration>
|
||||||
|
</plugin>
|
||||||
|
<plugin>
|
||||||
|
<groupId>com.github.eirslett</groupId>
|
||||||
|
<artifactId>frontend-maven-plugin</artifactId>
|
||||||
|
<version>${frontend-maven-plugin.version}</version>
|
||||||
|
</plugin>
|
||||||
|
</plugins>
|
||||||
|
</pluginManagement>
|
||||||
|
<plugins>
|
||||||
|
<plugin>
|
||||||
|
<groupId>org.apache.maven.plugins</groupId>
|
||||||
|
<artifactId>maven-compiler-plugin</artifactId>
|
||||||
|
<configuration>
|
||||||
|
<release>${java.version}</release>
|
||||||
|
<parameters>true</parameters>
|
||||||
|
</configuration>
|
||||||
|
</plugin>
|
||||||
|
</plugins>
|
||||||
|
</build>
|
||||||
|
</project>
|
||||||
611
tests/quality/phaser_rag_eval.py
Normal file
611
tests/quality/phaser_rag_eval.py
Normal file
@@ -0,0 +1,611 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Phaser RAG Quality Evaluation Suite
|
||||||
|
====================================
|
||||||
|
Simulates an LLM querying TrueRef for Phaser documentation and guidance.
|
||||||
|
Tests are designed to be hard and objective: each defines exact expected content
|
||||||
|
fragments and/or expected source files that MUST appear in the top-k results.
|
||||||
|
|
||||||
|
Scoring metrics per test:
|
||||||
|
file@1 - expected file appeared as hit #1
|
||||||
|
file@3 - expected file appeared in hits 1-3
|
||||||
|
file@5 - expected file appeared in hits 1-5
|
||||||
|
content@5 - at least one expected content fragment found across the top-5 hits combined
|
||||||
|
content@1 - expected content fragment found in hit #1
|
||||||
|
|
||||||
|
Overall suite scores:
|
||||||
|
MRR - Mean Reciprocal Rank (file position)
|
||||||
|
P@1..5 - Precision@k for file hits
|
||||||
|
C@5 - Content recall across top-5
|
||||||
|
|
||||||
|
Run:
|
||||||
|
python3 phaser_rag_eval.py [--base-url http://localhost:18080] [--verbose]
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Optional
|
||||||
|
import urllib.request
|
||||||
|
import urllib.error
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Config
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
REPO_ID = "50010965-aa3f-45f4-bb8d-72a0d50bf0db"
|
||||||
|
|
||||||
|
# Version IDs pinned to specific tags (fetched at startup if not found)
|
||||||
|
VERSIONS = {
|
||||||
|
"v4.1.0": "6c6a00f5-0945-4fd7-b62c-c0e69f14effe",
|
||||||
|
"v3.88.0": "d032d4d4-e6bc-4c9d-9c3c-8853e4a1cdc9",
|
||||||
|
"v3.85.2": "d1cf906e-54b9-416f-bd5b-9432d69d9935",
|
||||||
|
"v3.60.0": "95d0a8e2-9071-4986-85d4-59ae97893353",
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Test definition
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
@dataclass
|
||||||
|
class TestCase:
|
||||||
|
id: str
|
||||||
|
name: str
|
||||||
|
query: str
|
||||||
|
version: str # key into VERSIONS
|
||||||
|
topic: Optional[str] = None
|
||||||
|
expected_files: list[str] = field(default_factory=list) # substrings of filePath
|
||||||
|
expected_content: list[str] = field(default_factory=list) # substrings that MUST appear
|
||||||
|
required_content: list[str] = field(default_factory=list) # ALL of these must appear (stricter)
|
||||||
|
max_hits: int = 10
|
||||||
|
tokens_budget: int = 6000
|
||||||
|
# Optional: minimum rerank score the top hit should exceed
|
||||||
|
min_score: Optional[float] = None
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Test definitions — 25 hard, objective cases
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
TESTS: list[TestCase] = [
|
||||||
|
|
||||||
|
# ── 1. Tween system: basic config properties ──────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T01",
|
||||||
|
name="Tween config: yoyo/hold/repeatDelay properties",
|
||||||
|
query="What properties can I set in a TweenBuilderConfig to make a tween yoyo with a hold and repeat delay?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="tweens",
|
||||||
|
expected_files=["tweens/builders/TweenBuilder.js", "tweens/typedefs"],
|
||||||
|
expected_content=["yoyo", "hold", "repeatDelay"],
|
||||||
|
required_content=["yoyo", "repeatDelay"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 2. Tween system: onComplete / onUpdate callbacks ──────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T02",
|
||||||
|
name="Tween callbacks: onComplete and onUpdate signatures",
|
||||||
|
query="How do I use onComplete and onUpdate callbacks in a Phaser tween? What arguments do they receive?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="tweens",
|
||||||
|
expected_files=["tweens/"],
|
||||||
|
expected_content=["onComplete", "onUpdate", "onStart"],
|
||||||
|
required_content=["onComplete"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 3. Arcade physics: setCollideWorldBounds signature ────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T03",
|
||||||
|
name="Arcade physics: setCollideWorldBounds signature",
|
||||||
|
query="What are the parameters of setCollideWorldBounds in Phaser Arcade physics? Can I pass bounceX and bounceY to set bounce on world edges?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="physics",
|
||||||
|
expected_files=["physics/arcade/Body.js"],
|
||||||
|
expected_content=["setCollideWorldBounds", "bounceX", "bounceY", "onWorldBounds"],
|
||||||
|
required_content=["setCollideWorldBounds", "bounceX"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 4. Arcade physics: addCollider vs addOverlap ──────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T04",
|
||||||
|
name="Arcade physics: addCollider vs addOverlap difference",
|
||||||
|
query="What is the difference between addCollider and addOverlap in Phaser's Arcade physics World? How do I add a callback?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="physics",
|
||||||
|
expected_files=["physics/arcade/World.js"],
|
||||||
|
expected_content=["addCollider", "addOverlap", "collideCallback", "processCallback"],
|
||||||
|
required_content=["addCollider", "addOverlap"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 5. Camera: shake parameters ───────────────────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T05",
|
||||||
|
name="Camera shake: duration, intensity, force, callback",
|
||||||
|
query="How do I make the camera shake in Phaser? What parameters does camera.shake accept?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="camera",
|
||||||
|
expected_files=["cameras/2d/Camera.js"],
|
||||||
|
expected_content=["shake", "duration", "intensity", "force", "callback"],
|
||||||
|
required_content=["shake", "intensity"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 6. Camera: startFollow with lerp ─────────────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T06",
|
||||||
|
name="Camera follow: startFollow lerpX lerpY parameters",
|
||||||
|
query="How do I make the Phaser camera follow a player with smooth lerp? What are the lerpX and lerpY parameters?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="camera",
|
||||||
|
expected_files=["cameras/2d/Camera.js"],
|
||||||
|
expected_content=["startFollow", "lerpX", "lerpY", "roundPixels"],
|
||||||
|
required_content=["startFollow", "lerpX"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 7. Camera: setDeadzone ────────────────────────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T07",
|
||||||
|
name="Camera deadzone: setDeadzone width/height",
|
||||||
|
query="How does camera deadzone work in Phaser? How do I create a rectangular deadzone so the camera only moves when the player exits it?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="camera",
|
||||||
|
expected_files=["cameras/2d/Camera.js"],
|
||||||
|
expected_content=["setDeadzone", "deadzone"],
|
||||||
|
required_content=["setDeadzone"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 8. Scene: pass data when starting another scene ───────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T08",
|
||||||
|
name="Scene management: pass data on scene.start",
|
||||||
|
query="How do I pass data to another scene when calling scene.start() or scene.launch()? How does the init method receive it?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="scenes",
|
||||||
|
expected_files=["scene/"],
|
||||||
|
expected_content=["init", "data", "start", "launch"],
|
||||||
|
required_content=["init"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 9. Animation system: chaining animations ──────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T09",
|
||||||
|
name="Animation chaining: chain() and playAfterRepeat()",
|
||||||
|
query="How can I chain multiple animations so one plays after another finishes in Phaser? What is the chain() method?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="animations",
|
||||||
|
expected_files=["gameobjects/sprite/Sprite.js", "animations/"],
|
||||||
|
expected_content=["chain", "playAfterRepeat", "playAfterDelay"],
|
||||||
|
required_content=["chain"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 10. Animation system: events ─────────────────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T10",
|
||||||
|
name="Animation events: ANIMATION_COMPLETE, ANIMATION_START",
|
||||||
|
query="What events does the Phaser animation system emit? How do I listen for when an animation completes on a specific sprite?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="animations",
|
||||||
|
expected_files=["animations/events/"],
|
||||||
|
expected_content=["ANIMATION_COMPLETE", "ANIMATION_START", "ANIMATION_STOP"],
|
||||||
|
required_content=["ANIMATION_COMPLETE"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 11. Input: pointer events ─────────────────────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T11",
|
||||||
|
name="Input: setInteractive + pointerdown/pointerover events",
|
||||||
|
query="How do I call setInteractive on a game object and listen for pointerdown and pointerover events in Phaser?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="input",
|
||||||
|
expected_files=["input/"],
|
||||||
|
expected_content=["pointerdown", "pointerover", "pointerout", "setInteractive"],
|
||||||
|
required_content=["setInteractive", "pointerdown"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 12. Input: keyboard cursor keys ──────────────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T12",
|
||||||
|
name="Input: createCursorKeys and keyboard key states",
|
||||||
|
query="How do I read arrow key input in Phaser? How does createCursorKeys() work and how do I check if a key is down?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="input",
|
||||||
|
expected_files=["input/keyboard/"],
|
||||||
|
expected_content=["createCursorKeys", "isDown", "up", "down", "left", "right"],
|
||||||
|
required_content=["createCursorKeys"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 13. Loader: atlas and texture keys ───────────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T13",
|
||||||
|
name="Loader: load.atlas config object and frame keys",
|
||||||
|
query="How do I load a texture atlas in Phaser? What are the arguments to this.load.atlas() and how do I use frame keys?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="loader",
|
||||||
|
expected_files=["loader/filetypes/AtlasJSONFile.js", "loader/"],
|
||||||
|
expected_content=["atlas", "textureURL", "atlasURL", "frameConfig"],
|
||||||
|
required_content=["atlas"],
|
||||||
|
min_score=0.7,
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 14. Tilemaps: setCollisionBetween ────────────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T14",
|
||||||
|
name="Tilemap: setCollisionBetween start/stop parameters",
|
||||||
|
query="How do I set collision on a range of tile indices in a Phaser tilemap? What does setCollisionBetween do?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="tilemaps",
|
||||||
|
expected_files=["tilemaps/Tilemap.js", "tilemaps/"],
|
||||||
|
expected_content=["setCollisionBetween", "start", "stop", "collides", "recalculateFaces"],
|
||||||
|
required_content=["setCollisionBetween"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 15. Tilemaps: createFromObjects ──────────────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T15",
|
||||||
|
name="Tilemap: createFromObjects from Tiled object layer",
|
||||||
|
query="How do I convert Tiled object layer objects into Phaser game objects? How does createFromObjects work?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="tilemaps",
|
||||||
|
expected_files=["tilemaps/Tilemap.js"],
|
||||||
|
expected_content=["createFromObjects", "objectLayerName"],
|
||||||
|
required_content=["createFromObjects"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 16. RenderTexture: beginDraw / endDraw (v3 API) ──────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T16",
|
||||||
|
name="RenderTexture v3: beginDraw / batchDraw / endDraw pattern",
|
||||||
|
query="How do I use beginDraw and endDraw on a Phaser RenderTexture for batch drawing? What is the workflow?",
|
||||||
|
version="v3.85.2",
|
||||||
|
topic="rendering",
|
||||||
|
expected_files=["textures/DynamicTexture.js"],
|
||||||
|
expected_content=["beginDraw", "endDraw", "batchDraw", "batchDrawFrame"],
|
||||||
|
required_content=["beginDraw", "endDraw"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 17. Masking: BitmapMask vs GeometryMask (v3 API) ──────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T17",
|
||||||
|
name="Masking v3: createBitmapMask vs createGeometryMask",
|
||||||
|
query="What is the difference between a BitmapMask and a GeometryMask in Phaser? How do I create and apply them?",
|
||||||
|
version="v3.85.2",
|
||||||
|
topic="rendering",
|
||||||
|
expected_files=["gameobjects/components/Mask.js", "display/mask/"],
|
||||||
|
expected_content=["createBitmapMask", "createGeometryMask", "setMask", "BitmapMask", "GeometryMask"],
|
||||||
|
required_content=["BitmapMask", "GeometryMask"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 18. Groups: getFirstDead / getFirstAlive pool pattern ─────────────
|
||||||
|
TestCase(
|
||||||
|
id="T18",
|
||||||
|
name="Group: object pool with getFirstDead / getFirstAlive",
|
||||||
|
query="How do I implement an object pool in Phaser using a Group? What are getFirstDead and getFirstAlive?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="gameobjects",
|
||||||
|
expected_files=["gameobjects/group/Group.js"],
|
||||||
|
expected_content=["getFirstDead", "getFirstAlive", "createIfNull", "countActive"],
|
||||||
|
required_content=["getFirstDead", "getFirstAlive"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 19. Matter.js: fromVertices custom body shape ─────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T19",
|
||||||
|
name="Matter.js: custom body shape with fromVertices",
|
||||||
|
query="How do I create a custom polygon physics body from vertices in Phaser's Matter.js physics?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="physics",
|
||||||
|
expected_files=["physics/matter-js/Factory.js", "physics/matter-js/"],
|
||||||
|
expected_content=["fromVertices", "vertexSets", "options"],
|
||||||
|
required_content=["fromVertices"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 20. Game config: FPS limit / target ───────────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T20",
|
||||||
|
name="Game config: fps.target and fps.limit settings",
|
||||||
|
query="How do I configure the target frame rate and FPS limit in the Phaser game config? What is the difference between target and limit?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="core",
|
||||||
|
expected_files=["core/TimeStep.js", "core/Config.js"],
|
||||||
|
expected_content=["targetFps", "fpsLimit", "target", "fps"],
|
||||||
|
required_content=["targetFps"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 21. Scale Manager: ScaleModes ────────────────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T21",
|
||||||
|
name="Scale Manager: FIT vs ENVELOP scale modes",
|
||||||
|
query="What scale modes are available in Phaser's Scale Manager? How does FIT differ from ENVELOP? How do I make a responsive game?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="scale",
|
||||||
|
expected_files=["scale/"],
|
||||||
|
expected_content=["FIT", "ENVELOP", "ScaleManager", "autoCenter"],
|
||||||
|
required_content=["FIT"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 22. Data Manager: set/get/events ──────────────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T22",
|
||||||
|
name="Data Manager: set/get and CHANGE_DATA event",
|
||||||
|
query="How does the Phaser Data Manager work? How do I watch for data changes using events on a game object's data?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="data",
|
||||||
|
expected_files=["data/DataManager.js", "data/"],
|
||||||
|
expected_content=["CHANGE_DATA", "set", "get", "events"],
|
||||||
|
required_content=["CHANGE_DATA"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 23. Depth sort: setDepth and displayList ──────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T23",
|
||||||
|
name="Depth sorting: setDepth and display list ordering",
|
||||||
|
query="How does Phaser handle rendering order (z-order)? How do I use setDepth to control which objects render on top?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="rendering",
|
||||||
|
expected_files=["gameobjects/"],
|
||||||
|
expected_content=["setDepth", "depth", "displayList"],
|
||||||
|
required_content=["setDepth"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 24. Version diff: v3.60 TweenChain (new in 3.60) ─────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T24",
|
||||||
|
name="Version-specific: TweenChain introduced in v3.60",
|
||||||
|
query="How do I create a sequence of tweens that play one after another using TweenChain in Phaser 3.60+?",
|
||||||
|
version="v3.60.0",
|
||||||
|
topic="tweens",
|
||||||
|
expected_files=["tweens/"],
|
||||||
|
expected_content=["TweenChain", "chain"],
|
||||||
|
required_content=["TweenChain"],
|
||||||
|
),
|
||||||
|
|
||||||
|
# ── 25. Hard adversarial: camera.ignore() ─────────────────────────────
|
||||||
|
TestCase(
|
||||||
|
id="T25",
|
||||||
|
name="Camera: ignore() to exclude game objects from a camera",
|
||||||
|
query="How do I make a game object invisible to a specific camera in Phaser while remaining visible to others? What is camera.ignore()?",
|
||||||
|
version="v4.1.0",
|
||||||
|
topic="camera",
|
||||||
|
expected_files=["cameras/2d/"],
|
||||||
|
expected_content=["ignore", "camera"],
|
||||||
|
required_content=["ignore"],
|
||||||
|
),
|
||||||
|
]
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# HTTP helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def post_json(url: str, payload: dict) -> dict:
|
||||||
|
body = json.dumps(payload).encode()
|
||||||
|
req = urllib.request.Request(
|
||||||
|
url, data=body,
|
||||||
|
headers={"Content-Type": "application/json", "Accept": "application/json"},
|
||||||
|
method="POST",
|
||||||
|
)
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||||
|
return json.loads(resp.read().decode())
|
||||||
|
|
||||||
|
|
||||||
|
def get_json(url: str) -> dict | list:
|
||||||
|
req = urllib.request.Request(url, headers={"Accept": "application/json"})
|
||||||
|
with urllib.request.urlopen(req, timeout=15) as resp:
|
||||||
|
return json.loads(resp.read().decode())
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Evaluation logic
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class TestResult:
|
||||||
|
test: TestCase
|
||||||
|
hits: list[dict]
|
||||||
|
elapsed_ms: float
|
||||||
|
error: Optional[str] = None
|
||||||
|
|
||||||
|
# Computed below
|
||||||
|
file_rank: Optional[int] = None # 1-based rank of first expected-file match
|
||||||
|
content_ranks: list[int] = field(default_factory=list) # 1-based ranks where content found
|
||||||
|
required_found: bool = False
|
||||||
|
top_score: Optional[float] = None
|
||||||
|
|
||||||
|
def file_at(self, k: int) -> bool:
|
||||||
|
return self.file_rank is not None and self.file_rank <= k
|
||||||
|
|
||||||
|
def content_at(self, k: int) -> bool:
|
||||||
|
return any(r <= k for r in self.content_ranks)
|
||||||
|
|
||||||
|
def mrr(self) -> float:
|
||||||
|
if self.file_rank is None:
|
||||||
|
return 0.0
|
||||||
|
return 1.0 / self.file_rank
|
||||||
|
|
||||||
|
def summary_line(self) -> str:
|
||||||
|
f1 = "✓" if self.file_at(1) else "·"
|
||||||
|
f3 = "✓" if self.file_at(3) else "·"
|
||||||
|
f5 = "✓" if self.file_at(5) else "·"
|
||||||
|
c5 = "✓" if self.content_at(5) else "·"
|
||||||
|
req = "✓" if self.required_found else "✗"
|
||||||
|
rank_str = f"rank={self.file_rank}" if self.file_rank else "NOT FOUND"
|
||||||
|
score_str = f"score={self.top_score:.3f}" if self.top_score else ""
|
||||||
|
ms_str = f"{self.elapsed_ms:.0f}ms"
|
||||||
|
return (
|
||||||
|
f"[{self.test.id}] {self.test.name[:52]:<52} "
|
||||||
|
f"f@1={f1} f@3={f3} f@5={f5} c@5={c5} req={req} "
|
||||||
|
f"{rank_str:>12} {score_str} {ms_str}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def evaluate(result: TestResult, verbose: bool = False) -> None:
|
||||||
|
hits = result.hits
|
||||||
|
if not hits:
|
||||||
|
return
|
||||||
|
|
||||||
|
result.top_score = hits[0].get("score") if hits else None
|
||||||
|
|
||||||
|
# File rank: position of first hit whose filePath matches any expected_files substring
|
||||||
|
for i, hit in enumerate(hits):
|
||||||
|
fp = hit.get("filePath", "")
|
||||||
|
if any(ef in fp for ef in result.test.expected_files):
|
||||||
|
result.file_rank = i + 1
|
||||||
|
break
|
||||||
|
|
||||||
|
# Content rank: for each expected_content fragment, find the first hit that contains it
|
||||||
|
combined_content = {i: (hit.get("content") or "") for i, hit in enumerate(hits)}
|
||||||
|
|
||||||
|
for fragment in result.test.expected_content:
|
||||||
|
for i, content in combined_content.items():
|
||||||
|
if fragment.lower() in content.lower():
|
||||||
|
result.content_ranks.append(i + 1)
|
||||||
|
break
|
||||||
|
|
||||||
|
# Required content: ALL required fragments must appear somewhere in top-10
|
||||||
|
all_content = " ".join(combined_content.values()).lower()
|
||||||
|
result.required_found = all(
|
||||||
|
r.lower() in all_content for r in result.test.required_content
|
||||||
|
)
|
||||||
|
|
||||||
|
if verbose:
|
||||||
|
print(f"\n{'─'*80}")
|
||||||
|
print(f"[{result.test.id}] {result.test.name}")
|
||||||
|
print(f" Query: {result.test.query}")
|
||||||
|
print(f" Expected files: {result.test.expected_files}")
|
||||||
|
print(f" Expected content: {result.test.expected_content}")
|
||||||
|
print(f" Top hits:")
|
||||||
|
for i, hit in enumerate(hits[:5]):
|
||||||
|
fp = hit.get("filePath", "?")
|
||||||
|
score = hit.get("score", 0.0)
|
||||||
|
snip = (hit.get("content") or "")[:100].replace("\n", " ")
|
||||||
|
marker = " ← FILE MATCH" if any(ef in fp for ef in result.test.expected_files) else ""
|
||||||
|
print(f" [{i+1}] score={score:.3f} {fp}{marker}")
|
||||||
|
print(f" {snip}")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main runner
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def run(base_url: str, verbose: bool) -> None:
|
||||||
|
base_url = base_url.rstrip("/")
|
||||||
|
search_url = f"{base_url}/api/search"
|
||||||
|
versions_url = f"{base_url}/api/repos/{REPO_ID}/versions"
|
||||||
|
|
||||||
|
print(f"TrueRef Phaser RAG Evaluation Suite")
|
||||||
|
print(f"Server : {base_url}")
|
||||||
|
print(f"Tests : {len(TESTS)}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
# Resolve version IDs from server (in case they differ)
|
||||||
|
try:
|
||||||
|
all_versions = get_json(versions_url)
|
||||||
|
live_map = {v["tag"]: v["id"] for v in all_versions if v.get("status") == "INDEXED"}
|
||||||
|
for tag in list(VERSIONS.keys()):
|
||||||
|
if tag in live_map:
|
||||||
|
VERSIONS[tag] = live_map[tag]
|
||||||
|
except Exception as e:
|
||||||
|
print(f"WARN: could not refresh version IDs: {e}")
|
||||||
|
|
||||||
|
results: list[TestResult] = []
|
||||||
|
|
||||||
|
for tc in TESTS:
|
||||||
|
version_id = VERSIONS.get(tc.version)
|
||||||
|
if not version_id:
|
||||||
|
print(f"SKIP [{tc.id}]: version {tc.version} not available")
|
||||||
|
continue
|
||||||
|
|
||||||
|
payload = {
|
||||||
|
"text": tc.query,
|
||||||
|
"scope": [{"repoId": REPO_ID, "versionId": version_id}],
|
||||||
|
"maxHits": tc.max_hits,
|
||||||
|
"tokensBudget": tc.tokens_budget,
|
||||||
|
}
|
||||||
|
if tc.topic:
|
||||||
|
payload["topic"] = tc.topic
|
||||||
|
|
||||||
|
t0 = time.time()
|
||||||
|
try:
|
||||||
|
resp = post_json(search_url, payload)
|
||||||
|
elapsed = (time.time() - t0) * 1000
|
||||||
|
hits = resp.get("hits", [])
|
||||||
|
tr = TestResult(test=tc, hits=hits, elapsed_ms=elapsed)
|
||||||
|
evaluate(tr, verbose=verbose)
|
||||||
|
except Exception as e:
|
||||||
|
elapsed = (time.time() - t0) * 1000
|
||||||
|
tr = TestResult(test=tc, hits=[], elapsed_ms=elapsed, error=str(e))
|
||||||
|
print(f"ERROR [{tc.id}]: {e}")
|
||||||
|
|
||||||
|
results.append(tr)
|
||||||
|
|
||||||
|
# ── Summary table ─────────────────────────────────────────────────────
|
||||||
|
print()
|
||||||
|
print("=" * 110)
|
||||||
|
print(f"{'TEST ID + NAME':<56} {'f@1':>4} {'f@3':>4} {'f@5':>4} {'c@5':>4} {'req':>4} {'file rank':>12} {'score':>10} {'ms':>6}")
|
||||||
|
print("=" * 110)
|
||||||
|
|
||||||
|
for tr in results:
|
||||||
|
if tr.error:
|
||||||
|
print(f"[{tr.test.id}] {'ERROR: ' + tr.test.name[:45]:<52} ERROR: {tr.error[:40]}")
|
||||||
|
else:
|
||||||
|
print(tr.summary_line())
|
||||||
|
|
||||||
|
# ── Aggregate metrics ─────────────────────────────────────────────────
|
||||||
|
valid = [tr for tr in results if not tr.error]
|
||||||
|
n = len(valid)
|
||||||
|
if n == 0:
|
||||||
|
print("\nNo valid results.")
|
||||||
|
return
|
||||||
|
|
||||||
|
mrr = sum(tr.mrr() for tr in valid) / n
|
||||||
|
p_at_1 = sum(1 for tr in valid if tr.file_at(1)) / n
|
||||||
|
p_at_3 = sum(1 for tr in valid if tr.file_at(3)) / n
|
||||||
|
p_at_5 = sum(1 for tr in valid if tr.file_at(5)) / n
|
||||||
|
content_at5 = sum(1 for tr in valid if tr.content_at(5)) / n
|
||||||
|
req_recall = sum(1 for tr in valid if tr.required_found) / n
|
||||||
|
avg_ms = sum(tr.elapsed_ms for tr in valid) / n
|
||||||
|
|
||||||
|
print("=" * 110)
|
||||||
|
print()
|
||||||
|
print("Aggregate metrics:")
|
||||||
|
print(f" MRR (file) : {mrr:.4f} ({mrr*100:.1f}%)")
|
||||||
|
print(f" Precision@1 (file) : {p_at_1:.4f} ({p_at_1*100:.1f}%)")
|
||||||
|
print(f" Precision@3 (file) : {p_at_3:.4f} ({p_at_3*100:.1f}%)")
|
||||||
|
print(f" Precision@5 (file) : {p_at_5:.4f} ({p_at_5*100:.1f}%)")
|
||||||
|
print(f" Content recall@5 : {content_at5:.4f} ({content_at5*100:.1f}%)")
|
||||||
|
print(f" Required recall : {req_recall:.4f} ({req_recall*100:.1f}%) ← hardest: ALL required fragments in top-10")
|
||||||
|
print(f" Avg query latency : {avg_ms:.0f} ms")
|
||||||
|
print()
|
||||||
|
|
||||||
|
# ── Failure analysis ──────────────────────────────────────────────────
|
||||||
|
failures = [tr for tr in valid if not tr.file_at(5) or not tr.required_found]
|
||||||
|
if failures:
|
||||||
|
print(f"Improvement targets ({len(failures)} tests below par):")
|
||||||
|
for tr in failures:
|
||||||
|
issues = []
|
||||||
|
if not tr.file_at(5):
|
||||||
|
issues.append(f"file not in top-5 (rank={tr.file_rank})")
|
||||||
|
if not tr.required_found:
|
||||||
|
missing = [r for r in tr.test.required_content
|
||||||
|
if r.lower() not in " ".join(h.get("content","") for h in tr.hits).lower()]
|
||||||
|
issues.append(f"required content missing: {missing}")
|
||||||
|
print(f" [{tr.test.id}] {tr.test.name}: {'; '.join(issues)}")
|
||||||
|
else:
|
||||||
|
print("All tests passed file@5 and required-content checks.")
|
||||||
|
|
||||||
|
# Exit code: 0 if MRR ≥ 0.5 AND required recall ≥ 0.8, else 1
|
||||||
|
if mrr >= 0.5 and req_recall >= 0.8:
|
||||||
|
print("\nResult: PASS")
|
||||||
|
sys.exit(0)
|
||||||
|
else:
|
||||||
|
print(f"\nResult: FAIL (MRR={mrr:.3f} threshold=0.5, req_recall={req_recall:.3f} threshold=0.8)")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Entry point
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
if __name__ == "__main__":
|
||||||
|
parser = argparse.ArgumentParser(description="Phaser RAG quality evaluation")
|
||||||
|
parser.add_argument("--base-url", default="http://localhost:18080",
|
||||||
|
help="TrueRef server base URL")
|
||||||
|
parser.add_argument("--verbose", action="store_true",
|
||||||
|
help="Print per-test hit details")
|
||||||
|
args = parser.parse_args()
|
||||||
|
run(args.base_url, args.verbose)
|
||||||
115
trueref
Executable file
115
trueref
Executable file
@@ -0,0 +1,115 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# trueref launcher (workspace root)
|
||||||
|
#
|
||||||
|
# Wraps the fat JAR with:
|
||||||
|
# - --enable-native-access=ALL-UNNAMED (silences FFM Linker warning from DJL tokenizers)
|
||||||
|
# - --add-modules=jdk.incubator.vector (enables Lucene 10 SIMD codepath)
|
||||||
|
# - cuDNN 9 (cu12 build) on LD_LIBRARY_PATH so ONNX Runtime CUDA EP loads
|
||||||
|
# - CUDA_VISIBLE_DEVICES isolation so ORT BFC arena doesn't trip over the second card
|
||||||
|
# - per-session GPU memory cap so embedder + reranker fit on one card
|
||||||
|
#
|
||||||
|
# Defaults are tuned for this machine (LMDE 7, CUDA 12.4, RTX 2080 SUPER + RTX 3060).
|
||||||
|
# Override anything via env vars or by appending Spring properties to the command line.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./trueref # run with defaults (port 18080)
|
||||||
|
# ./trueref --server.port=8080 # forward Spring properties
|
||||||
|
# TRUEREF_GPU=0 ./trueref # use the 2080 SUPER instead# TRUEREF_GPU=cpu ./trueref # disable CUDA, run on CPU
|
||||||
|
# TRUEREF_HOME=/data/trueref ./trueref # custom data dir
|
||||||
|
#
|
||||||
|
# Env vars:
|
||||||
|
# TRUEREF_GPU GPU index (matches `nvidia-smi -L`) or "cpu". Default: 1
|
||||||
|
# TRUEREF_HOME Data directory. Default: ./data
|
||||||
|
# TRUEREF_PORT HTTP port. Default: 18080
|
||||||
|
# TRUEREF_MEM_LIMIT Per-session GPU mem cap in bytes. Default: 0 (unbounded).
|
||||||
|
# With session-count=1 there is no multi-session contention, so the BFC
|
||||||
|
# arena can grow freely — capping it risks running out of budget during
|
||||||
|
# model-weight loading (~1.5-2 GB) before inference even starts.
|
||||||
|
# Set to e.g. 8589934592 (8 GiB) only if you run multiple pools on one card.
|
||||||
|
# TRUEREF_CUDNN_LIB Directory containing libcudnn.so.9. Default: ./runtime/cudnn/nvidia/cudnn/lib
|
||||||
|
# TRUEREF_JAR Path to the fat JAR. Default: ./trueref-bootstrap/target/trueref.jar
|
||||||
|
# JAVA java binary. Default: $JAVA_HOME/bin/java or `java` on PATH
|
||||||
|
# JAVA_OPTS Extra JVM flags (e.g. -Xmx16g)
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
JAR="${TRUEREF_JAR:-$ROOT/trueref-bootstrap/target/trueref.jar}"
|
||||||
|
GPU="${TRUEREF_GPU:-1}"
|
||||||
|
HOME_DIR="${TRUEREF_HOME:-$ROOT/data}"
|
||||||
|
PORT="${TRUEREF_PORT:-18080}"
|
||||||
|
MEM_LIMIT="${TRUEREF_MEM_LIMIT:-0}"
|
||||||
|
CUDNN_LIB="${TRUEREF_CUDNN_LIB:-$ROOT/runtime/cudnn/nvidia/cudnn/lib}"
|
||||||
|
|
||||||
|
if [[ ! -f "$JAR" ]]; then
|
||||||
|
echo "trueref: jar not found at $JAR" >&2
|
||||||
|
echo "trueref: build it first with: mvn -DskipTests -pl trueref-bootstrap -am package" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Resolve java
|
||||||
|
if [[ -n "${JAVA:-}" ]]; then
|
||||||
|
:
|
||||||
|
elif [[ -n "${JAVA_HOME:-}" && -x "${JAVA_HOME}/bin/java" ]]; then
|
||||||
|
JAVA="${JAVA_HOME}/bin/java"
|
||||||
|
else
|
||||||
|
JAVA="$(command -v java || true)"
|
||||||
|
fi
|
||||||
|
if [[ -z "${JAVA:-}" || ! -x "${JAVA}" ]]; then
|
||||||
|
echo "trueref: java not found; install JDK 21+ or set JAVA_HOME" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
mkdir -p "$HOME_DIR"
|
||||||
|
|
||||||
|
SPRING_ARGS=(
|
||||||
|
"--server.port=$PORT"
|
||||||
|
"--trueref.home=$HOME_DIR"
|
||||||
|
)
|
||||||
|
|
||||||
|
# CUDA setup. "cpu" disables CUDA entirely; otherwise pass the physical GPU index
|
||||||
|
# directly to ORT. ORT's CUDA EP uses the physical device index regardless of
|
||||||
|
# CUDA_VISIBLE_DEVICES remapping — so we pass the physical index and explicitly
|
||||||
|
# unset CUDA_VISIBLE_DEVICES to avoid the two-layer renumbering problem where
|
||||||
|
# CUDA runtime remaps N→0 but ORT still expects the physical N.
|
||||||
|
if [[ "$GPU" == "cpu" || "$GPU" == "CPU" ]]; then
|
||||||
|
echo "trueref: GPU disabled (TRUEREF_GPU=cpu) — embedder/reranker will run on CPU"
|
||||||
|
SPRING_ARGS+=(
|
||||||
|
"--trueref.embedding.onnx-providers=cpu"
|
||||||
|
)
|
||||||
|
else
|
||||||
|
if [[ -d "$CUDNN_LIB" ]]; then
|
||||||
|
export LD_LIBRARY_PATH="${CUDNN_LIB}${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
|
||||||
|
else
|
||||||
|
echo "trueref: TRUEREF_CUDNN_LIB=$CUDNN_LIB not found — CUDA EP will fall back to CPU" >&2
|
||||||
|
echo "trueref: download cu12 cuDNN with:" >&2
|
||||||
|
echo " mkdir -p runtime/cudnn && cd runtime/cudnn && \\" >&2
|
||||||
|
echo " pip download --no-deps --only-binary=:all: --python-version 3.12 \\" >&2
|
||||||
|
echo " --platform manylinux2014_x86_64 'nvidia-cudnn-cu12<10' -d . && \\" >&2
|
||||||
|
echo " unzip -q -o nvidia_cudnn_cu12-*.whl 'nvidia/cudnn/lib/*' && rm *.whl" >&2
|
||||||
|
fi
|
||||||
|
# CUDA runtime respects CUDA_VISIBLE_DEVICES for all allocations (cudaMalloc, BFC arena,
|
||||||
|
# etc.). By restricting CUDA's view to exactly the target GPU, we prevent the runtime from
|
||||||
|
# creating a default context on device 0 before ORT's cudaSetDevice() takes effect.
|
||||||
|
# We always pass gpu-device-id=0 to ORT because CUDA_VISIBLE_DEVICES makes the target
|
||||||
|
# card the ONLY visible device (index 0 in the runtime's view).
|
||||||
|
#
|
||||||
|
# CUDA_DEVICE_ORDER=PCI_BUS_ID ensures CUDA runtime numbering matches nvidia-smi numbering.
|
||||||
|
# Without it, the default FASTEST_FIRST ordering can rank GPUs differently from nvidia-smi,
|
||||||
|
# so CUDA_VISIBLE_DEVICES=N would expose a different physical card than nvidia-smi GPU N.
|
||||||
|
export CUDA_DEVICE_ORDER="PCI_BUS_ID"
|
||||||
|
export CUDA_VISIBLE_DEVICES="$GPU"
|
||||||
|
SPRING_ARGS+=(
|
||||||
|
"--trueref.embedding.gpu-device-id=0"
|
||||||
|
"--trueref.embedding.gpu-mem-limit-bytes=$MEM_LIMIT"
|
||||||
|
)
|
||||||
|
fi
|
||||||
|
|
||||||
|
exec "$JAVA" \
|
||||||
|
--enable-native-access=ALL-UNNAMED \
|
||||||
|
--add-modules=jdk.incubator.vector \
|
||||||
|
${JAVA_OPTS:-} \
|
||||||
|
-jar "$JAR" \
|
||||||
|
"${SPRING_ARGS[@]}" \
|
||||||
|
"$@"
|
||||||
106
trueref-adapters/pom.xml
Normal file
106
trueref-adapters/pom.xml
Normal file
@@ -0,0 +1,106 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||||
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||||
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||||
|
<modelVersion>4.0.0</modelVersion>
|
||||||
|
|
||||||
|
<parent>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-parent</artifactId>
|
||||||
|
<version>0.1.0-SNAPSHOT</version>
|
||||||
|
</parent>
|
||||||
|
|
||||||
|
<artifactId>trueref-adapters</artifactId>
|
||||||
|
<name>trueref-adapters</name>
|
||||||
|
<description>All driving (REST, MCP) and driven (H2, Lucene, ONNX, JGit, tree-sitter, disk cache) adapters.</description>
|
||||||
|
|
||||||
|
<dependencies>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-domain</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-application</artifactId>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!-- Spring Web + JDBC + validation (no auto-config; bootstrap controls @ComponentScan) -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.springframework.boot</groupId>
|
||||||
|
<artifactId>spring-boot-starter-web</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.springframework.boot</groupId>
|
||||||
|
<artifactId>spring-boot-starter-jdbc</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.springframework.boot</groupId>
|
||||||
|
<artifactId>spring-boot-starter-validation</artifactId>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!-- OpenAPI / Swagger UI -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.springdoc</groupId>
|
||||||
|
<artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!-- Spring AI MCP server (Streamable HTTP) -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.springframework.ai</groupId>
|
||||||
|
<artifactId>spring-ai-starter-mcp-server-webmvc</artifactId>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!-- H2 + Flyway -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.h2database</groupId>
|
||||||
|
<artifactId>h2</artifactId>
|
||||||
|
<version>${h2.version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.flywaydb</groupId>
|
||||||
|
<artifactId>flyway-core</artifactId>
|
||||||
|
<version>${flyway.version}</version>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!-- Lucene -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.apache.lucene</groupId>
|
||||||
|
<artifactId>lucene-core</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.apache.lucene</groupId>
|
||||||
|
<artifactId>lucene-analysis-common</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.apache.lucene</groupId>
|
||||||
|
<artifactId>lucene-queryparser</artifactId>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!-- ONNX Runtime: GPU jar contains both CUDA and CPU providers; DirectML jar is added at runtime via classifier on Windows. -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.microsoft.onnxruntime</groupId>
|
||||||
|
<artifactId>onnxruntime_gpu</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>ai.djl.huggingface</groupId>
|
||||||
|
<artifactId>tokenizers</artifactId>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!-- JGit -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.eclipse.jgit</groupId>
|
||||||
|
<artifactId>org.eclipse.jgit</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.eclipse.jgit</groupId>
|
||||||
|
<artifactId>org.eclipse.jgit.ssh.apache</artifactId>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<!-- Test -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.springframework.boot</groupId>
|
||||||
|
<artifactId>spring-boot-starter-test</artifactId>
|
||||||
|
<scope>test</scope>
|
||||||
|
</dependency>
|
||||||
|
</dependencies>
|
||||||
|
</project>
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
package com.trueref.adapter.in.mcp;
|
||||||
|
|
||||||
|
import org.springframework.ai.tool.ToolCallbackProvider;
|
||||||
|
import org.springframework.ai.tool.method.MethodToolCallbackProvider;
|
||||||
|
import org.springframework.boot.context.properties.EnableConfigurationProperties;
|
||||||
|
import org.springframework.context.annotation.Bean;
|
||||||
|
import org.springframework.context.annotation.Configuration;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Registers the trueref MCP tool callbacks with Spring AI's MCP WebMVC auto-configuration. The
|
||||||
|
* {@link MethodToolCallbackProvider} scans {@link TrueRefMcpTools} for methods annotated with
|
||||||
|
* {@link org.springframework.ai.tool.annotation.Tool} and publishes them on the MCP endpoint
|
||||||
|
* configured in {@code application.yml} (POST {@code /mcp} via
|
||||||
|
* {@code spring.ai.mcp.server.sse-message-endpoint}).
|
||||||
|
*/
|
||||||
|
@Configuration
|
||||||
|
@EnableConfigurationProperties(McpProperties.class)
|
||||||
|
public class McpConfig {
|
||||||
|
|
||||||
|
@Bean
|
||||||
|
public ToolCallbackProvider trueRefMcpToolCallbacks(TrueRefMcpTools tools) {
|
||||||
|
return MethodToolCallbackProvider.builder().toolObjects(tools).build();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
package com.trueref.adapter.in.mcp;
|
||||||
|
|
||||||
|
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Token-budget defaults for {@code get-library-docs}. Matches Context7 semantics: clients may
|
||||||
|
* request an explicit token budget per call; unspecified calls use {@link #tokensDefault}. All
|
||||||
|
* requests are clamped to {@code [tokensMin, tokensMax]}.
|
||||||
|
*/
|
||||||
|
@ConfigurationProperties(prefix = "trueref.mcp")
|
||||||
|
public record McpProperties(int tokensDefault, int tokensMin, int tokensMax) {
|
||||||
|
|
||||||
|
public McpProperties {
|
||||||
|
if (tokensDefault <= 0) tokensDefault = 5000;
|
||||||
|
if (tokensMin <= 0) tokensMin = 500;
|
||||||
|
if (tokensMax <= 0) tokensMax = 50_000;
|
||||||
|
}
|
||||||
|
|
||||||
|
public int clamp(int requested) {
|
||||||
|
return Math.max(tokensMin, Math.min(tokensMax, requested));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,297 @@
|
|||||||
|
package com.trueref.adapter.in.mcp;
|
||||||
|
|
||||||
|
import com.trueref.application.resolve.LibraryResolver;
|
||||||
|
import com.trueref.domain.model.Repository;
|
||||||
|
import com.trueref.domain.model.SearchHit;
|
||||||
|
import com.trueref.domain.model.SearchScope;
|
||||||
|
import com.trueref.domain.model.Version;
|
||||||
|
import com.trueref.domain.model.VersionStatus;
|
||||||
|
import com.trueref.domain.port.in.IndexVersion;
|
||||||
|
import com.trueref.domain.port.in.QueryCatalog;
|
||||||
|
import com.trueref.domain.port.in.ResolveLibraryId;
|
||||||
|
import com.trueref.domain.port.in.SearchLibraryDocs;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Optional;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.ai.tool.annotation.Tool;
|
||||||
|
import org.springframework.ai.tool.annotation.ToolParam;
|
||||||
|
import org.springframework.stereotype.Service;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Context7-compatible MCP tool handlers. The two tool names, parameter names, and response
|
||||||
|
* shapes are intentionally 1:1 with upstream Context7 so that any MCP client written against
|
||||||
|
* Context7 works against trueref unchanged.
|
||||||
|
*
|
||||||
|
* <p>Ranking and version→tag mapping are delegated to the application layer
|
||||||
|
* ({@link ResolveLibraryId}, {@link LibraryResolver}); hybrid search goes through
|
||||||
|
* {@link SearchLibraryDocs}; on-demand indexing is enqueued via {@link IndexVersion}.
|
||||||
|
*/
|
||||||
|
@Service
|
||||||
|
public class TrueRefMcpTools {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(TrueRefMcpTools.class);
|
||||||
|
private static final int DEFAULT_MAX_HITS = 50;
|
||||||
|
/** Matches Context7's banner retry hint (see ARCHITECTURE §7 on-demand indexing flow). */
|
||||||
|
private static final int INDEXING_RETRY_AFTER_SEC = 30;
|
||||||
|
|
||||||
|
private final ResolveLibraryId resolver;
|
||||||
|
private final LibraryResolver libraryResolver;
|
||||||
|
private final QueryCatalog catalog;
|
||||||
|
private final SearchLibraryDocs search;
|
||||||
|
private final IndexVersion indexer;
|
||||||
|
private final McpProperties props;
|
||||||
|
|
||||||
|
public TrueRefMcpTools(
|
||||||
|
ResolveLibraryId resolver,
|
||||||
|
LibraryResolver libraryResolver,
|
||||||
|
QueryCatalog catalog,
|
||||||
|
SearchLibraryDocs search,
|
||||||
|
IndexVersion indexer,
|
||||||
|
McpProperties props) {
|
||||||
|
this.resolver = resolver;
|
||||||
|
this.libraryResolver = libraryResolver;
|
||||||
|
this.catalog = catalog;
|
||||||
|
this.search = search;
|
||||||
|
this.indexer = indexer;
|
||||||
|
this.props = props;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Tool(
|
||||||
|
name = "resolve-library-id",
|
||||||
|
description =
|
||||||
|
"Resolves a package/product name to a trueref-compatible library ID and "
|
||||||
|
+ "returns matching libraries. Context7-compatible. Each result "
|
||||||
|
+ "includes: Title, Context7-compatible library ID (format "
|
||||||
|
+ "/owner/repo[/version]), Description, Code Snippets, Versions, "
|
||||||
|
+ "and a relevance Score.")
|
||||||
|
public String resolveLibraryId(
|
||||||
|
@ToolParam(
|
||||||
|
description =
|
||||||
|
"Library name to search for and retrieve a Context7-"
|
||||||
|
+ "compatible library ID.")
|
||||||
|
String libraryName,
|
||||||
|
@ToolParam(
|
||||||
|
required = false,
|
||||||
|
description =
|
||||||
|
"Optional natural-language query used to rank matching "
|
||||||
|
+ "libraries by relevance.")
|
||||||
|
@Nullable String query) {
|
||||||
|
ResolveLibraryId.Result result =
|
||||||
|
resolver.resolve(new ResolveLibraryId.Query(libraryName, query, null));
|
||||||
|
if (result.matches().isEmpty()) {
|
||||||
|
return "No matching libraries found for: " + libraryName;
|
||||||
|
}
|
||||||
|
StringBuilder sb = new StringBuilder();
|
||||||
|
for (ResolveLibraryId.Match m : result.matches()) {
|
||||||
|
appendMatchBlock(sb, m);
|
||||||
|
}
|
||||||
|
return sb.toString();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Tool(
|
||||||
|
name = "get-library-docs",
|
||||||
|
description =
|
||||||
|
"Fetches up-to-date documentation for a library. You MUST call "
|
||||||
|
+ "'resolve-library-id' first to obtain the exact library ID "
|
||||||
|
+ "required to use this tool, UNLESS the user explicitly provides "
|
||||||
|
+ "a library ID in the format /org/project or /org/project/version.")
|
||||||
|
public String getLibraryDocs(
|
||||||
|
@ToolParam(
|
||||||
|
description =
|
||||||
|
"Exact trueref-compatible library ID (format: "
|
||||||
|
+ "/org/project or /org/project/version) "
|
||||||
|
+ "retrieved from 'resolve-library-id'.")
|
||||||
|
String libraryId,
|
||||||
|
@ToolParam(
|
||||||
|
required = false,
|
||||||
|
description =
|
||||||
|
"Topic to focus the documentation on (e.g. "
|
||||||
|
+ "'routing', 'hooks', 'authentication').")
|
||||||
|
@Nullable String topic,
|
||||||
|
@ToolParam(
|
||||||
|
required = false,
|
||||||
|
description =
|
||||||
|
"Max number of tokens to return. Clamped to "
|
||||||
|
+ "[500, 50000]; defaults to 5000.")
|
||||||
|
@Nullable Integer tokens) {
|
||||||
|
ParsedId parsed = parseLibraryId(libraryId);
|
||||||
|
if (parsed == null) {
|
||||||
|
return "Invalid libraryId: " + libraryId
|
||||||
|
+ ". Expected format: /org/project or /org/project/version.";
|
||||||
|
}
|
||||||
|
|
||||||
|
Optional<Repository> repoOpt = catalog.listRepositories().stream()
|
||||||
|
.filter(r -> r.name().equalsIgnoreCase(parsed.repoName()))
|
||||||
|
.findFirst();
|
||||||
|
if (repoOpt.isEmpty()) {
|
||||||
|
return "No matching library found for ID: " + libraryId;
|
||||||
|
}
|
||||||
|
Repository repo = repoOpt.get();
|
||||||
|
List<Version> versions = catalog.listVersions(repo.id());
|
||||||
|
|
||||||
|
SelectedVersion selected = selectVersion(repo, versions, parsed.version());
|
||||||
|
if (selected.searchTarget() == null) {
|
||||||
|
return "No indexed version available for /" + repo.name()
|
||||||
|
+ ". Indexing has been enqueued; retry in ~"
|
||||||
|
+ INDEXING_RETRY_AFTER_SEC + " seconds.";
|
||||||
|
}
|
||||||
|
|
||||||
|
int budget = props.clamp(tokens == null ? props.tokensDefault() : tokens);
|
||||||
|
String text = (topic == null || topic.isBlank()) ? repo.name() : topic;
|
||||||
|
SearchLibraryDocs.Query q = new SearchLibraryDocs.Query(
|
||||||
|
text,
|
||||||
|
topic,
|
||||||
|
new SearchScope(List.of(new SearchScope.RepoVersionRef(repo.id(), selected.searchTarget().id()))),
|
||||||
|
budget,
|
||||||
|
DEFAULT_MAX_HITS);
|
||||||
|
SearchLibraryDocs.Result res;
|
||||||
|
try {
|
||||||
|
res = search.search(q);
|
||||||
|
} catch (Exception e) {
|
||||||
|
log.warn("MCP search failed for {}: {}", libraryId, e.toString());
|
||||||
|
return "Search failed for " + libraryId + ": " + e.getMessage();
|
||||||
|
}
|
||||||
|
|
||||||
|
return formatDocs(res.hits(), selected.banner());
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- helpers -----------------------------------------------------------
|
||||||
|
|
||||||
|
private void appendMatchBlock(StringBuilder sb, ResolveLibraryId.Match m) {
|
||||||
|
sb.append("----------\n");
|
||||||
|
sb.append("- Title: ").append(m.name()).append('\n');
|
||||||
|
sb.append("- Context7-compatible library ID: ").append(m.libraryId()).append('\n');
|
||||||
|
if (m.description() != null && !m.description().isBlank()) {
|
||||||
|
sb.append("- Description: ").append(m.description()).append('\n');
|
||||||
|
}
|
||||||
|
sb.append("- Code Snippets: ").append(m.snippetCount()).append('\n');
|
||||||
|
if (!m.availableVersions().isEmpty()) {
|
||||||
|
sb.append("- Versions: ");
|
||||||
|
for (int i = 0; i < m.availableVersions().size(); i++) {
|
||||||
|
if (i > 0) sb.append(", ");
|
||||||
|
ResolveLibraryId.VersionRef v = m.availableVersions().get(i);
|
||||||
|
sb.append(v.tag()).append(" (").append(v.status()).append(')');
|
||||||
|
}
|
||||||
|
sb.append('\n');
|
||||||
|
}
|
||||||
|
sb.append("- Score: ").append(String.format("%.2f", m.score())).append('\n');
|
||||||
|
}
|
||||||
|
|
||||||
|
private SelectedVersion selectVersion(
|
||||||
|
Repository repo, List<Version> versions, @Nullable String requestedVersion) {
|
||||||
|
if (requestedVersion == null || requestedVersion.isBlank()) {
|
||||||
|
// No version in libraryId: prefer most-recent INDEXED; else nearest DISCOVERED +
|
||||||
|
// enqueue indexing + banner.
|
||||||
|
Optional<Version> latestIndexed = versions.stream()
|
||||||
|
.filter(v -> v.status() == VersionStatus.INDEXED)
|
||||||
|
.max(Comparator.comparing(Version::tag));
|
||||||
|
if (latestIndexed.isPresent()) {
|
||||||
|
return new SelectedVersion(latestIndexed.get(), null);
|
||||||
|
}
|
||||||
|
Optional<Version> latestDiscovered = versions.stream()
|
||||||
|
.filter(v -> v.status() == VersionStatus.DISCOVERED)
|
||||||
|
.max(Comparator.comparing(Version::tag));
|
||||||
|
if (latestDiscovered.isPresent()) {
|
||||||
|
Version v = latestDiscovered.get();
|
||||||
|
enqueueSafely(repo, v);
|
||||||
|
return new SelectedVersion(null, indexingBanner(v.tag(), v.tag()));
|
||||||
|
}
|
||||||
|
return new SelectedVersion(null, null);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Explicit version requested: use application-layer mapper.
|
||||||
|
Optional<Version> mapped = libraryResolver.mapVersion(repo, versions, requestedVersion);
|
||||||
|
if (mapped.isEmpty()) {
|
||||||
|
// Fall back to nearest INDEXED; if any, show banner for the requested version.
|
||||||
|
Optional<Version> nearest = versions.stream()
|
||||||
|
.filter(v -> v.status() == VersionStatus.INDEXED)
|
||||||
|
.max(Comparator.comparing(Version::tag));
|
||||||
|
return new SelectedVersion(
|
||||||
|
nearest.orElse(null), nearest.map(v -> indexingBanner(requestedVersion, v.tag())).orElse(null));
|
||||||
|
}
|
||||||
|
Version target = mapped.get();
|
||||||
|
if (target.status() == VersionStatus.INDEXED) {
|
||||||
|
return new SelectedVersion(target, null);
|
||||||
|
}
|
||||||
|
enqueueSafely(repo, target);
|
||||||
|
Optional<Version> nearestIndexed = versions.stream()
|
||||||
|
.filter(v -> v.status() == VersionStatus.INDEXED)
|
||||||
|
.max(Comparator.comparing(Version::tag));
|
||||||
|
return new SelectedVersion(
|
||||||
|
nearestIndexed.orElse(null),
|
||||||
|
indexingBanner(target.tag(), nearestIndexed.map(Version::tag).orElse("none")));
|
||||||
|
}
|
||||||
|
|
||||||
|
private void enqueueSafely(Repository repo, Version v) {
|
||||||
|
if (v.status() == VersionStatus.INDEXING) return;
|
||||||
|
try {
|
||||||
|
indexer.enqueue(repo.id(), v.id(), false);
|
||||||
|
} catch (Exception e) {
|
||||||
|
log.warn("MCP on-demand indexing enqueue failed for {}@{}: {}", repo.name(), v.tag(), e.toString());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private String indexingBanner(String requestedTag, String fallbackTag) {
|
||||||
|
return "[indexing] version " + requestedTag
|
||||||
|
+ " is being indexed now; showing nearest indexed version " + fallbackTag
|
||||||
|
+ " (retryAfterSec=" + INDEXING_RETRY_AFTER_SEC + ")";
|
||||||
|
}
|
||||||
|
|
||||||
|
private String formatDocs(List<SearchHit> hits, @Nullable String banner) {
|
||||||
|
StringBuilder sb = new StringBuilder();
|
||||||
|
if (banner != null) {
|
||||||
|
sb.append(banner).append('\n').append('\n');
|
||||||
|
}
|
||||||
|
sb.append("================\n");
|
||||||
|
sb.append("CODE SNIPPETS\n");
|
||||||
|
sb.append("================\n");
|
||||||
|
if (hits.isEmpty()) {
|
||||||
|
sb.append("(no matching snippets)\n");
|
||||||
|
return sb.toString();
|
||||||
|
}
|
||||||
|
for (SearchHit h : hits) {
|
||||||
|
sb.append("TITLE: ")
|
||||||
|
.append(h.filePath())
|
||||||
|
.append(':')
|
||||||
|
.append(h.startLine())
|
||||||
|
.append('-')
|
||||||
|
.append(h.endLine())
|
||||||
|
.append(" (")
|
||||||
|
.append(h.language())
|
||||||
|
.append(")\n");
|
||||||
|
sb.append("SOURCE: ")
|
||||||
|
.append(h.repoName())
|
||||||
|
.append('@')
|
||||||
|
.append(h.tag())
|
||||||
|
.append(" — ")
|
||||||
|
.append(h.filePath())
|
||||||
|
.append("\n\n");
|
||||||
|
sb.append("```").append(h.language()).append('\n');
|
||||||
|
sb.append(h.content());
|
||||||
|
if (!h.content().endsWith("\n")) sb.append('\n');
|
||||||
|
sb.append("```\n");
|
||||||
|
sb.append("----------------------------------------\n");
|
||||||
|
}
|
||||||
|
return sb.toString();
|
||||||
|
}
|
||||||
|
|
||||||
|
static @Nullable ParsedId parseLibraryId(String raw) {
|
||||||
|
if (raw == null || raw.isBlank()) return null;
|
||||||
|
String s = raw.startsWith("/") ? raw.substring(1) : raw;
|
||||||
|
String[] parts = s.split("/");
|
||||||
|
if (parts.length == 2) {
|
||||||
|
return new ParsedId(parts[0] + "/" + parts[1], null);
|
||||||
|
}
|
||||||
|
if (parts.length == 3) {
|
||||||
|
return new ParsedId(parts[0] + "/" + parts[1], parts[2]);
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
record ParsedId(String repoName, @Nullable String version) {}
|
||||||
|
|
||||||
|
private record SelectedVersion(@Nullable Version searchTarget, @Nullable String banner) {}
|
||||||
|
}
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
/**
|
||||||
|
* Driving adapter: Model Context Protocol (MCP) server exposing the two Context7-compatible
|
||||||
|
* tools ({@code resolve-library-id}, {@code get-library-docs}) over Spring AI's MCP WebMVC
|
||||||
|
* transport. The HTTP message endpoint is wired to {@code POST /mcp} via
|
||||||
|
* {@code spring.ai.mcp.server.sse-message-endpoint}.
|
||||||
|
*
|
||||||
|
* <p>Spring AI 1.0.0 ships the SSE-based WebMVC transport
|
||||||
|
* ({@code WebMvcSseServerTransportProvider}); the 2025-03-26 "Streamable HTTP" transport is
|
||||||
|
* not a separate selectable property in this version. Clients that POST JSON-RPC to the
|
||||||
|
* configured message endpoint receive JSON-RPC responses; the server additionally opens an
|
||||||
|
* SSE stream on the configured {@code sse-endpoint} for server-initiated notifications. This
|
||||||
|
* is the closest equivalent Spring AI 1.0.0 provides to Streamable HTTP and is the
|
||||||
|
* intended/only transport of this adapter.
|
||||||
|
*/
|
||||||
|
@org.jspecify.annotations.NullMarked
|
||||||
|
package com.trueref.adapter.in.mcp;
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
package com.trueref.adapter.in.rest;
|
||||||
|
|
||||||
|
import io.swagger.v3.oas.annotations.media.Schema;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
/** Uniform error envelope returned by {@link GlobalExceptionHandler}. */
|
||||||
|
@Schema(description = "Error response envelope.")
|
||||||
|
public record ErrorResponse(String code, String message, List<FieldError> fieldErrors) {
|
||||||
|
|
||||||
|
public static ErrorResponse of(String code, String message) {
|
||||||
|
return new ErrorResponse(code, message, List.of());
|
||||||
|
}
|
||||||
|
|
||||||
|
@Schema(description = "A single field-level validation error.")
|
||||||
|
public record FieldError(String field, String message) {}
|
||||||
|
}
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
package com.trueref.adapter.in.rest;
|
||||||
|
|
||||||
|
import com.trueref.domain.error.IngestionFailed;
|
||||||
|
import com.trueref.domain.error.InvalidSearchRequest;
|
||||||
|
import com.trueref.domain.error.RepositoryAlreadyRegistered;
|
||||||
|
import com.trueref.domain.error.RepositoryNotFound;
|
||||||
|
import com.trueref.domain.error.TagNotFound;
|
||||||
|
import com.trueref.domain.error.TrueRefException;
|
||||||
|
import com.trueref.domain.error.VersionNotFound;
|
||||||
|
import com.trueref.domain.error.VersionNotIndexed;
|
||||||
|
import jakarta.validation.ConstraintViolationException;
|
||||||
|
import java.util.List;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.http.HttpStatus;
|
||||||
|
import org.springframework.http.ResponseEntity;
|
||||||
|
import org.springframework.validation.FieldError;
|
||||||
|
import org.springframework.web.bind.MethodArgumentNotValidException;
|
||||||
|
import org.springframework.web.bind.annotation.ExceptionHandler;
|
||||||
|
import org.springframework.web.bind.annotation.RestControllerAdvice;
|
||||||
|
import org.springframework.web.server.ResponseStatusException;
|
||||||
|
import org.springframework.web.servlet.resource.NoResourceFoundException;
|
||||||
|
|
||||||
|
/** Central translator from domain / validation exceptions to HTTP + {@link ErrorResponse} JSON. */
|
||||||
|
@RestControllerAdvice
|
||||||
|
public class GlobalExceptionHandler {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(GlobalExceptionHandler.class);
|
||||||
|
|
||||||
|
@ExceptionHandler({RepositoryNotFound.class, VersionNotFound.class, TagNotFound.class})
|
||||||
|
public ResponseEntity<ErrorResponse> handleNotFound(TrueRefException ex) {
|
||||||
|
return status(HttpStatus.NOT_FOUND, ex);
|
||||||
|
}
|
||||||
|
|
||||||
|
@ExceptionHandler(RepositoryAlreadyRegistered.class)
|
||||||
|
public ResponseEntity<ErrorResponse> handleConflict(RepositoryAlreadyRegistered ex) {
|
||||||
|
return status(HttpStatus.CONFLICT, ex);
|
||||||
|
}
|
||||||
|
|
||||||
|
@ExceptionHandler(VersionNotIndexed.class)
|
||||||
|
public ResponseEntity<ErrorResponse> handleNotIndexed(VersionNotIndexed ex) {
|
||||||
|
return status(HttpStatus.CONFLICT, ex);
|
||||||
|
}
|
||||||
|
|
||||||
|
@ExceptionHandler(InvalidSearchRequest.class)
|
||||||
|
public ResponseEntity<ErrorResponse> handleInvalidSearch(InvalidSearchRequest ex) {
|
||||||
|
return status(HttpStatus.BAD_REQUEST, ex);
|
||||||
|
}
|
||||||
|
|
||||||
|
@ExceptionHandler(MethodArgumentNotValidException.class)
|
||||||
|
public ResponseEntity<ErrorResponse> handleValidation(MethodArgumentNotValidException ex) {
|
||||||
|
List<ErrorResponse.FieldError> fieldErrors = ex.getBindingResult().getFieldErrors().stream()
|
||||||
|
.map(this::toFieldError)
|
||||||
|
.toList();
|
||||||
|
ErrorResponse body = new ErrorResponse("validation_failed", "Request validation failed", fieldErrors);
|
||||||
|
return ResponseEntity.status(HttpStatus.BAD_REQUEST).body(body);
|
||||||
|
}
|
||||||
|
|
||||||
|
@ExceptionHandler(ConstraintViolationException.class)
|
||||||
|
public ResponseEntity<ErrorResponse> handleConstraintViolation(ConstraintViolationException ex) {
|
||||||
|
List<ErrorResponse.FieldError> fieldErrors = ex.getConstraintViolations().stream()
|
||||||
|
.map(v -> new ErrorResponse.FieldError(v.getPropertyPath().toString(), v.getMessage()))
|
||||||
|
.toList();
|
||||||
|
ErrorResponse body = new ErrorResponse("validation_failed", "Request validation failed", fieldErrors);
|
||||||
|
return ResponseEntity.status(HttpStatus.BAD_REQUEST).body(body);
|
||||||
|
}
|
||||||
|
|
||||||
|
@ExceptionHandler(IllegalArgumentException.class)
|
||||||
|
public ResponseEntity<ErrorResponse> handleIllegalArgument(IllegalArgumentException ex) {
|
||||||
|
ErrorResponse body = new ErrorResponse("invalid_request", safeMessage(ex), List.of());
|
||||||
|
return ResponseEntity.status(HttpStatus.BAD_REQUEST).body(body);
|
||||||
|
}
|
||||||
|
|
||||||
|
@ExceptionHandler(ResponseStatusException.class)
|
||||||
|
public ResponseEntity<ErrorResponse> handleResponseStatus(ResponseStatusException ex) {
|
||||||
|
HttpStatus resolved = HttpStatus.resolve(ex.getStatusCode().value());
|
||||||
|
HttpStatus status = resolved == null ? HttpStatus.INTERNAL_SERVER_ERROR : resolved;
|
||||||
|
String code = ex.getReason() == null ? status.name().toLowerCase() : ex.getReason();
|
||||||
|
return ResponseEntity.status(status).body(ErrorResponse.of(code, code));
|
||||||
|
}
|
||||||
|
|
||||||
|
@ExceptionHandler(NoResourceFoundException.class)
|
||||||
|
public ResponseEntity<Void> handleNoResource(NoResourceFoundException ex) {
|
||||||
|
// Static resource not found (e.g. browser/.well-known probes). Return 404 without
|
||||||
|
// logging — these are not application errors and would flood the log at ERROR level.
|
||||||
|
return ResponseEntity.notFound().build();
|
||||||
|
}
|
||||||
|
|
||||||
|
@ExceptionHandler(IngestionFailed.class)
|
||||||
|
public ResponseEntity<ErrorResponse> handleIngestionFailed(IngestionFailed ex) {
|
||||||
|
log.error("ingestion failed", ex);
|
||||||
|
return status(HttpStatus.INTERNAL_SERVER_ERROR, ex);
|
||||||
|
}
|
||||||
|
|
||||||
|
@ExceptionHandler(TrueRefException.class)
|
||||||
|
public ResponseEntity<ErrorResponse> handleDomain(TrueRefException ex) {
|
||||||
|
log.error("unhandled domain error code={}", ex.code(), ex);
|
||||||
|
return status(HttpStatus.INTERNAL_SERVER_ERROR, ex);
|
||||||
|
}
|
||||||
|
|
||||||
|
@ExceptionHandler(Exception.class)
|
||||||
|
public ResponseEntity<ErrorResponse> handleUnexpected(Exception ex) {
|
||||||
|
log.error("unexpected error", ex);
|
||||||
|
ErrorResponse body = new ErrorResponse("internal_error", "An unexpected error occurred", List.of());
|
||||||
|
return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(body);
|
||||||
|
}
|
||||||
|
|
||||||
|
private ResponseEntity<ErrorResponse> status(HttpStatus status, TrueRefException ex) {
|
||||||
|
return ResponseEntity.status(status).body(ErrorResponse.of(ex.code(), safeMessage(ex)));
|
||||||
|
}
|
||||||
|
|
||||||
|
private ErrorResponse.FieldError toFieldError(FieldError fe) {
|
||||||
|
String msg = fe.getDefaultMessage() == null ? "invalid" : fe.getDefaultMessage();
|
||||||
|
return new ErrorResponse.FieldError(fe.getField(), msg);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static String safeMessage(Throwable t) {
|
||||||
|
return t.getMessage() == null ? t.getClass().getSimpleName() : t.getMessage();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,145 @@
|
|||||||
|
package com.trueref.adapter.in.rest;
|
||||||
|
|
||||||
|
import com.trueref.adapter.in.rest.dto.JobDto;
|
||||||
|
import com.trueref.adapter.in.rest.dto.JobLogEventDto;
|
||||||
|
import com.trueref.domain.model.JobId;
|
||||||
|
import com.trueref.domain.model.JobStatus;
|
||||||
|
import com.trueref.domain.model.RepositoryId;
|
||||||
|
import com.trueref.domain.model.VersionId;
|
||||||
|
import com.trueref.domain.port.in.ObserveJobs;
|
||||||
|
import io.swagger.v3.oas.annotations.Operation;
|
||||||
|
import io.swagger.v3.oas.annotations.tags.Tag;
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.util.List;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.http.MediaType;
|
||||||
|
import org.springframework.web.bind.annotation.GetMapping;
|
||||||
|
import org.springframework.web.bind.annotation.PathVariable;
|
||||||
|
import org.springframework.web.bind.annotation.RequestMapping;
|
||||||
|
import org.springframework.web.bind.annotation.RequestParam;
|
||||||
|
import org.springframework.web.bind.annotation.RestController;
|
||||||
|
import org.springframework.web.server.ResponseStatusException;
|
||||||
|
import org.springframework.web.servlet.mvc.method.annotation.SseEmitter;
|
||||||
|
|
||||||
|
/** REST + SSE resource: {@code /api/jobs}. */
|
||||||
|
@RestController
|
||||||
|
@RequestMapping("/api/jobs")
|
||||||
|
@Tag(name = "jobs", description = "Inspect ingestion jobs and stream live progress via SSE.")
|
||||||
|
public class JobController {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(JobController.class);
|
||||||
|
|
||||||
|
private final ObserveJobs observeJobs;
|
||||||
|
|
||||||
|
public JobController(ObserveJobs observeJobs) {
|
||||||
|
this.observeJobs = observeJobs;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "List jobs, optionally filtered by repo / version / status.")
|
||||||
|
@GetMapping
|
||||||
|
public List<JobDto> list(
|
||||||
|
@RequestParam(value = "repoId", required = false) @Nullable String repoId,
|
||||||
|
@RequestParam(value = "versionId", required = false) @Nullable String versionId,
|
||||||
|
@RequestParam(value = "status", required = false) @Nullable JobStatus status,
|
||||||
|
@RequestParam(value = "limit", defaultValue = "100") int limit) {
|
||||||
|
RepositoryId repo = repoId == null ? null : RepositoryId.of(repoId);
|
||||||
|
VersionId ver = versionId == null ? null : VersionId.of(versionId);
|
||||||
|
return observeJobs.listJobs(repo, ver, status, limit).stream()
|
||||||
|
.map(JobDto::of)
|
||||||
|
.toList();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "Get a single job by id.")
|
||||||
|
@GetMapping("/{id}")
|
||||||
|
public JobDto detail(@PathVariable("id") String id) {
|
||||||
|
JobId jobId = JobId.of(id);
|
||||||
|
return observeJobs
|
||||||
|
.findJob(jobId)
|
||||||
|
.map(JobDto::of)
|
||||||
|
.orElseThrow(() ->
|
||||||
|
new ResponseStatusException(org.springframework.http.HttpStatus.NOT_FOUND, "job_not_found"));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "Server-Sent Events stream of log events for a single job.")
|
||||||
|
@GetMapping(value = "/{id}/log", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
|
||||||
|
public SseEmitter logStream(@PathVariable("id") String id) {
|
||||||
|
JobId jobId = JobId.of(id);
|
||||||
|
SseEmitter emitter = new SseEmitter(0L);
|
||||||
|
AutoCloseable subscription = observeJobs.subscribeLogs(jobId, event -> {
|
||||||
|
try {
|
||||||
|
emitter.send(SseEmitter.event().name("log").data(JobLogEventDto.of(event)));
|
||||||
|
} catch (IOException ex) {
|
||||||
|
emitter.completeWithError(ex);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
attachCleanup(emitter, subscription, "job-log " + id);
|
||||||
|
return emitter;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "Server-Sent Events stream of status updates for all jobs.")
|
||||||
|
@GetMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
|
||||||
|
public SseEmitter stream() {
|
||||||
|
SseEmitter emitter = new SseEmitter(0L);
|
||||||
|
|
||||||
|
// Send an immediate ping so Tomcat flushes the response headers to the client.
|
||||||
|
// Without this, the response buffer may not be flushed until the first job event
|
||||||
|
// arrives, keeping the EventSource in CONNECTING state and never firing 'open'.
|
||||||
|
try {
|
||||||
|
emitter.send(SseEmitter.event().name("ping").data(""));
|
||||||
|
} catch (IOException e) {
|
||||||
|
emitter.completeWithError(e);
|
||||||
|
return emitter;
|
||||||
|
}
|
||||||
|
|
||||||
|
AutoCloseable subscription = observeJobs.subscribeJobs(job -> {
|
||||||
|
try {
|
||||||
|
emitter.send(SseEmitter.event().name("job").data(JobDto.of(job)));
|
||||||
|
} catch (IOException ex) {
|
||||||
|
emitter.completeWithError(ex);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// Keepalive: send a ping every 20 s to keep the connection alive through idle
|
||||||
|
// periods and detect disconnected clients promptly.
|
||||||
|
Thread keepalive = Thread.startVirtualThread(() -> {
|
||||||
|
try {
|
||||||
|
while (!Thread.currentThread().isInterrupted()) {
|
||||||
|
Thread.sleep(20_000);
|
||||||
|
emitter.send(SseEmitter.event().name("ping").data(""));
|
||||||
|
}
|
||||||
|
} catch (InterruptedException ignored) {
|
||||||
|
// normal shutdown
|
||||||
|
} catch (Exception ignored) {
|
||||||
|
// emitter already completed or errored; exit
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
Runnable cleanup = () -> {
|
||||||
|
keepalive.interrupt();
|
||||||
|
try {
|
||||||
|
subscription.close();
|
||||||
|
} catch (Exception ex) {
|
||||||
|
log.debug("failed to close SSE subscription job-stream: {}", ex.toString());
|
||||||
|
}
|
||||||
|
};
|
||||||
|
emitter.onCompletion(cleanup);
|
||||||
|
emitter.onTimeout(cleanup);
|
||||||
|
emitter.onError(e -> cleanup.run());
|
||||||
|
return emitter;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void attachCleanup(SseEmitter emitter, AutoCloseable subscription, String label) {
|
||||||
|
Runnable cleanup = () -> {
|
||||||
|
try {
|
||||||
|
subscription.close();
|
||||||
|
} catch (Exception ex) {
|
||||||
|
log.debug("failed to close SSE subscription {}: {}", label, ex.toString());
|
||||||
|
}
|
||||||
|
};
|
||||||
|
emitter.onCompletion(cleanup);
|
||||||
|
emitter.onTimeout(cleanup);
|
||||||
|
emitter.onError(e -> cleanup.run());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,200 @@
|
|||||||
|
package com.trueref.adapter.in.rest;
|
||||||
|
|
||||||
|
import com.trueref.adapter.in.rest.dto.JobDto;
|
||||||
|
import com.trueref.adapter.out.embedding.onnx.OnnxProperties;
|
||||||
|
import com.trueref.domain.model.IngestionJob;
|
||||||
|
import com.trueref.domain.model.JobStatus;
|
||||||
|
import com.trueref.domain.model.Repository;
|
||||||
|
import com.trueref.domain.model.Version;
|
||||||
|
import com.trueref.domain.model.VersionStatus;
|
||||||
|
import com.trueref.domain.port.in.ObserveJobs;
|
||||||
|
import com.trueref.domain.port.in.QueryCatalog;
|
||||||
|
import io.swagger.v3.oas.annotations.Operation;
|
||||||
|
import io.swagger.v3.oas.annotations.tags.Tag;
|
||||||
|
import java.io.BufferedReader;
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.io.InputStreamReader;
|
||||||
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.nio.file.FileVisitResult;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.nio.file.SimpleFileVisitor;
|
||||||
|
import java.nio.file.attribute.BasicFileAttributes;
|
||||||
|
import java.util.EnumMap;
|
||||||
|
import java.util.HashMap;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.beans.factory.annotation.Value;
|
||||||
|
import org.springframework.web.bind.annotation.GetMapping;
|
||||||
|
import org.springframework.web.bind.annotation.RequestMapping;
|
||||||
|
import org.springframework.web.bind.annotation.RestController;
|
||||||
|
|
||||||
|
/** REST resource: {@code /api/observability}. */
|
||||||
|
@RestController
|
||||||
|
@RequestMapping("/api/observability")
|
||||||
|
@Tag(name = "observability", description = "UI-friendly aggregates: metrics + resource usage.")
|
||||||
|
public class ObservabilityController {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(ObservabilityController.class);
|
||||||
|
private static final int JOB_SAMPLE_LIMIT = 10_000;
|
||||||
|
|
||||||
|
private final ObserveJobs observeJobs;
|
||||||
|
private final QueryCatalog queryCatalog;
|
||||||
|
private final Path trueRefHome;
|
||||||
|
private final boolean embedderAvailable;
|
||||||
|
private final boolean rerankerAvailable;
|
||||||
|
private final int gpuDeviceId;
|
||||||
|
|
||||||
|
public ObservabilityController(
|
||||||
|
ObserveJobs observeJobs,
|
||||||
|
QueryCatalog queryCatalog,
|
||||||
|
OnnxProperties onnxProperties,
|
||||||
|
@Value("${trueref.home:./data}") String trueRefHome,
|
||||||
|
@Value("${trueref.embedder.available:true}") boolean embedderAvailable,
|
||||||
|
@Value("${trueref.reranker.available:true}") boolean rerankerAvailable) {
|
||||||
|
this.observeJobs = observeJobs;
|
||||||
|
this.queryCatalog = queryCatalog;
|
||||||
|
this.trueRefHome = Path.of(trueRefHome);
|
||||||
|
this.embedderAvailable = embedderAvailable;
|
||||||
|
this.rerankerAvailable = rerankerAvailable;
|
||||||
|
this.gpuDeviceId = onnxProperties.gpuDeviceId();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "Aggregated metrics for the dashboard (job counts, totals, availability).")
|
||||||
|
@GetMapping("/metrics")
|
||||||
|
public Map<String, Object> metrics() {
|
||||||
|
Map<JobStatus, Long> jobsByStatus = new EnumMap<>(JobStatus.class);
|
||||||
|
for (JobStatus status : JobStatus.values()) {
|
||||||
|
jobsByStatus.put(status, 0L);
|
||||||
|
}
|
||||||
|
List<IngestionJob> jobs = observeJobs.listJobs(null, null, null, JOB_SAMPLE_LIMIT);
|
||||||
|
for (IngestionJob job : jobs) {
|
||||||
|
jobsByStatus.merge(job.status(), 1L, Long::sum);
|
||||||
|
}
|
||||||
|
|
||||||
|
long totalChunks = 0L;
|
||||||
|
long totalVersionsIndexed = 0L;
|
||||||
|
long totalRepos = 0L;
|
||||||
|
for (Repository repo : queryCatalog.listRepositories()) {
|
||||||
|
totalRepos++;
|
||||||
|
for (Version v : queryCatalog.listVersions(repo.id())) {
|
||||||
|
totalChunks += v.chunkCount();
|
||||||
|
if (v.status() == VersionStatus.INDEXED) {
|
||||||
|
totalVersionsIndexed++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Map<String, Object> result = new HashMap<>();
|
||||||
|
result.put("jobsByStatus", toStringKeys(jobsByStatus));
|
||||||
|
result.put("jobsSampled", jobs.size());
|
||||||
|
result.put("jobsSampleLimit", JOB_SAMPLE_LIMIT);
|
||||||
|
result.put("totalRepositories", totalRepos);
|
||||||
|
result.put("totalChunks", totalChunks);
|
||||||
|
result.put("totalVersionsIndexed", totalVersionsIndexed);
|
||||||
|
result.put("embedderAvailable", embedderAvailable);
|
||||||
|
result.put("rerankerAvailable", rerankerAvailable);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "Heap / index-size / cache-size snapshot.")
|
||||||
|
@GetMapping("/resources")
|
||||||
|
public Map<String, Object> resources() {
|
||||||
|
Runtime runtime = Runtime.getRuntime();
|
||||||
|
long heapMax = runtime.maxMemory();
|
||||||
|
long heapTotal = runtime.totalMemory();
|
||||||
|
long heapFree = runtime.freeMemory();
|
||||||
|
long heapUsed = heapTotal - heapFree;
|
||||||
|
|
||||||
|
long luceneBytes = directorySizeBytes(trueRefHome.resolve("lucene"));
|
||||||
|
long cacheBytes = directorySizeBytes(trueRefHome.resolve("embedding-cache"));
|
||||||
|
|
||||||
|
Map<String, Object> heap = new HashMap<>();
|
||||||
|
heap.put("usedBytes", heapUsed);
|
||||||
|
heap.put("totalBytes", heapTotal);
|
||||||
|
heap.put("maxBytes", heapMax);
|
||||||
|
|
||||||
|
Map<String, Object> result = new HashMap<>();
|
||||||
|
result.put("heap", heap);
|
||||||
|
result.put("luceneIndexBytes", luceneBytes);
|
||||||
|
result.put("embeddingCacheBytes", cacheBytes);
|
||||||
|
result.put("trueRefHome", trueRefHome.toAbsolutePath().toString());
|
||||||
|
result.put("gpu", queryGpuMemory(gpuDeviceId));
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Queries {@code nvidia-smi} for memory stats of the given GPU device index.
|
||||||
|
* Returns {@code null} when nvidia-smi is absent or the device index is out of range.
|
||||||
|
*/
|
||||||
|
private @Nullable Map<String, Object> queryGpuMemory(int deviceId) {
|
||||||
|
try {
|
||||||
|
Process proc = new ProcessBuilder(
|
||||||
|
"nvidia-smi",
|
||||||
|
"--query-gpu=memory.used,memory.free,memory.total",
|
||||||
|
"--format=csv,noheader,nounits",
|
||||||
|
"-i",
|
||||||
|
String.valueOf(deviceId))
|
||||||
|
.redirectErrorStream(true)
|
||||||
|
.start();
|
||||||
|
String line;
|
||||||
|
try (BufferedReader br =
|
||||||
|
new BufferedReader(new InputStreamReader(proc.getInputStream(), StandardCharsets.UTF_8))) {
|
||||||
|
line = br.readLine();
|
||||||
|
}
|
||||||
|
int exit = proc.waitFor();
|
||||||
|
if (exit != 0 || line == null || line.isBlank()) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
// Output: "usedMiB, freeMiB, totalMiB"
|
||||||
|
String[] parts = line.split(",");
|
||||||
|
if (parts.length < 3) return null;
|
||||||
|
long usedMiB = Long.parseLong(parts[0].trim());
|
||||||
|
long freeMiB = Long.parseLong(parts[1].trim());
|
||||||
|
long totalMiB = Long.parseLong(parts[2].trim());
|
||||||
|
long mibToBytes = 1024L * 1024L;
|
||||||
|
Map<String, Object> gpu = new HashMap<>();
|
||||||
|
gpu.put("deviceId", deviceId);
|
||||||
|
gpu.put("usedBytes", usedMiB * mibToBytes);
|
||||||
|
gpu.put("freeBytes", freeMiB * mibToBytes);
|
||||||
|
gpu.put("totalBytes", totalMiB * mibToBytes);
|
||||||
|
return gpu;
|
||||||
|
} catch (IOException | InterruptedException | NumberFormatException e) {
|
||||||
|
log.debug("nvidia-smi query failed: {}", e.toString());
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static Map<String, Long> toStringKeys(Map<JobStatus, Long> in) {
|
||||||
|
Map<String, Long> out = new HashMap<>();
|
||||||
|
in.forEach((k, v) -> out.put(k.name(), v));
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static long directorySizeBytes(Path dir) {
|
||||||
|
if (!Files.isDirectory(dir)) {
|
||||||
|
return 0L;
|
||||||
|
}
|
||||||
|
long[] total = {0L};
|
||||||
|
try {
|
||||||
|
Files.walkFileTree(dir, new SimpleFileVisitor<>() {
|
||||||
|
@Override
|
||||||
|
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) {
|
||||||
|
total[0] += attrs.size();
|
||||||
|
return FileVisitResult.CONTINUE;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public FileVisitResult visitFileFailed(Path file, IOException exc) {
|
||||||
|
return FileVisitResult.CONTINUE;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
} catch (IOException e) {
|
||||||
|
log.warn("failed to walk {}: {}", dir, e.toString());
|
||||||
|
}
|
||||||
|
return total[0];
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
package com.trueref.adapter.in.rest;
|
||||||
|
|
||||||
|
import io.swagger.v3.oas.models.OpenAPI;
|
||||||
|
import io.swagger.v3.oas.models.info.Info;
|
||||||
|
import io.swagger.v3.oas.models.servers.Server;
|
||||||
|
import java.util.List;
|
||||||
|
import org.springframework.context.annotation.Bean;
|
||||||
|
import org.springframework.context.annotation.Configuration;
|
||||||
|
|
||||||
|
/** Springdoc OpenAPI customization — title, version, summary and default local server. */
|
||||||
|
@Configuration
|
||||||
|
public class OpenApiConfig {
|
||||||
|
|
||||||
|
@Bean
|
||||||
|
public OpenAPI trueRefOpenApi() {
|
||||||
|
Info info = new Info()
|
||||||
|
.title("trueref API")
|
||||||
|
.version("0.1.0")
|
||||||
|
.summary("Self-hosted Context7-compatible ingestion + retrieval HTTP API.")
|
||||||
|
.description(
|
||||||
|
"REST endpoints for repository registration, ingestion orchestration, hybrid search and library resolution.");
|
||||||
|
Server local = new Server().url("http://localhost:8080").description("Local development server");
|
||||||
|
return new OpenAPI().info(info).servers(List.of(local));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,156 @@
|
|||||||
|
package com.trueref.adapter.in.rest;
|
||||||
|
|
||||||
|
import com.trueref.adapter.in.rest.dto.RegisterRepositoryRequest;
|
||||||
|
import com.trueref.adapter.in.rest.dto.RepositoryDto;
|
||||||
|
import com.trueref.adapter.in.rest.dto.VersionDto;
|
||||||
|
import com.trueref.domain.error.RepositoryNotFound;
|
||||||
|
import com.trueref.domain.error.TagNotFound;
|
||||||
|
import com.trueref.domain.model.Repository;
|
||||||
|
import com.trueref.domain.model.RepositoryId;
|
||||||
|
import com.trueref.domain.model.Version;
|
||||||
|
import com.trueref.domain.port.in.DiscoverVersions;
|
||||||
|
import com.trueref.domain.port.in.IndexVersion;
|
||||||
|
import com.trueref.domain.port.in.QueryCatalog;
|
||||||
|
import com.trueref.domain.port.in.RegisterRepository;
|
||||||
|
import io.swagger.v3.oas.annotations.Operation;
|
||||||
|
import io.swagger.v3.oas.annotations.tags.Tag;
|
||||||
|
import jakarta.validation.Valid;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.Optional;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.http.HttpStatus;
|
||||||
|
import org.springframework.http.ResponseEntity;
|
||||||
|
import org.springframework.web.bind.annotation.DeleteMapping;
|
||||||
|
import org.springframework.web.bind.annotation.GetMapping;
|
||||||
|
import org.springframework.web.bind.annotation.PathVariable;
|
||||||
|
import org.springframework.web.bind.annotation.PostMapping;
|
||||||
|
import org.springframework.web.bind.annotation.RequestBody;
|
||||||
|
import org.springframework.web.bind.annotation.RequestMapping;
|
||||||
|
import org.springframework.web.bind.annotation.ResponseStatus;
|
||||||
|
import org.springframework.web.bind.annotation.RestController;
|
||||||
|
|
||||||
|
/** REST resource: {@code /api/repos}. */
|
||||||
|
@RestController
|
||||||
|
@RequestMapping("/api/repos")
|
||||||
|
@Tag(name = "repositories", description = "Register, list, and index git repositories.")
|
||||||
|
public class RepositoryController {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(RepositoryController.class);
|
||||||
|
|
||||||
|
private final RegisterRepository registerRepository;
|
||||||
|
private final QueryCatalog queryCatalog;
|
||||||
|
private final DiscoverVersions discoverVersions;
|
||||||
|
private final IndexVersion indexVersion;
|
||||||
|
|
||||||
|
public RepositoryController(
|
||||||
|
RegisterRepository registerRepository,
|
||||||
|
QueryCatalog queryCatalog,
|
||||||
|
DiscoverVersions discoverVersions,
|
||||||
|
IndexVersion indexVersion) {
|
||||||
|
this.registerRepository = registerRepository;
|
||||||
|
this.queryCatalog = queryCatalog;
|
||||||
|
this.discoverVersions = discoverVersions;
|
||||||
|
this.indexVersion = indexVersion;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "List all registered repositories.")
|
||||||
|
@GetMapping
|
||||||
|
public List<RepositoryDto> list() {
|
||||||
|
return queryCatalog.listRepositories().stream().map(RepositoryDto::of).toList();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "Register a new repository (remote URL or local path).")
|
||||||
|
@PostMapping
|
||||||
|
@ResponseStatus(HttpStatus.CREATED)
|
||||||
|
public RepositoryDto register(@Valid @RequestBody RegisterRepositoryRequest request) {
|
||||||
|
log.info("registering repository name={}", request.name());
|
||||||
|
Repository repo = registerRepository.register(request.toCommand());
|
||||||
|
return RepositoryDto.of(repo);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "Get details of a single repository.")
|
||||||
|
@GetMapping("/{id}")
|
||||||
|
public RepositoryDto detail(@PathVariable("id") String id) {
|
||||||
|
RepositoryId repoId = parseRepoId(id);
|
||||||
|
return queryCatalog.findRepository(repoId).map(RepositoryDto::of).orElseThrow(() -> new RepositoryNotFound(id));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "Unregister a repository and soft-delete its versions.")
|
||||||
|
@DeleteMapping("/{id}")
|
||||||
|
@ResponseStatus(HttpStatus.NO_CONTENT)
|
||||||
|
public void unregister(@PathVariable("id") String id) {
|
||||||
|
RepositoryId repoId = parseRepoId(id);
|
||||||
|
log.info("unregistering repository id={}", id);
|
||||||
|
registerRepository.unregister(repoId);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "Force tag discovery (git fetch + enumerate tags). Returns the resulting versions.")
|
||||||
|
@PostMapping("/{id}/discover")
|
||||||
|
public List<VersionDto> discover(@PathVariable("id") String id) {
|
||||||
|
RepositoryId repoId = parseRepoId(id);
|
||||||
|
log.info("discovering versions for repository id={}", id);
|
||||||
|
return discoverVersions.discover(repoId).stream().map(VersionDto::of).toList();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "List all versions (git tags) known for this repository.")
|
||||||
|
@GetMapping("/{id}/versions")
|
||||||
|
public List<VersionDto> versions(@PathVariable("id") String id) {
|
||||||
|
RepositoryId repoId = parseRepoId(id);
|
||||||
|
// Ensure 404 if the repo does not exist.
|
||||||
|
queryCatalog.findRepository(repoId).orElseThrow(() -> new RepositoryNotFound(id));
|
||||||
|
return queryCatalog.listVersions(repoId).stream().map(VersionDto::of).toList();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "Enqueue indexing of a specific tag. If the tag is unknown, discovery runs first.")
|
||||||
|
@PostMapping("/{id}/versions/{tag}/index")
|
||||||
|
public ResponseEntity<Map<String, String>> index(
|
||||||
|
@PathVariable("id") String id,
|
||||||
|
@PathVariable("tag") String tag,
|
||||||
|
@RequestBody(required = false) IndexBody body) {
|
||||||
|
boolean force = body != null && Boolean.TRUE.equals(body.force());
|
||||||
|
return enqueueIndex(id, tag, force);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "Force re-indexing of a specific tag (equivalent to index with force=true).")
|
||||||
|
@PostMapping("/{id}/versions/{tag}/reindex")
|
||||||
|
public ResponseEntity<Map<String, String>> reindex(
|
||||||
|
@PathVariable("id") String id, @PathVariable("tag") String tag) {
|
||||||
|
return enqueueIndex(id, tag, true);
|
||||||
|
}
|
||||||
|
|
||||||
|
private ResponseEntity<Map<String, String>> enqueueIndex(String id, String tag, boolean force) {
|
||||||
|
RepositoryId repoId = parseRepoId(id);
|
||||||
|
Repository repo = queryCatalog.findRepository(repoId).orElseThrow(() -> new RepositoryNotFound(id));
|
||||||
|
|
||||||
|
Optional<Version> existing = findByTag(repoId, tag);
|
||||||
|
if (existing.isEmpty()) {
|
||||||
|
log.info("tag {} unknown for repo {}, triggering discovery", tag, repo.name());
|
||||||
|
discoverVersions.discover(repoId);
|
||||||
|
existing = findByTag(repoId, tag);
|
||||||
|
}
|
||||||
|
Version version = existing.orElseThrow(() -> new TagNotFound(repo.name(), tag));
|
||||||
|
|
||||||
|
log.info("enqueueing index job repo={} tag={} force={}", repo.name(), tag, force);
|
||||||
|
var jobId = indexVersion.enqueue(repoId, version.id(), force);
|
||||||
|
return ResponseEntity.accepted().body(Map.of("jobId", jobId.toString()));
|
||||||
|
}
|
||||||
|
|
||||||
|
private Optional<Version> findByTag(RepositoryId repoId, String tag) {
|
||||||
|
return queryCatalog.listVersions(repoId).stream()
|
||||||
|
.filter(v -> v.tag().equals(tag))
|
||||||
|
.findFirst();
|
||||||
|
}
|
||||||
|
|
||||||
|
private static RepositoryId parseRepoId(String id) {
|
||||||
|
try {
|
||||||
|
return RepositoryId.of(id);
|
||||||
|
} catch (IllegalArgumentException e) {
|
||||||
|
throw new RepositoryNotFound(id);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Body of {@code POST /api/repos/{id}/versions/{tag}/index}. */
|
||||||
|
public record IndexBody(Boolean force) {}
|
||||||
|
}
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
package com.trueref.adapter.in.rest;
|
||||||
|
|
||||||
|
import com.trueref.adapter.in.rest.dto.ResolveMatchDto;
|
||||||
|
import com.trueref.adapter.in.rest.dto.ResolveRequest;
|
||||||
|
import com.trueref.adapter.in.rest.dto.ResolveResponse;
|
||||||
|
import com.trueref.domain.port.in.ResolveLibraryId;
|
||||||
|
import io.swagger.v3.oas.annotations.Operation;
|
||||||
|
import io.swagger.v3.oas.annotations.tags.Tag;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
import org.springframework.web.bind.annotation.GetMapping;
|
||||||
|
import org.springframework.web.bind.annotation.RequestMapping;
|
||||||
|
import org.springframework.web.bind.annotation.RequestParam;
|
||||||
|
import org.springframework.web.bind.annotation.RestController;
|
||||||
|
|
||||||
|
/** REST resource: {@code /api/resolve}. */
|
||||||
|
@RestController
|
||||||
|
@RequestMapping("/api/resolve")
|
||||||
|
@Tag(name = "resolve", description = "Turn a fuzzy library name (and optional version) into concrete (repo, version) handles.")
|
||||||
|
public class ResolveController {
|
||||||
|
|
||||||
|
private final ResolveLibraryId resolveLibraryId;
|
||||||
|
|
||||||
|
public ResolveController(ResolveLibraryId resolveLibraryId) {
|
||||||
|
this.resolveLibraryId = resolveLibraryId;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "Preview library ID resolution for the given name / version.")
|
||||||
|
@GetMapping
|
||||||
|
public ResolveResponse resolve(
|
||||||
|
@RequestParam("libraryName") String libraryName,
|
||||||
|
@RequestParam(value = "version", required = false) @Nullable String version,
|
||||||
|
@RequestParam(value = "query", required = false) @Nullable String query) {
|
||||||
|
if (libraryName == null || libraryName.isBlank()) {
|
||||||
|
throw new IllegalArgumentException("libraryName must not be blank");
|
||||||
|
}
|
||||||
|
ResolveRequest req = new ResolveRequest(libraryName, query, version);
|
||||||
|
ResolveLibraryId.Result result = resolveLibraryId.resolve(req.toQuery());
|
||||||
|
return new ResolveResponse(
|
||||||
|
result.matches().stream().map(ResolveMatchDto::of).toList());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
package com.trueref.adapter.in.rest;
|
||||||
|
|
||||||
|
import com.trueref.adapter.in.rest.dto.SearchHitDto;
|
||||||
|
import com.trueref.adapter.in.rest.dto.SearchRequest;
|
||||||
|
import com.trueref.adapter.in.rest.dto.SearchResponse;
|
||||||
|
import com.trueref.domain.port.in.SearchLibraryDocs;
|
||||||
|
import io.swagger.v3.oas.annotations.Operation;
|
||||||
|
import io.swagger.v3.oas.annotations.tags.Tag;
|
||||||
|
import jakarta.validation.Valid;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.web.bind.annotation.PostMapping;
|
||||||
|
import org.springframework.web.bind.annotation.RequestBody;
|
||||||
|
import org.springframework.web.bind.annotation.RequestMapping;
|
||||||
|
import org.springframework.web.bind.annotation.RestController;
|
||||||
|
|
||||||
|
/** REST resource: {@code /api/search}. */
|
||||||
|
@RestController
|
||||||
|
@RequestMapping("/api/search")
|
||||||
|
@Tag(name = "search", description = "Hybrid BM25 + dense search with cross-encoder rerank.")
|
||||||
|
public class SearchController {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(SearchController.class);
|
||||||
|
|
||||||
|
private final SearchLibraryDocs searchLibraryDocs;
|
||||||
|
|
||||||
|
public SearchController(SearchLibraryDocs searchLibraryDocs) {
|
||||||
|
this.searchLibraryDocs = searchLibraryDocs;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Operation(summary = "Hybrid search scoped to one or more (repo, version) pairs.")
|
||||||
|
@PostMapping
|
||||||
|
public SearchResponse search(@Valid @RequestBody SearchRequest request) {
|
||||||
|
log.debug(
|
||||||
|
"search text='{}' topic={} scopes={} tokensBudget={} maxHits={}",
|
||||||
|
request.text(),
|
||||||
|
request.topic(),
|
||||||
|
request.scope().size(),
|
||||||
|
request.tokensBudget(),
|
||||||
|
request.maxHits());
|
||||||
|
SearchLibraryDocs.Result result = searchLibraryDocs.search(request.toQuery());
|
||||||
|
return new SearchResponse(
|
||||||
|
result.hits().stream().map(SearchHitDto::of).toList(),
|
||||||
|
result.totalTokensReturned(),
|
||||||
|
request.topic());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
package com.trueref.adapter.in.rest;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.util.Set;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
import org.springframework.context.annotation.Configuration;
|
||||||
|
import org.springframework.core.io.Resource;
|
||||||
|
import org.springframework.web.servlet.config.annotation.ResourceHandlerRegistry;
|
||||||
|
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
|
||||||
|
import org.springframework.web.servlet.resource.PathResourceResolver;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* SPA fallback: any unmatched request that looks like a client-side route (no file extension in
|
||||||
|
* the final path segment) is served {@code index.html}. API, MCP, springdoc and actuator paths are
|
||||||
|
* explicitly excluded — Spring routes those first anyway, but we exclude them defensively so the
|
||||||
|
* resource resolver does not attempt a fallback for them.
|
||||||
|
*/
|
||||||
|
@Configuration
|
||||||
|
public class WebConfig implements WebMvcConfigurer {
|
||||||
|
|
||||||
|
private static final Set<String> EXCLUDED_PREFIXES =
|
||||||
|
Set.of("api/", "mcp", "swagger-ui/", "v3/api-docs", "actuator/");
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void addResourceHandlers(ResourceHandlerRegistry registry) {
|
||||||
|
registry.addResourceHandler("/**")
|
||||||
|
.addResourceLocations("classpath:/static/")
|
||||||
|
.resourceChain(true)
|
||||||
|
.addResolver(new SpaFallbackResolver());
|
||||||
|
}
|
||||||
|
|
||||||
|
static final class SpaFallbackResolver extends PathResourceResolver {
|
||||||
|
@Override
|
||||||
|
protected @Nullable Resource getResource(String resourcePath, Resource location) throws IOException {
|
||||||
|
for (String prefix : EXCLUDED_PREFIXES) {
|
||||||
|
if (resourcePath.equals(prefix) || resourcePath.startsWith(prefix)) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Resource requested = location.createRelative(resourcePath);
|
||||||
|
if (requested.exists() && requested.isReadable()) {
|
||||||
|
return requested;
|
||||||
|
}
|
||||||
|
// Fallback to index.html only for client-side route-like paths (no extension on last segment).
|
||||||
|
int lastSlash = resourcePath.lastIndexOf('/');
|
||||||
|
String lastSegment = lastSlash < 0 ? resourcePath : resourcePath.substring(lastSlash + 1);
|
||||||
|
if (!lastSegment.isEmpty() && lastSegment.contains(".")) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
Resource indexHtml = location.createRelative("index.html");
|
||||||
|
return indexHtml.exists() && indexHtml.isReadable() ? indexHtml : null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
package com.trueref.adapter.in.rest.dto;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.IngestionJob;
|
||||||
|
import com.trueref.domain.model.JobStatus;
|
||||||
|
import com.trueref.domain.model.JobType;
|
||||||
|
import io.swagger.v3.oas.annotations.media.Schema;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
@Schema(description = "An orchestrated ingestion job with its stages.")
|
||||||
|
public record JobDto(
|
||||||
|
String id,
|
||||||
|
String repoId,
|
||||||
|
@Nullable String versionId,
|
||||||
|
JobType type,
|
||||||
|
JobStatus status,
|
||||||
|
@Nullable Instant startedAt,
|
||||||
|
@Nullable Instant finishedAt,
|
||||||
|
List<JobStageDto> stages) {
|
||||||
|
|
||||||
|
public static JobDto of(IngestionJob j) {
|
||||||
|
return new JobDto(
|
||||||
|
j.id().toString(),
|
||||||
|
j.repoId().toString(),
|
||||||
|
j.versionId() == null ? null : j.versionId().toString(),
|
||||||
|
j.type(),
|
||||||
|
j.status(),
|
||||||
|
j.startedAt(),
|
||||||
|
j.finishedAt(),
|
||||||
|
j.stages().stream().map(JobStageDto::of).toList());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
package com.trueref.adapter.in.rest.dto;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.JobLogEvent;
|
||||||
|
import com.trueref.domain.model.JobStage;
|
||||||
|
import io.swagger.v3.oas.annotations.media.Schema;
|
||||||
|
import java.time.Instant;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
@Schema(description = "A single log event emitted by an ingestion job.")
|
||||||
|
public record JobLogEventDto(
|
||||||
|
String jobId, Instant ts, JobLogEvent.Level level, JobStage.@Nullable StageName stage, String message) {
|
||||||
|
|
||||||
|
public static JobLogEventDto of(JobLogEvent e) {
|
||||||
|
return new JobLogEventDto(e.jobId().toString(), e.ts(), e.level(), e.stage(), e.message());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
package com.trueref.adapter.in.rest.dto;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.JobStage;
|
||||||
|
import io.swagger.v3.oas.annotations.media.Schema;
|
||||||
|
import java.time.Instant;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
@Schema(description = "A single stage within an ingestion job.")
|
||||||
|
public record JobStageDto(
|
||||||
|
String jobId,
|
||||||
|
JobStage.StageName name,
|
||||||
|
JobStage.StageStatus status,
|
||||||
|
@Nullable Instant startedAt,
|
||||||
|
@Nullable Instant finishedAt,
|
||||||
|
long itemsProcessed,
|
||||||
|
long itemsTotal,
|
||||||
|
long bytesProcessed,
|
||||||
|
@Nullable String errorMessage) {
|
||||||
|
|
||||||
|
public static JobStageDto of(JobStage s) {
|
||||||
|
return new JobStageDto(
|
||||||
|
s.jobId().toString(),
|
||||||
|
s.name(),
|
||||||
|
s.status(),
|
||||||
|
s.startedAt(),
|
||||||
|
s.finishedAt(),
|
||||||
|
s.itemsProcessed(),
|
||||||
|
s.itemsTotal(),
|
||||||
|
s.bytesProcessed(),
|
||||||
|
s.errorMessage());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
package com.trueref.adapter.in.rest.dto;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.TagPattern;
|
||||||
|
import com.trueref.domain.port.in.RegisterRepository;
|
||||||
|
import io.swagger.v3.oas.annotations.media.Schema;
|
||||||
|
import jakarta.validation.Valid;
|
||||||
|
import jakarta.validation.constraints.NotBlank;
|
||||||
|
import java.time.Duration;
|
||||||
|
import java.time.format.DateTimeParseException;
|
||||||
|
import java.util.List;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
@Schema(description = "Request payload to register a new repository.")
|
||||||
|
public record RegisterRepositoryRequest(
|
||||||
|
@NotBlank @Schema(description = "Human-readable display name, e.g. spring-projects/spring-boot") String name,
|
||||||
|
@Schema(description = "Remote git URL (mutually exclusive with localPath)") @Nullable String remoteUrl,
|
||||||
|
@Schema(description = "Absolute local path to an already-cloned repo") @Nullable String localPath,
|
||||||
|
@Schema(description = "Per-repo ignore globs, ANDed with .gitignore") @Nullable List<String> ignoreGlobs,
|
||||||
|
@Schema(description = "Max file size in bytes; default 1MiB") @Nullable Long maxFileSizeBytes,
|
||||||
|
@Schema(description = "ISO-8601 duration (e.g. PT1H); 0 disables polling") @Nullable String pollInterval,
|
||||||
|
@Schema(description = "Max most-recent tags auto-indexed") @Nullable Integer tagCap,
|
||||||
|
@Schema(description = "Ordered tag-pattern rules for client version → tag mapping") @Valid @Nullable
|
||||||
|
List<TagPatternDto> versionMappingRules) {
|
||||||
|
|
||||||
|
public RegisterRepository.Command toCommand() {
|
||||||
|
Duration poll = parseDuration(pollInterval);
|
||||||
|
List<String> globs = ignoreGlobs == null ? List.of() : List.copyOf(ignoreGlobs);
|
||||||
|
List<TagPattern> rules = versionMappingRules == null
|
||||||
|
? List.of()
|
||||||
|
: versionMappingRules.stream().map(TagPatternDto::toModel).toList();
|
||||||
|
return new RegisterRepository.Command(
|
||||||
|
name, remoteUrl, localPath, globs, maxFileSizeBytes, poll, tagCap, rules);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static @Nullable Duration parseDuration(@Nullable String iso) {
|
||||||
|
if (iso == null || iso.isBlank()) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
return Duration.parse(iso);
|
||||||
|
} catch (DateTimeParseException e) {
|
||||||
|
throw new IllegalArgumentException("Invalid ISO-8601 duration: " + iso, e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
package com.trueref.adapter.in.rest.dto;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.Repository;
|
||||||
|
import io.swagger.v3.oas.annotations.media.Schema;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
@Schema(description = "A registered repository.")
|
||||||
|
public record RepositoryDto(
|
||||||
|
String id,
|
||||||
|
String name,
|
||||||
|
@Nullable String remoteUrl,
|
||||||
|
String localPath,
|
||||||
|
boolean managedClone,
|
||||||
|
List<String> ignoreGlobs,
|
||||||
|
long maxFileSizeBytes,
|
||||||
|
@Schema(description = "ISO-8601 duration, e.g. PT1H") String pollInterval,
|
||||||
|
int tagCap,
|
||||||
|
List<TagPatternDto> versionMappingRules,
|
||||||
|
Instant createdAt,
|
||||||
|
Instant updatedAt) {
|
||||||
|
|
||||||
|
public static RepositoryDto of(Repository repo) {
|
||||||
|
return new RepositoryDto(
|
||||||
|
repo.id().toString(),
|
||||||
|
repo.name(),
|
||||||
|
repo.remoteUrl(),
|
||||||
|
repo.localPath(),
|
||||||
|
repo.managedClone(),
|
||||||
|
List.copyOf(repo.ignoreGlobs()),
|
||||||
|
repo.maxFileSizeBytes(),
|
||||||
|
repo.pollInterval().toString(),
|
||||||
|
repo.tagCap(),
|
||||||
|
repo.versionMappingRules().stream().map(TagPatternDto::of).toList(),
|
||||||
|
repo.createdAt(),
|
||||||
|
repo.updatedAt());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
package com.trueref.adapter.in.rest.dto;
|
||||||
|
|
||||||
|
import com.trueref.domain.port.in.ResolveLibraryId;
|
||||||
|
import io.swagger.v3.oas.annotations.media.Schema;
|
||||||
|
import java.util.List;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
@Schema(description = "A single candidate library matching a resolve request.")
|
||||||
|
public record ResolveMatchDto(
|
||||||
|
String repoId,
|
||||||
|
String libraryId,
|
||||||
|
String name,
|
||||||
|
@Nullable String description,
|
||||||
|
int snippetCount,
|
||||||
|
List<ResolveVersionRefDto> availableVersions,
|
||||||
|
double score) {
|
||||||
|
|
||||||
|
public static ResolveMatchDto of(ResolveLibraryId.Match m) {
|
||||||
|
return new ResolveMatchDto(
|
||||||
|
m.repoId().toString(),
|
||||||
|
m.libraryId(),
|
||||||
|
m.name(),
|
||||||
|
m.description(),
|
||||||
|
m.snippetCount(),
|
||||||
|
m.availableVersions().stream().map(ResolveVersionRefDto::of).toList(),
|
||||||
|
m.score());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
package com.trueref.adapter.in.rest.dto;
|
||||||
|
|
||||||
|
import com.trueref.domain.port.in.ResolveLibraryId;
|
||||||
|
import io.swagger.v3.oas.annotations.media.Schema;
|
||||||
|
import jakarta.validation.constraints.NotBlank;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
@Schema(description = "Fuzzy library resolution request.")
|
||||||
|
public record ResolveRequest(
|
||||||
|
@NotBlank String libraryName,
|
||||||
|
@Schema(description = "Optional hint to rerank candidates by relevance") @Nullable String query,
|
||||||
|
@Nullable String version) {
|
||||||
|
|
||||||
|
public ResolveLibraryId.Query toQuery() {
|
||||||
|
return new ResolveLibraryId.Query(libraryName, query, version);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
package com.trueref.adapter.in.rest.dto;
|
||||||
|
|
||||||
|
import io.swagger.v3.oas.annotations.media.Schema;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
@Schema(description = "Ranked library matches for a resolve request.")
|
||||||
|
public record ResolveResponse(List<ResolveMatchDto> matches) {}
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
package com.trueref.adapter.in.rest.dto;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.VersionStatus;
|
||||||
|
import com.trueref.domain.port.in.ResolveLibraryId;
|
||||||
|
import io.swagger.v3.oas.annotations.media.Schema;
|
||||||
|
|
||||||
|
@Schema(description = "One available version of a resolved library.")
|
||||||
|
public record ResolveVersionRefDto(String versionId, String tag, VersionStatus status) {
|
||||||
|
|
||||||
|
public static ResolveVersionRefDto of(ResolveLibraryId.VersionRef v) {
|
||||||
|
return new ResolveVersionRefDto(v.versionId().toString(), v.tag(), v.status());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
package com.trueref.adapter.in.rest.dto;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.SearchHit;
|
||||||
|
import io.swagger.v3.oas.annotations.media.Schema;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
@Schema(description = "A single ranked snippet returned by search.")
|
||||||
|
public record SearchHitDto(
|
||||||
|
String chunkId,
|
||||||
|
String repoId,
|
||||||
|
String versionId,
|
||||||
|
String repoName,
|
||||||
|
String tag,
|
||||||
|
String filePath,
|
||||||
|
int startLine,
|
||||||
|
int endLine,
|
||||||
|
String language,
|
||||||
|
@Nullable String symbol,
|
||||||
|
String content,
|
||||||
|
double score) {
|
||||||
|
|
||||||
|
public static SearchHitDto of(SearchHit h) {
|
||||||
|
return new SearchHitDto(
|
||||||
|
h.chunkId().toString(),
|
||||||
|
h.repoId().toString(),
|
||||||
|
h.versionId().toString(),
|
||||||
|
h.repoName(),
|
||||||
|
h.tag(),
|
||||||
|
h.filePath(),
|
||||||
|
h.startLine(),
|
||||||
|
h.endLine(),
|
||||||
|
h.language(),
|
||||||
|
h.symbol(),
|
||||||
|
h.content(),
|
||||||
|
h.score());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
package com.trueref.adapter.in.rest.dto;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.RepositoryId;
|
||||||
|
import com.trueref.domain.model.SearchScope;
|
||||||
|
import com.trueref.domain.model.VersionId;
|
||||||
|
import com.trueref.domain.port.in.SearchLibraryDocs;
|
||||||
|
import io.swagger.v3.oas.annotations.media.Schema;
|
||||||
|
import jakarta.validation.Valid;
|
||||||
|
import jakarta.validation.constraints.NotBlank;
|
||||||
|
import jakarta.validation.constraints.NotEmpty;
|
||||||
|
import jakarta.validation.constraints.Positive;
|
||||||
|
import java.util.List;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
@Schema(description = "Hybrid search request scoped to one or more (repo, version) pairs.")
|
||||||
|
public record SearchRequest(
|
||||||
|
@NotBlank String text,
|
||||||
|
@Nullable String topic,
|
||||||
|
@NotEmpty @Valid List<ScopeRef> scope,
|
||||||
|
@Schema(description = "Token budget; clamped by the service to [500, 50000]") @Positive @Nullable
|
||||||
|
Integer tokensBudget,
|
||||||
|
@Positive @Nullable Integer maxHits) {
|
||||||
|
|
||||||
|
public static final int DEFAULT_TOKENS_BUDGET = 5000;
|
||||||
|
public static final int DEFAULT_MAX_HITS = 20;
|
||||||
|
|
||||||
|
public SearchLibraryDocs.Query toQuery() {
|
||||||
|
List<SearchScope.RepoVersionRef> refs = scope.stream()
|
||||||
|
.map(r -> new SearchScope.RepoVersionRef(RepositoryId.of(r.repoId()), VersionId.of(r.versionId())))
|
||||||
|
.toList();
|
||||||
|
return new SearchLibraryDocs.Query(
|
||||||
|
text,
|
||||||
|
topic,
|
||||||
|
new SearchScope(refs),
|
||||||
|
tokensBudget == null ? DEFAULT_TOKENS_BUDGET : tokensBudget,
|
||||||
|
maxHits == null ? DEFAULT_MAX_HITS : maxHits);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Schema(description = "A (repo, version) pair to scope the search on.")
|
||||||
|
public record ScopeRef(@NotBlank String repoId, @NotBlank String versionId) {}
|
||||||
|
}
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
package com.trueref.adapter.in.rest.dto;
|
||||||
|
|
||||||
|
import io.swagger.v3.oas.annotations.media.Schema;
|
||||||
|
import java.util.List;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
@Schema(description = "Response body for search.")
|
||||||
|
public record SearchResponse(List<SearchHitDto> hits, int totalTokensReturned, @Nullable String topic) {}
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
package com.trueref.adapter.in.rest.dto;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.TagPattern;
|
||||||
|
import io.swagger.v3.oas.annotations.media.Schema;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
/** DTO for {@link TagPattern}. The sealed hierarchy is flattened into a {@code type} + optional {@code template}. */
|
||||||
|
@Schema(description = "Rule mapping a client-supplied version string to a git tag.")
|
||||||
|
public record TagPatternDto(
|
||||||
|
@Schema(
|
||||||
|
description = "Pattern kind",
|
||||||
|
allowableValues = {"EXACT", "V_PREFIX", "RELEASE_PREFIX", "SEMVER_FUZZY", "CUSTOM"})
|
||||||
|
String type,
|
||||||
|
@Schema(description = "Required when type=CUSTOM, e.g. release-{semver}") @Nullable String template) {
|
||||||
|
|
||||||
|
public static TagPatternDto of(TagPattern pattern) {
|
||||||
|
return switch (pattern) {
|
||||||
|
case TagPattern.Exact e -> new TagPatternDto("EXACT", null);
|
||||||
|
case TagPattern.VPrefix v -> new TagPatternDto("V_PREFIX", null);
|
||||||
|
case TagPattern.ReleasePrefix r -> new TagPatternDto("RELEASE_PREFIX", null);
|
||||||
|
case TagPattern.SemverFuzzy s -> new TagPatternDto("SEMVER_FUZZY", null);
|
||||||
|
case TagPattern.Custom c -> new TagPatternDto("CUSTOM", c.template());
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
public TagPattern toModel() {
|
||||||
|
return switch (type) {
|
||||||
|
case "EXACT" -> new TagPattern.Exact();
|
||||||
|
case "V_PREFIX" -> new TagPattern.VPrefix();
|
||||||
|
case "RELEASE_PREFIX" -> new TagPattern.ReleasePrefix();
|
||||||
|
case "SEMVER_FUZZY" -> new TagPattern.SemverFuzzy();
|
||||||
|
case "CUSTOM" -> {
|
||||||
|
if (template == null || template.isBlank()) {
|
||||||
|
throw new IllegalArgumentException("CUSTOM tag pattern requires a template");
|
||||||
|
}
|
||||||
|
yield new TagPattern.Custom(template);
|
||||||
|
}
|
||||||
|
default -> throw new IllegalArgumentException("Unknown tag pattern type: " + type);
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
package com.trueref.adapter.in.rest.dto;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.Version;
|
||||||
|
import com.trueref.domain.model.VersionStatus;
|
||||||
|
import io.swagger.v3.oas.annotations.media.Schema;
|
||||||
|
import java.time.Instant;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
@Schema(description = "A specific git tag (or branch) of a repository.")
|
||||||
|
public record VersionDto(
|
||||||
|
String id,
|
||||||
|
String repoId,
|
||||||
|
String tag,
|
||||||
|
String commitSha,
|
||||||
|
VersionStatus status,
|
||||||
|
@Nullable Instant indexedAt,
|
||||||
|
int chunkCount,
|
||||||
|
@Nullable String errorMessage) {
|
||||||
|
|
||||||
|
public static VersionDto of(Version v) {
|
||||||
|
return new VersionDto(
|
||||||
|
v.id().toString(),
|
||||||
|
v.repoId().toString(),
|
||||||
|
v.tag(),
|
||||||
|
v.commitSha(),
|
||||||
|
v.status(),
|
||||||
|
v.indexedAt(),
|
||||||
|
v.chunkCount(),
|
||||||
|
v.errorMessage());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
/** Transport-layer DTO records used by {@code com.trueref.adapter.in.rest} controllers. */
|
||||||
|
@org.jspecify.annotations.NullMarked
|
||||||
|
package com.trueref.adapter.in.rest.dto;
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
/**
|
||||||
|
* REST driving adapter: controllers, DTOs, OpenAPI configuration, exception handling and SSE
|
||||||
|
* streaming for the trueref HTTP API.
|
||||||
|
*/
|
||||||
|
@org.jspecify.annotations.NullMarked
|
||||||
|
package com.trueref.adapter.in.rest;
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
-- trueref schema V1
|
||||||
|
-- All UUIDs stored as CHAR(36) for H2 portability.
|
||||||
|
|
||||||
|
CREATE TABLE repositories (
|
||||||
|
id CHAR(36) PRIMARY KEY,
|
||||||
|
name VARCHAR(512) NOT NULL UNIQUE,
|
||||||
|
remote_url VARCHAR(2048) NULL,
|
||||||
|
local_path VARCHAR(2048) NOT NULL,
|
||||||
|
managed_clone BOOLEAN NOT NULL,
|
||||||
|
ignore_globs CLOB NOT NULL, -- JSON array of strings
|
||||||
|
max_file_size_bytes BIGINT NOT NULL,
|
||||||
|
poll_interval_seconds BIGINT NOT NULL,
|
||||||
|
tag_cap INT NOT NULL,
|
||||||
|
version_mapping_rules CLOB NOT NULL, -- JSON array of TagPattern
|
||||||
|
created_at TIMESTAMP(9) WITH TIME ZONE NOT NULL,
|
||||||
|
updated_at TIMESTAMP(9) WITH TIME ZONE NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE versions (
|
||||||
|
id CHAR(36) PRIMARY KEY,
|
||||||
|
repo_id CHAR(36) NOT NULL REFERENCES repositories(id) ON DELETE CASCADE,
|
||||||
|
tag VARCHAR(512) NOT NULL,
|
||||||
|
commit_sha CHAR(40) NOT NULL,
|
||||||
|
status VARCHAR(32) NOT NULL,
|
||||||
|
indexed_at TIMESTAMP(9) WITH TIME ZONE NULL,
|
||||||
|
chunk_count INT NOT NULL DEFAULT 0,
|
||||||
|
error_message CLOB NULL,
|
||||||
|
UNIQUE (repo_id, tag)
|
||||||
|
);
|
||||||
|
CREATE INDEX idx_versions_repo_status ON versions(repo_id, status);
|
||||||
|
|
||||||
|
CREATE TABLE ingestion_jobs (
|
||||||
|
id CHAR(36) PRIMARY KEY,
|
||||||
|
repo_id CHAR(36) NOT NULL REFERENCES repositories(id) ON DELETE CASCADE,
|
||||||
|
version_id CHAR(36) NULL REFERENCES versions(id) ON DELETE CASCADE,
|
||||||
|
type VARCHAR(32) NOT NULL,
|
||||||
|
status VARCHAR(32) NOT NULL,
|
||||||
|
started_at TIMESTAMP(9) WITH TIME ZONE NULL,
|
||||||
|
finished_at TIMESTAMP(9) WITH TIME ZONE NULL,
|
||||||
|
created_at TIMESTAMP(9) WITH TIME ZONE NOT NULL
|
||||||
|
);
|
||||||
|
CREATE INDEX idx_jobs_repo_status ON ingestion_jobs(repo_id, status);
|
||||||
|
CREATE INDEX idx_jobs_status_created ON ingestion_jobs(status, created_at);
|
||||||
|
|
||||||
|
CREATE TABLE job_stages (
|
||||||
|
job_id CHAR(36) NOT NULL REFERENCES ingestion_jobs(id) ON DELETE CASCADE,
|
||||||
|
name VARCHAR(32) NOT NULL,
|
||||||
|
status VARCHAR(32) NOT NULL,
|
||||||
|
started_at TIMESTAMP(9) WITH TIME ZONE NULL,
|
||||||
|
finished_at TIMESTAMP(9) WITH TIME ZONE NULL,
|
||||||
|
items_processed BIGINT NOT NULL DEFAULT 0,
|
||||||
|
items_total BIGINT NOT NULL DEFAULT 0,
|
||||||
|
bytes_processed BIGINT NOT NULL DEFAULT 0,
|
||||||
|
error_message CLOB NULL,
|
||||||
|
PRIMARY KEY (job_id, name)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Persisted log buffer (last N per job kept by application logic; SSE streams from in-memory bus).
|
||||||
|
CREATE TABLE job_log_events (
|
||||||
|
id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
||||||
|
job_id CHAR(36) NOT NULL REFERENCES ingestion_jobs(id) ON DELETE CASCADE,
|
||||||
|
ts TIMESTAMP(9) WITH TIME ZONE NOT NULL,
|
||||||
|
level VARCHAR(8) NOT NULL,
|
||||||
|
stage VARCHAR(32),
|
||||||
|
message CLOB NOT NULL
|
||||||
|
);
|
||||||
|
CREATE INDEX idx_job_log_job_ts ON job_log_events(job_id, ts);
|
||||||
28
trueref-application/pom.xml
Normal file
28
trueref-application/pom.xml
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||||
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||||
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||||
|
<modelVersion>4.0.0</modelVersion>
|
||||||
|
|
||||||
|
<parent>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-parent</artifactId>
|
||||||
|
<version>0.1.0-SNAPSHOT</version>
|
||||||
|
</parent>
|
||||||
|
|
||||||
|
<artifactId>trueref-application</artifactId>
|
||||||
|
<name>trueref-application</name>
|
||||||
|
<description>Use-case implementations. Depends only on the domain.</description>
|
||||||
|
|
||||||
|
<dependencies>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-domain</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<!-- SLF4J only; orchestration may use virtual threads via JDK -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.slf4j</groupId>
|
||||||
|
<artifactId>slf4j-api</artifactId>
|
||||||
|
</dependency>
|
||||||
|
</dependencies>
|
||||||
|
</project>
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
package com.trueref.application.catalog;
|
||||||
|
|
||||||
|
import com.trueref.domain.error.RepositoryAlreadyRegistered;
|
||||||
|
import com.trueref.domain.error.RepositoryNotFound;
|
||||||
|
import com.trueref.domain.model.Repository;
|
||||||
|
import com.trueref.domain.model.RepositoryId;
|
||||||
|
import com.trueref.domain.model.TagPattern;
|
||||||
|
import com.trueref.domain.model.Version;
|
||||||
|
import com.trueref.domain.port.in.QueryCatalog;
|
||||||
|
import com.trueref.domain.port.in.RegisterRepository;
|
||||||
|
import com.trueref.domain.port.out.RepositoryStore;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Duration;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Optional;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
/** Implements {@link RegisterRepository} and {@link QueryCatalog}. */
|
||||||
|
public final class CatalogService implements RegisterRepository, QueryCatalog {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(CatalogService.class);
|
||||||
|
|
||||||
|
private static final List<TagPattern> DEFAULT_RULES = List.of(
|
||||||
|
new TagPattern.Exact(),
|
||||||
|
new TagPattern.VPrefix(),
|
||||||
|
new TagPattern.ReleasePrefix(),
|
||||||
|
new TagPattern.SemverFuzzy());
|
||||||
|
|
||||||
|
private final RepositoryStore store;
|
||||||
|
private final Path trueRefHome;
|
||||||
|
|
||||||
|
public CatalogService(RepositoryStore store, Path trueRefHome) {
|
||||||
|
this.store = store;
|
||||||
|
this.trueRefHome = trueRefHome;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Repository register(Command cmd) {
|
||||||
|
store.findByName(cmd.name()).ifPresent(r -> {
|
||||||
|
throw new RepositoryAlreadyRegistered(cmd.name());
|
||||||
|
});
|
||||||
|
boolean managed = cmd.remoteUrl() != null && cmd.localPath() == null;
|
||||||
|
String localPath = cmd.localPath() != null
|
||||||
|
? cmd.localPath()
|
||||||
|
: trueRefHome.resolve("repos").resolve(cmd.name().replace('/', '_')).toString();
|
||||||
|
Instant now = Instant.now();
|
||||||
|
Repository repo = new Repository(
|
||||||
|
RepositoryId.random(),
|
||||||
|
cmd.name(),
|
||||||
|
cmd.remoteUrl(),
|
||||||
|
localPath,
|
||||||
|
managed,
|
||||||
|
cmd.ignoreGlobs(),
|
||||||
|
cmd.maxFileSizeBytes() == null ? 1_048_576L : cmd.maxFileSizeBytes(),
|
||||||
|
cmd.pollInterval() == null ? Duration.ofHours(1) : cmd.pollInterval(),
|
||||||
|
cmd.tagCap() == null ? 100 : cmd.tagCap(),
|
||||||
|
cmd.versionMappingRules().isEmpty() ? DEFAULT_RULES : cmd.versionMappingRules(),
|
||||||
|
now,
|
||||||
|
now);
|
||||||
|
Repository saved = store.save(repo);
|
||||||
|
log.info("registered repository name={} id={} managed={} localPath={}",
|
||||||
|
saved.name(), saved.id(), managed, localPath);
|
||||||
|
return saved;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void unregister(RepositoryId id) {
|
||||||
|
Repository existing = store.findById(id).orElseThrow(() -> new RepositoryNotFound(id.toString()));
|
||||||
|
store.delete(id);
|
||||||
|
log.info("unregistered repository name={} id={}", existing.name(), id);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public List<Repository> listRepositories() {
|
||||||
|
return store.findAll();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Optional<Repository> findRepository(RepositoryId id) {
|
||||||
|
return store.findById(id);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public List<Version> listVersions(RepositoryId repoId) {
|
||||||
|
return store.findVersionsByRepo(repoId);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
package com.trueref.application.ingest;
|
||||||
|
|
||||||
|
import com.trueref.domain.error.RepositoryNotFound;
|
||||||
|
import com.trueref.domain.model.Repository;
|
||||||
|
import com.trueref.domain.model.Version;
|
||||||
|
import com.trueref.domain.model.VersionId;
|
||||||
|
import com.trueref.domain.model.VersionStatus;
|
||||||
|
import com.trueref.domain.port.in.DiscoverVersions;
|
||||||
|
import com.trueref.domain.port.out.GitClient;
|
||||||
|
import com.trueref.domain.port.out.RepositoryStore;
|
||||||
|
import com.trueref.domain.model.RepositoryId;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Optional;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
/** Fetches tags (git fetch + tag list) and persists new/updated {@link Version}s. */
|
||||||
|
public final class DiscoveryService implements DiscoverVersions {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(DiscoveryService.class);
|
||||||
|
|
||||||
|
private final RepositoryStore store;
|
||||||
|
private final GitClient git;
|
||||||
|
|
||||||
|
public DiscoveryService(RepositoryStore store, GitClient git) {
|
||||||
|
this.store = store;
|
||||||
|
this.git = git;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public List<Version> discover(RepositoryId repoId) {
|
||||||
|
Repository repo = store.findById(repoId).orElseThrow(() -> new RepositoryNotFound(repoId.toString()));
|
||||||
|
Path path = Path.of(repo.localPath());
|
||||||
|
|
||||||
|
// clone if managed and not present
|
||||||
|
if (repo.managedClone() && !Files.exists(path.resolve(".git"))) {
|
||||||
|
log.info("cloning for discovery: {}", repo.name());
|
||||||
|
git.cloneRepo(repo.remoteUrl(), path);
|
||||||
|
} else {
|
||||||
|
try { git.fetch(path); } catch (Exception e) {
|
||||||
|
log.warn("fetch failed for {}: {}", repo.name(), e.toString());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
List<GitClient.TagInfo> tags = git.listTags(path);
|
||||||
|
// apply tag cap: keep top-N by epoch DESC (already sorted)
|
||||||
|
List<GitClient.TagInfo> capped = tags.stream().limit(Math.max(1, repo.tagCap())).toList();
|
||||||
|
|
||||||
|
for (GitClient.TagInfo t : capped) {
|
||||||
|
Optional<Version> existing = store.findVersionByTag(repoId, t.name());
|
||||||
|
if (existing.isPresent()) {
|
||||||
|
// refresh commit sha only if changed
|
||||||
|
if (!existing.get().commitSha().equalsIgnoreCase(t.commitSha())) {
|
||||||
|
Version updated = new Version(
|
||||||
|
existing.get().id(),
|
||||||
|
existing.get().repoId(),
|
||||||
|
t.name(),
|
||||||
|
t.commitSha(),
|
||||||
|
VersionStatus.DISCOVERED, // needs re-index
|
||||||
|
existing.get().indexedAt(),
|
||||||
|
existing.get().chunkCount(),
|
||||||
|
null);
|
||||||
|
store.saveVersion(updated);
|
||||||
|
log.info("tag {} changed commit; marked DISCOVERED", t.name());
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
Version v = new Version(
|
||||||
|
VersionId.random(),
|
||||||
|
repoId,
|
||||||
|
t.name(),
|
||||||
|
t.commitSha(),
|
||||||
|
VersionStatus.DISCOVERED,
|
||||||
|
null,
|
||||||
|
0,
|
||||||
|
null);
|
||||||
|
store.saveVersion(v);
|
||||||
|
log.info("discovered new tag {}", t.name());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return store.findVersionsByRepo(repoId).stream()
|
||||||
|
.sorted(Comparator.comparing(Version::tag))
|
||||||
|
.toList();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,604 @@
|
|||||||
|
package com.trueref.application.ingest;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.Chunk;
|
||||||
|
import com.trueref.domain.model.ChunkId;
|
||||||
|
import com.trueref.domain.model.ChunkVersion;
|
||||||
|
import com.trueref.domain.model.Embedding;
|
||||||
|
import com.trueref.domain.model.IngestionJob;
|
||||||
|
import com.trueref.domain.model.JobId;
|
||||||
|
import com.trueref.domain.model.JobLogEvent;
|
||||||
|
import com.trueref.domain.model.JobStage;
|
||||||
|
import com.trueref.domain.model.JobStatus;
|
||||||
|
import com.trueref.domain.model.JobType;
|
||||||
|
import com.trueref.domain.model.Repository;
|
||||||
|
import com.trueref.domain.model.RepositoryId;
|
||||||
|
import com.trueref.domain.model.Version;
|
||||||
|
import com.trueref.domain.model.VersionId;
|
||||||
|
import com.trueref.domain.model.VersionStatus;
|
||||||
|
import com.trueref.domain.port.in.IndexVersion;
|
||||||
|
import com.trueref.domain.port.out.ChunkStore;
|
||||||
|
import com.trueref.domain.port.out.CodeParser;
|
||||||
|
import com.trueref.domain.port.out.EmbeddingCache;
|
||||||
|
import com.trueref.domain.port.out.EmbeddingService;
|
||||||
|
import com.trueref.domain.port.out.GitClient;
|
||||||
|
import com.trueref.domain.port.out.JobEventBus;
|
||||||
|
import com.trueref.domain.port.out.JobStore;
|
||||||
|
import com.trueref.domain.port.out.RepositoryStore;
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.nio.file.FileSystems;
|
||||||
|
import java.nio.file.FileVisitResult;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.nio.file.PathMatcher;
|
||||||
|
import java.nio.file.SimpleFileVisitor;
|
||||||
|
import java.nio.file.attribute.BasicFileAttributes;
|
||||||
|
import java.security.MessageDigest;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.Arrays;
|
||||||
|
import java.util.Collection;
|
||||||
|
import java.util.HashMap;
|
||||||
|
import java.util.HashSet;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.Set;
|
||||||
|
import java.util.concurrent.BlockingQueue;
|
||||||
|
import java.util.concurrent.ConcurrentHashMap;
|
||||||
|
import java.util.concurrent.ExecutorService;
|
||||||
|
import java.util.concurrent.Executors;
|
||||||
|
import java.util.concurrent.LinkedBlockingQueue;
|
||||||
|
import java.util.concurrent.Semaphore;
|
||||||
|
import java.util.concurrent.TimeUnit;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.slf4j.MDC;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Orchestrates the full ingestion pipeline for one (repo, version): clone/fetch → checkout →
|
||||||
|
* discover files → diff-vs-parent → parse → chunk → dedupe by hash → embed (with cache) → index
|
||||||
|
* into Lucene → commit.
|
||||||
|
*
|
||||||
|
* <p>The pipeline is split into two concurrent stages:
|
||||||
|
* <ol>
|
||||||
|
* <li><b>Parse phase</b> (virtual threads, up to {@code maxParseJobs} in parallel):
|
||||||
|
* FETCH/CLONE → CHECKOUT → DISCOVER_FILES → DIFF_FILES → PARSE.
|
||||||
|
* I/O-bound; no GPU use; worktree is removed immediately after parse.
|
||||||
|
* </li>
|
||||||
|
* <li><b>Embed phase</b> (single dedicated platform thread):
|
||||||
|
* EMBED → INDEX → COMMIT. GPU-bound; serialises ONNX inference to prevent CUDA
|
||||||
|
* context races. Runs on a platform thread for a stable OS thread identity.
|
||||||
|
* </li>
|
||||||
|
* </ol>
|
||||||
|
* Completed parse batches are handed off via a bounded {@link BlockingQueue}: if the embed
|
||||||
|
* worker is busy, parse workers block before queuing, naturally capping in-memory pressure.
|
||||||
|
* One orchestrator instance is shared across all jobs.
|
||||||
|
*/
|
||||||
|
public final class IngestionOrchestrator implements IndexVersion {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(IngestionOrchestrator.class);
|
||||||
|
|
||||||
|
// Built-in ignore globs (applied in addition to .gitignore + per-repo globs).
|
||||||
|
private static final List<String> BUILTIN_IGNORES = List.of(
|
||||||
|
"**/.git/**",
|
||||||
|
"**/node_modules/**",
|
||||||
|
"**/target/**",
|
||||||
|
"**/build/**",
|
||||||
|
"**/dist/**",
|
||||||
|
"**/out/**",
|
||||||
|
"**/.idea/**",
|
||||||
|
"**/.vscode/**",
|
||||||
|
"**/__pycache__/**",
|
||||||
|
"**/*.png", "**/*.jpg", "**/*.jpeg", "**/*.gif", "**/*.webp", "**/*.ico",
|
||||||
|
"**/*.pdf", "**/*.zip", "**/*.tar", "**/*.gz", "**/*.jar", "**/*.class",
|
||||||
|
"**/*.so", "**/*.dll", "**/*.dylib", "**/*.exe", "**/*.bin");
|
||||||
|
|
||||||
|
private final RepositoryStore repoStore;
|
||||||
|
private final JobStore jobStore;
|
||||||
|
private final ChunkStore chunkStore;
|
||||||
|
private final EmbeddingService embeddings;
|
||||||
|
private final EmbeddingCache embeddingCache;
|
||||||
|
private final GitClient git;
|
||||||
|
private final CodeParser parser;
|
||||||
|
private final JobEventBus bus;
|
||||||
|
|
||||||
|
private final ExecutorService parseExecutor;
|
||||||
|
private final Semaphore parseConcurrencyLimit;
|
||||||
|
private final BlockingQueue<ParsedBatch> embedQueue;
|
||||||
|
private final Thread embedWorker;
|
||||||
|
private volatile boolean shuttingDown = false;
|
||||||
|
private final Map<JobId, Boolean> running = new ConcurrentHashMap<>();
|
||||||
|
|
||||||
|
public IngestionOrchestrator(
|
||||||
|
RepositoryStore repoStore,
|
||||||
|
JobStore jobStore,
|
||||||
|
ChunkStore chunkStore,
|
||||||
|
EmbeddingService embeddings,
|
||||||
|
EmbeddingCache embeddingCache,
|
||||||
|
GitClient git,
|
||||||
|
CodeParser parser,
|
||||||
|
JobEventBus bus,
|
||||||
|
int maxParseJobs,
|
||||||
|
int embedQueueCapacity) {
|
||||||
|
this.repoStore = repoStore;
|
||||||
|
this.jobStore = jobStore;
|
||||||
|
this.chunkStore = chunkStore;
|
||||||
|
this.embeddings = embeddings;
|
||||||
|
this.embeddingCache = embeddingCache;
|
||||||
|
this.git = git;
|
||||||
|
this.parser = parser;
|
||||||
|
this.bus = bus;
|
||||||
|
this.parseExecutor = Executors.newVirtualThreadPerTaskExecutor();
|
||||||
|
// Fair semaphore caps parallel parse jobs (I/O + CPU heavy, no GPU).
|
||||||
|
this.parseConcurrencyLimit = new Semaphore(Math.max(1, maxParseJobs), true);
|
||||||
|
// Bounded queue between parse workers and the embed worker.
|
||||||
|
// Backpressure: parse workers block here when the embed worker is saturated,
|
||||||
|
// preventing unbounded in-memory accumulation of parsed chunks.
|
||||||
|
this.embedQueue = new LinkedBlockingQueue<>(Math.max(1, embedQueueCapacity));
|
||||||
|
// Single platform thread for GPU inference. Platform (not virtual) gives a
|
||||||
|
// stable OS thread identity for CUDA — the synchronized(session) in OnnxEmbeddingService
|
||||||
|
// already pins virtual threads, but a dedicated platform thread removes all doubt.
|
||||||
|
this.embedWorker = Thread.ofPlatform()
|
||||||
|
.name("embed-worker")
|
||||||
|
.daemon(false)
|
||||||
|
.start(this::drainEmbedQueue);
|
||||||
|
log.info("IngestionOrchestrator ready: maxParseJobs={} embedQueueCapacity={}",
|
||||||
|
Math.max(1, maxParseJobs), Math.max(1, embedQueueCapacity));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public JobId enqueue(RepositoryId repoId, VersionId versionId, boolean force) {
|
||||||
|
Repository repo = repoStore.findById(repoId).orElseThrow();
|
||||||
|
Version ver = repoStore.findVersion(versionId).orElseThrow();
|
||||||
|
if (!force && ver.status() == VersionStatus.INDEXED) {
|
||||||
|
log.info("version already indexed and not forcing; skipping repo={} tag={}", repo.name(), ver.tag());
|
||||||
|
JobId id = JobId.random();
|
||||||
|
IngestionJob skipped = new IngestionJob(
|
||||||
|
id,
|
||||||
|
repoId,
|
||||||
|
versionId,
|
||||||
|
JobType.INDEX_VERSION,
|
||||||
|
JobStatus.SUCCEEDED,
|
||||||
|
Instant.now(),
|
||||||
|
Instant.now(),
|
||||||
|
List.of());
|
||||||
|
jobStore.save(skipped);
|
||||||
|
bus.publishJob(skipped);
|
||||||
|
return id;
|
||||||
|
}
|
||||||
|
|
||||||
|
JobId jobId = JobId.random();
|
||||||
|
IngestionJob job = new IngestionJob(
|
||||||
|
jobId,
|
||||||
|
repoId,
|
||||||
|
versionId,
|
||||||
|
JobType.INDEX_VERSION,
|
||||||
|
JobStatus.QUEUED,
|
||||||
|
null,
|
||||||
|
null,
|
||||||
|
List.of());
|
||||||
|
jobStore.save(job);
|
||||||
|
bus.publishJob(job);
|
||||||
|
running.put(jobId, Boolean.TRUE);
|
||||||
|
parseExecutor.submit(() -> runParsePhase(jobId, repo, ver));
|
||||||
|
return jobId;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Carry struct that transfers parse-phase output to the embed worker.
|
||||||
|
* The git worktree has already been removed before this batch enters the queue;
|
||||||
|
* only in-memory chunk data travels across the thread boundary.
|
||||||
|
*/
|
||||||
|
private record ParsedBatch(
|
||||||
|
JobId jobId, Repository repo, Version ver, List<ParsedPiece> pieces) {}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Parse phase — runs on a virtual thread, up to {@code maxParseJobs} in parallel.
|
||||||
|
* Stages: FETCH/CLONE → CHECKOUT → DISCOVER_FILES → DIFF_FILES → PARSE.
|
||||||
|
* On completion, removes the worktree, releases the parse slot so the next job can
|
||||||
|
* start immediately, then blocks on {@link #embedQueue} until the embed worker has
|
||||||
|
* room (natural backpressure).
|
||||||
|
*/
|
||||||
|
private void runParsePhase(JobId jobId, Repository repo, Version ver) {
|
||||||
|
MDC.put("jobId", jobId.toString());
|
||||||
|
MDC.put("repo", repo.name());
|
||||||
|
MDC.put("tag", ver.tag());
|
||||||
|
try {
|
||||||
|
parseConcurrencyLimit.acquire();
|
||||||
|
} catch (InterruptedException e) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
log.warn("job {} interrupted while waiting for parse slot — failing", jobId);
|
||||||
|
repoStore.updateVersionStatus(ver.id(), VersionStatus.FAILED, "interrupted");
|
||||||
|
transitionJob(jobId, JobStatus.FAILED, null, Instant.now());
|
||||||
|
running.remove(jobId);
|
||||||
|
MDC.clear();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
boolean slotReleased = false;
|
||||||
|
Path worktree = null;
|
||||||
|
Path repoPath = Path.of(repo.localPath());
|
||||||
|
try {
|
||||||
|
transitionJob(jobId, JobStatus.RUNNING, Instant.now(), null);
|
||||||
|
repoStore.updateVersionStatus(ver.id(), VersionStatus.INDEXING, null);
|
||||||
|
|
||||||
|
// STAGE: FETCH (or CLONE if managed and absent)
|
||||||
|
if (repo.managedClone() && !Files.exists(repoPath.resolve(".git"))) {
|
||||||
|
stage(jobId, JobStage.StageName.CLONE, () -> {
|
||||||
|
logEvent(jobId, JobLogEvent.Level.INFO, JobStage.StageName.CLONE,
|
||||||
|
"cloning " + repo.remoteUrl() + " → " + repoPath);
|
||||||
|
git.cloneRepo(repo.remoteUrl(), repoPath);
|
||||||
|
return 1L;
|
||||||
|
});
|
||||||
|
} else {
|
||||||
|
stage(jobId, JobStage.StageName.FETCH, () -> {
|
||||||
|
git.fetch(repoPath);
|
||||||
|
return 1L;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// STAGE: CHECKOUT
|
||||||
|
final Path wt = stageReturning(jobId, JobStage.StageName.CHECKOUT, () -> {
|
||||||
|
Path w = git.checkoutWorktree(repoPath, ver.tag());
|
||||||
|
logEvent(jobId, JobLogEvent.Level.INFO, JobStage.StageName.CHECKOUT,
|
||||||
|
"checked out at " + w);
|
||||||
|
return w;
|
||||||
|
});
|
||||||
|
worktree = wt;
|
||||||
|
|
||||||
|
// STAGE: DISCOVER_FILES
|
||||||
|
List<Path> files = stageReturning(jobId, JobStage.StageName.DISCOVER_FILES, () ->
|
||||||
|
discoverFiles(wt, repo));
|
||||||
|
logEvent(jobId, JobLogEvent.Level.INFO, JobStage.StageName.DISCOVER_FILES,
|
||||||
|
"found " + files.size() + " indexable files");
|
||||||
|
|
||||||
|
// STAGE: DIFF_FILES (select subset)
|
||||||
|
String baseRef = pickParentIndexedTag(repo, ver);
|
||||||
|
final List<Path> selectedFiles;
|
||||||
|
if (baseRef != null) {
|
||||||
|
Set<String> changedRel = stageReturning(jobId, JobStage.StageName.DIFF_FILES, () -> {
|
||||||
|
List<GitClient.DiffEntry> diff = git.diff(repoPath, baseRef, ver.tag());
|
||||||
|
Set<String> s = new HashSet<>();
|
||||||
|
for (GitClient.DiffEntry e : diff) {
|
||||||
|
if (e.change() != GitClient.DiffEntry.ChangeType.DELETED) s.add(e.path());
|
||||||
|
}
|
||||||
|
return s;
|
||||||
|
});
|
||||||
|
selectedFiles = files.stream()
|
||||||
|
.filter(f -> changedRel.contains(wt.relativize(f).toString().replace('\\', '/')))
|
||||||
|
.toList();
|
||||||
|
logEvent(jobId, JobLogEvent.Level.INFO, JobStage.StageName.DIFF_FILES,
|
||||||
|
"diff vs " + baseRef + " selects " + selectedFiles.size() + "/" + files.size());
|
||||||
|
} else {
|
||||||
|
selectedFiles = files;
|
||||||
|
}
|
||||||
|
|
||||||
|
// STAGE: PARSE + CHUNK + HASH (combined)
|
||||||
|
List<ParsedPiece> pieces = stageReturning(jobId, JobStage.StageName.PARSE, () ->
|
||||||
|
parseAll(selectedFiles, wt));
|
||||||
|
|
||||||
|
// Worktree no longer needed — free disk space before blocking on embed queue.
|
||||||
|
removeWorktreeQuietly(jobId, repoPath, wt);
|
||||||
|
worktree = null;
|
||||||
|
|
||||||
|
// Release parse slot before blocking so the next job can start parsing
|
||||||
|
// while this batch waits for the embed worker (maximises CPU/GPU overlap).
|
||||||
|
parseConcurrencyLimit.release();
|
||||||
|
slotReleased = true;
|
||||||
|
|
||||||
|
// Hand off to embed worker — blocks if the queue is at capacity.
|
||||||
|
embedQueue.put(new ParsedBatch(jobId, repo, ver, pieces));
|
||||||
|
|
||||||
|
} catch (InterruptedException e) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
log.warn("job {} interrupted during parse — failing", jobId);
|
||||||
|
repoStore.updateVersionStatus(ver.id(), VersionStatus.FAILED, "interrupted");
|
||||||
|
transitionJob(jobId, JobStatus.FAILED, null, Instant.now());
|
||||||
|
running.remove(jobId);
|
||||||
|
} catch (Exception e) {
|
||||||
|
log.error("parse phase failed for job {}", jobId, e);
|
||||||
|
logEvent(jobId, JobLogEvent.Level.ERROR, null, "parse phase failed: " + e.getMessage());
|
||||||
|
repoStore.updateVersionStatus(ver.id(), VersionStatus.FAILED, e.getMessage());
|
||||||
|
transitionJob(jobId, JobStatus.FAILED, null, Instant.now());
|
||||||
|
running.remove(jobId);
|
||||||
|
} finally {
|
||||||
|
if (worktree != null) removeWorktreeQuietly(jobId, repoPath, worktree);
|
||||||
|
if (!slotReleased) parseConcurrencyLimit.release();
|
||||||
|
MDC.clear();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Embed worker — runs on a single dedicated platform thread.
|
||||||
|
* Drains {@link #embedQueue} until {@link #shutdown()} signals stop.
|
||||||
|
* Stages per batch: EMBED → INDEX → COMMIT → mark version indexed → transition SUCCEEDED.
|
||||||
|
*/
|
||||||
|
private void drainEmbedQueue() {
|
||||||
|
log.info("embed worker started ({})", Thread.currentThread().getName());
|
||||||
|
while (!shuttingDown || !embedQueue.isEmpty()) {
|
||||||
|
ParsedBatch batch;
|
||||||
|
try {
|
||||||
|
batch = embedQueue.poll(500, TimeUnit.MILLISECONDS);
|
||||||
|
} catch (InterruptedException e) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if (batch == null) continue;
|
||||||
|
runEmbedPhase(batch);
|
||||||
|
}
|
||||||
|
log.info("embed worker stopped");
|
||||||
|
}
|
||||||
|
|
||||||
|
private void runEmbedPhase(ParsedBatch batch) {
|
||||||
|
MDC.put("jobId", batch.jobId().toString());
|
||||||
|
MDC.put("repo", batch.repo().name());
|
||||||
|
MDC.put("tag", batch.ver().tag());
|
||||||
|
try {
|
||||||
|
// STAGE: EMBED
|
||||||
|
List<Chunk> chunks = stageReturning(batch.jobId(), JobStage.StageName.EMBED, () ->
|
||||||
|
embedAll(batch.jobId(), batch.pieces()));
|
||||||
|
|
||||||
|
// STAGE: INDEX
|
||||||
|
stage(batch.jobId(), JobStage.StageName.INDEX, () -> {
|
||||||
|
chunkStore.unlinkVersion(batch.ver().id());
|
||||||
|
List<ChunkVersion> links = buildLinks(batch.ver().id(), batch.pieces());
|
||||||
|
chunkStore.linkChunks(links);
|
||||||
|
return (long) links.size();
|
||||||
|
});
|
||||||
|
|
||||||
|
// STAGE: COMMIT
|
||||||
|
stage(batch.jobId(), JobStage.StageName.COMMIT, () -> {
|
||||||
|
chunkStore.commit();
|
||||||
|
return 1L;
|
||||||
|
});
|
||||||
|
|
||||||
|
repoStore.markVersionIndexed(batch.ver().id(), batch.pieces().size());
|
||||||
|
transitionJob(batch.jobId(), JobStatus.SUCCEEDED, null, Instant.now());
|
||||||
|
logEvent(batch.jobId(), JobLogEvent.Level.INFO, null,
|
||||||
|
"indexed " + chunks.size() + " chunks across " + batch.pieces().size() + " pieces");
|
||||||
|
} catch (Exception e) {
|
||||||
|
log.error("embed phase failed for job {}", batch.jobId(), e);
|
||||||
|
logEvent(batch.jobId(), JobLogEvent.Level.ERROR, null,
|
||||||
|
"embed phase failed: " + e.getMessage());
|
||||||
|
repoStore.updateVersionStatus(batch.ver().id(), VersionStatus.FAILED, e.getMessage());
|
||||||
|
transitionJob(batch.jobId(), JobStatus.FAILED, null, Instant.now());
|
||||||
|
} finally {
|
||||||
|
running.remove(batch.jobId());
|
||||||
|
MDC.clear();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void removeWorktreeQuietly(JobId jobId, Path repoPath, Path worktree) {
|
||||||
|
try {
|
||||||
|
git.removeWorktree(repoPath, worktree);
|
||||||
|
} catch (Exception e) {
|
||||||
|
logEvent(jobId, JobLogEvent.Level.WARN, null, "worktree cleanup failed: " + e.getMessage());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Orderly shutdown: stops the parse executor, signals the embed worker to stop after
|
||||||
|
* finishing its current batch, then fails any batches still in the queue.
|
||||||
|
* Called by Spring via {@code @Bean(destroyMethod = "shutdown")} in ApplicationBeans.
|
||||||
|
*/
|
||||||
|
void shutdown() {
|
||||||
|
log.info("IngestionOrchestrator shutting down — stopping embed worker");
|
||||||
|
shuttingDown = true;
|
||||||
|
parseExecutor.shutdownNow();
|
||||||
|
embedWorker.interrupt();
|
||||||
|
try {
|
||||||
|
embedWorker.join(10_000);
|
||||||
|
} catch (InterruptedException e) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
}
|
||||||
|
// Fail any batches that parsed OK but never got to embed (restart mid-queue).
|
||||||
|
ParsedBatch orphan;
|
||||||
|
while ((orphan = embedQueue.poll()) != null) {
|
||||||
|
log.warn("failing orphaned embed batch for job {} (shutdown)", orphan.jobId());
|
||||||
|
repoStore.updateVersionStatus(orphan.ver().id(), VersionStatus.FAILED, "application shutdown");
|
||||||
|
transitionJob(orphan.jobId(), JobStatus.FAILED, null, Instant.now());
|
||||||
|
running.remove(orphan.jobId());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ */
|
||||||
|
/* Stage helpers */
|
||||||
|
/* ------------------------------------------------------------------ */
|
||||||
|
|
||||||
|
private interface StageBody {
|
||||||
|
long execute() throws Exception;
|
||||||
|
}
|
||||||
|
|
||||||
|
private interface StageBodyReturning<T> {
|
||||||
|
T execute() throws Exception;
|
||||||
|
}
|
||||||
|
|
||||||
|
private void stage(JobId id, JobStage.StageName name, StageBody body) {
|
||||||
|
stageReturning(id, name, () -> {
|
||||||
|
long n = body.execute();
|
||||||
|
return n;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private <T> T stageReturning(JobId id, JobStage.StageName name, StageBodyReturning<T> body) {
|
||||||
|
Instant start = Instant.now();
|
||||||
|
JobStage running = new JobStage(
|
||||||
|
id, name, JobStage.StageStatus.RUNNING, start, null, 0, 0, 0, null);
|
||||||
|
jobStore.upsertStage(running);
|
||||||
|
publishJob(id);
|
||||||
|
try {
|
||||||
|
T out = body.execute();
|
||||||
|
long items = (out instanceof Long l) ? l : (out instanceof List<?> l ? l.size() : 1);
|
||||||
|
JobStage done = new JobStage(
|
||||||
|
id, name, JobStage.StageStatus.SUCCEEDED, start, Instant.now(), items, items, 0, null);
|
||||||
|
jobStore.upsertStage(done);
|
||||||
|
publishJob(id);
|
||||||
|
return out;
|
||||||
|
} catch (Exception e) {
|
||||||
|
JobStage failed = new JobStage(
|
||||||
|
id, name, JobStage.StageStatus.FAILED, start, Instant.now(), 0, 0, 0, e.getMessage());
|
||||||
|
jobStore.upsertStage(failed);
|
||||||
|
publishJob(id);
|
||||||
|
throw new RuntimeException(e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void transitionJob(JobId id, JobStatus s, Instant startedAt, Instant finishedAt) {
|
||||||
|
jobStore.updateStatus(id, s, startedAt, finishedAt);
|
||||||
|
publishJob(id);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void publishJob(JobId id) {
|
||||||
|
jobStore.findById(id).ifPresent(bus::publishJob);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void logEvent(JobId id, JobLogEvent.Level level, JobStage.StageName stage, String msg) {
|
||||||
|
bus.publishLog(new JobLogEvent(id, Instant.now(), level, stage, msg));
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ */
|
||||||
|
/* Pipeline steps */
|
||||||
|
/* ------------------------------------------------------------------ */
|
||||||
|
|
||||||
|
private List<Path> discoverFiles(Path root, Repository repo) throws IOException {
|
||||||
|
List<PathMatcher> matchers = new ArrayList<>();
|
||||||
|
for (String g : BUILTIN_IGNORES) matchers.add(FileSystems.getDefault().getPathMatcher("glob:" + g));
|
||||||
|
for (String g : repo.ignoreGlobs()) matchers.add(FileSystems.getDefault().getPathMatcher("glob:" + g));
|
||||||
|
long maxBytes = repo.maxFileSizeBytes();
|
||||||
|
List<Path> out = new ArrayList<>();
|
||||||
|
Files.walkFileTree(root, new SimpleFileVisitor<>() {
|
||||||
|
@Override
|
||||||
|
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) {
|
||||||
|
Path rel = root.relativize(file);
|
||||||
|
for (PathMatcher m : matchers) {
|
||||||
|
if (m.matches(rel) || m.matches(file.getFileName())) return FileVisitResult.CONTINUE;
|
||||||
|
}
|
||||||
|
if (attrs.size() > maxBytes) return FileVisitResult.CONTINUE;
|
||||||
|
out.add(file);
|
||||||
|
return FileVisitResult.CONTINUE;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) {
|
||||||
|
if (dir.equals(root)) return FileVisitResult.CONTINUE;
|
||||||
|
Path rel = root.relativize(dir);
|
||||||
|
for (PathMatcher m : matchers) {
|
||||||
|
if (m.matches(rel)) return FileVisitResult.SKIP_SUBTREE;
|
||||||
|
}
|
||||||
|
return FileVisitResult.CONTINUE;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
private record ParsedPiece(
|
||||||
|
String contentHash,
|
||||||
|
String content,
|
||||||
|
String language,
|
||||||
|
String symbol,
|
||||||
|
int tokenCount,
|
||||||
|
String filePath,
|
||||||
|
int startLine,
|
||||||
|
int endLine) {}
|
||||||
|
|
||||||
|
private List<ParsedPiece> parseAll(List<Path> files, Path root) {
|
||||||
|
List<ParsedPiece> out = new ArrayList<>();
|
||||||
|
MessageDigest sha;
|
||||||
|
try {
|
||||||
|
sha = MessageDigest.getInstance("SHA-256");
|
||||||
|
} catch (Exception e) {
|
||||||
|
throw new IllegalStateException(e);
|
||||||
|
}
|
||||||
|
for (Path f : files) {
|
||||||
|
String rel = root.relativize(f).toString().replace('\\', '/');
|
||||||
|
try {
|
||||||
|
List<CodeParser.ParsedChunk> parsed = parser.parse(f, rel);
|
||||||
|
for (var pc : parsed) {
|
||||||
|
String hash = bytesToHex(sha.digest(pc.content().getBytes(StandardCharsets.UTF_8)));
|
||||||
|
sha.reset();
|
||||||
|
int tokens = Math.max(1, pc.content().length() / 4); // heuristic; refined if needed
|
||||||
|
out.add(new ParsedPiece(
|
||||||
|
hash, pc.content(), pc.language(), pc.symbol(), tokens, rel, pc.startLine(), pc.endLine()));
|
||||||
|
}
|
||||||
|
} catch (Exception e) {
|
||||||
|
log.warn("parse failed for {}: {}", rel, e.toString());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<Chunk> embedAll(JobId jobId, List<ParsedPiece> pieces) {
|
||||||
|
// Dedupe by hash across this batch AND against existing chunks in the store/cache.
|
||||||
|
Map<String, Chunk> resolved = new HashMap<>();
|
||||||
|
List<ParsedPiece> toEmbed = new ArrayList<>();
|
||||||
|
for (var p : pieces) {
|
||||||
|
if (resolved.containsKey(p.contentHash())) continue;
|
||||||
|
var existing = chunkStore.findByContentHash(p.contentHash());
|
||||||
|
if (existing.isPresent()) {
|
||||||
|
resolved.put(p.contentHash(), existing.get());
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
// cache?
|
||||||
|
var cached = embeddingCache.get(p.contentHash());
|
||||||
|
if (cached.isPresent()) {
|
||||||
|
Chunk c = upsert(p, cached.get());
|
||||||
|
resolved.put(p.contentHash(), c);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
toEmbed.add(p);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!toEmbed.isEmpty()) {
|
||||||
|
List<String> texts = toEmbed.stream().map(ParsedPiece::content).toList();
|
||||||
|
List<float[]> vecs = embeddings.embed(texts);
|
||||||
|
for (int i = 0; i < toEmbed.size(); i++) {
|
||||||
|
var p = toEmbed.get(i);
|
||||||
|
float[] v = vecs.get(i);
|
||||||
|
embeddingCache.put(p.contentHash(), v);
|
||||||
|
Chunk c = upsert(p, v);
|
||||||
|
resolved.put(p.contentHash(), c);
|
||||||
|
}
|
||||||
|
logEvent(jobId, JobLogEvent.Level.INFO, JobStage.StageName.EMBED,
|
||||||
|
"embedded " + toEmbed.size() + " new chunks (cache/dedupe hits = " + (pieces.size() - toEmbed.size()) + ")");
|
||||||
|
}
|
||||||
|
return new ArrayList<>(resolved.values());
|
||||||
|
}
|
||||||
|
|
||||||
|
private Chunk upsert(ParsedPiece p, float[] vector) {
|
||||||
|
Chunk c = new Chunk(ChunkId.random(), p.contentHash(), p.content(), p.language(), p.symbol(), p.tokenCount());
|
||||||
|
return chunkStore.upsertChunk(c, new Embedding(c.id(), vector));
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<ChunkVersion> buildLinks(VersionId versionId, List<ParsedPiece> pieces) {
|
||||||
|
// Piece → ChunkId requires knowing the chunk id assigned on upsert.
|
||||||
|
// We re-resolve via findByContentHash — cheap because it's a Term query.
|
||||||
|
List<ChunkVersion> links = new ArrayList<>(pieces.size());
|
||||||
|
Map<String, ChunkId> hashToId = new HashMap<>();
|
||||||
|
for (var p : pieces) {
|
||||||
|
ChunkId id = hashToId.computeIfAbsent(p.contentHash(), h ->
|
||||||
|
chunkStore.findByContentHash(h).orElseThrow().id());
|
||||||
|
links.add(new ChunkVersion(id, versionId, p.filePath(), p.startLine(), p.endLine()));
|
||||||
|
}
|
||||||
|
return links;
|
||||||
|
}
|
||||||
|
|
||||||
|
private String pickParentIndexedTag(Repository repo, Version ver) {
|
||||||
|
// Most recent previously-indexed version for this repo that isn't this one.
|
||||||
|
List<Version> indexed = repoStore.findVersionsByStatus(repo.id(), VersionStatus.INDEXED);
|
||||||
|
return indexed.stream()
|
||||||
|
.filter(v -> !v.id().equals(ver.id()))
|
||||||
|
.max((a, b) -> a.tag().compareTo(b.tag()))
|
||||||
|
.map(Version::tag)
|
||||||
|
.orElse(null);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static String bytesToHex(byte[] bytes) {
|
||||||
|
char[] hex = "0123456789abcdef".toCharArray();
|
||||||
|
char[] out = new char[bytes.length * 2];
|
||||||
|
for (int i = 0; i < bytes.length; i++) {
|
||||||
|
int v = bytes[i] & 0xff;
|
||||||
|
out[i * 2] = hex[v >>> 4];
|
||||||
|
out[i * 2 + 1] = hex[v & 0x0f];
|
||||||
|
}
|
||||||
|
return new String(out);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
package com.trueref.application.observability;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.IngestionJob;
|
||||||
|
import com.trueref.domain.model.JobId;
|
||||||
|
import com.trueref.domain.model.JobLogEvent;
|
||||||
|
import com.trueref.domain.port.out.JobEventBus;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.concurrent.ConcurrentHashMap;
|
||||||
|
import java.util.concurrent.CopyOnWriteArrayList;
|
||||||
|
import java.util.function.Consumer;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* In-process publish/subscribe implementation of {@link JobEventBus}. Listeners receive events on
|
||||||
|
* the publisher's thread; consumers should defer expensive work (e.g. SSE writes) to a virtual
|
||||||
|
* thread to keep the publisher fast.
|
||||||
|
*/
|
||||||
|
public final class InMemoryJobEventBus implements JobEventBus {
|
||||||
|
|
||||||
|
private final CopyOnWriteArrayList<Consumer<IngestionJob>> jobListeners = new CopyOnWriteArrayList<>();
|
||||||
|
private final Map<JobId, CopyOnWriteArrayList<Consumer<JobLogEvent>>> logListeners = new ConcurrentHashMap<>();
|
||||||
|
private final CopyOnWriteArrayList<Consumer<JobLogEvent>> globalLogListeners = new CopyOnWriteArrayList<>();
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void publishJob(IngestionJob job) {
|
||||||
|
for (Consumer<IngestionJob> l : jobListeners) {
|
||||||
|
try {
|
||||||
|
l.accept(job);
|
||||||
|
} catch (Exception ignored) {
|
||||||
|
// listener failures must not break publishing
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void publishLog(JobLogEvent event) {
|
||||||
|
var perJob = logListeners.get(event.jobId());
|
||||||
|
if (perJob != null) {
|
||||||
|
for (Consumer<JobLogEvent> l : perJob) {
|
||||||
|
try {
|
||||||
|
l.accept(event);
|
||||||
|
} catch (Exception ignored) {
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (Consumer<JobLogEvent> l : globalLogListeners) {
|
||||||
|
try {
|
||||||
|
l.accept(event);
|
||||||
|
} catch (Exception ignored) {
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public AutoCloseable subscribeJobs(Consumer<IngestionJob> listener) {
|
||||||
|
jobListeners.add(listener);
|
||||||
|
return () -> jobListeners.remove(listener);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public AutoCloseable subscribeLogs(JobId jobId, Consumer<JobLogEvent> listener) {
|
||||||
|
var list = logListeners.computeIfAbsent(jobId, k -> new CopyOnWriteArrayList<>());
|
||||||
|
list.add(listener);
|
||||||
|
return () -> list.remove(listener);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Subscribe to ALL log events regardless of job (used by the dashboard). */
|
||||||
|
public AutoCloseable subscribeAllLogs(Consumer<JobLogEvent> listener) {
|
||||||
|
globalLogListeners.add(listener);
|
||||||
|
return () -> globalLogListeners.remove(listener);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
package com.trueref.application.observability;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.IngestionJob;
|
||||||
|
import com.trueref.domain.model.JobId;
|
||||||
|
import com.trueref.domain.model.JobLogEvent;
|
||||||
|
import com.trueref.domain.model.JobStatus;
|
||||||
|
import com.trueref.domain.model.RepositoryId;
|
||||||
|
import com.trueref.domain.model.VersionId;
|
||||||
|
import com.trueref.domain.port.in.ObserveJobs;
|
||||||
|
import com.trueref.domain.port.out.JobEventBus;
|
||||||
|
import com.trueref.domain.port.out.JobStore;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Optional;
|
||||||
|
import java.util.function.Consumer;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
public final class JobObservationService implements ObserveJobs {
|
||||||
|
|
||||||
|
private final JobStore jobs;
|
||||||
|
private final JobEventBus bus;
|
||||||
|
|
||||||
|
public JobObservationService(JobStore jobs, JobEventBus bus) {
|
||||||
|
this.jobs = jobs;
|
||||||
|
this.bus = bus;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Optional<IngestionJob> findJob(JobId id) {
|
||||||
|
return jobs.findById(id);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public List<IngestionJob> listJobs(
|
||||||
|
@Nullable RepositoryId repoId, @Nullable VersionId versionId, @Nullable JobStatus status, int limit) {
|
||||||
|
return jobs.find(repoId, versionId, status, limit);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public AutoCloseable subscribeJobs(Consumer<IngestionJob> listener) {
|
||||||
|
return bus.subscribeJobs(listener);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public AutoCloseable subscribeLogs(JobId jobId, Consumer<JobLogEvent> listener) {
|
||||||
|
return bus.subscribeLogs(jobId, listener);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
/** In-process implementations of cross-cutting application services. */
|
||||||
|
@org.jspecify.annotations.NullMarked
|
||||||
|
package com.trueref.application.observability;
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
/** Application services: use-case implementations. */
|
||||||
|
@org.jspecify.annotations.NullMarked
|
||||||
|
package com.trueref.application;
|
||||||
@@ -0,0 +1,161 @@
|
|||||||
|
package com.trueref.application.resolve;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.Repository;
|
||||||
|
import com.trueref.domain.model.RepositoryId;
|
||||||
|
import com.trueref.domain.model.TagPattern;
|
||||||
|
import com.trueref.domain.model.Version;
|
||||||
|
import com.trueref.domain.model.VersionStatus;
|
||||||
|
import com.trueref.domain.port.in.IndexVersion;
|
||||||
|
import com.trueref.domain.port.in.ResolveLibraryId;
|
||||||
|
import com.trueref.domain.port.out.RepositoryStore;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Optional;
|
||||||
|
import java.util.regex.Matcher;
|
||||||
|
import java.util.regex.Pattern;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fuzzy library-name matching + version→tag mapping. Mirrors Context7's {@code resolve-library-id}
|
||||||
|
* semantics. When {@code version} is provided and maps to a known-but-not-yet-indexed tag, triggers
|
||||||
|
* an async index job (fire-and-forget).
|
||||||
|
*/
|
||||||
|
public final class LibraryResolver implements ResolveLibraryId {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(LibraryResolver.class);
|
||||||
|
|
||||||
|
private static final Pattern SEMVER = Pattern.compile("^v?(\\d+)(?:\\.(\\d+))?(?:\\.(\\d+))?.*$");
|
||||||
|
|
||||||
|
private final RepositoryStore store;
|
||||||
|
private final IndexVersion indexer;
|
||||||
|
|
||||||
|
public LibraryResolver(RepositoryStore store, IndexVersion indexer) {
|
||||||
|
this.store = store;
|
||||||
|
this.indexer = indexer;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Result resolve(Query q) {
|
||||||
|
String needle = q.libraryName().toLowerCase();
|
||||||
|
List<Repository> all = store.findAll();
|
||||||
|
List<Match> matches = new ArrayList<>();
|
||||||
|
for (Repository r : all) {
|
||||||
|
double score = nameScore(r.name().toLowerCase(), needle);
|
||||||
|
if (score <= 0) continue;
|
||||||
|
List<Version> versions = store.findVersionsByRepo(r.id());
|
||||||
|
|
||||||
|
// If a version was requested, map it to a tag and ensure indexing.
|
||||||
|
if (q.version() != null && !q.version().isBlank()) {
|
||||||
|
Optional<Version> target = mapVersion(r, versions, q.version());
|
||||||
|
target.ifPresent(v -> ensureIndexed(r.id(), v));
|
||||||
|
}
|
||||||
|
|
||||||
|
int snippetCount = versions.stream()
|
||||||
|
.filter(v -> v.status() == VersionStatus.INDEXED)
|
||||||
|
.mapToInt(Version::chunkCount)
|
||||||
|
.sum();
|
||||||
|
List<VersionRef> refs = versions.stream()
|
||||||
|
.sorted(Comparator.comparing(Version::tag).reversed())
|
||||||
|
.map(v -> new VersionRef(v.id(), v.tag(), v.status()))
|
||||||
|
.toList();
|
||||||
|
String libraryId = "/" + r.name();
|
||||||
|
matches.add(new Match(r.id(), libraryId, r.name(), null, snippetCount, refs, score));
|
||||||
|
}
|
||||||
|
matches.sort(Comparator.comparingDouble(Match::score).reversed());
|
||||||
|
return new Result(matches);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Fuzzy name scoring: exact 1.0, prefix 0.9, contains 0.7, token overlap otherwise. */
|
||||||
|
private double nameScore(String haystack, String needle) {
|
||||||
|
if (haystack.equals(needle)) return 1.0;
|
||||||
|
if (haystack.endsWith("/" + needle) || haystack.startsWith(needle + "/")) return 0.95;
|
||||||
|
if (haystack.contains(needle)) return 0.8;
|
||||||
|
// token overlap
|
||||||
|
String[] hTok = haystack.split("[^a-z0-9]+");
|
||||||
|
String[] nTok = needle.split("[^a-z0-9]+");
|
||||||
|
int hit = 0;
|
||||||
|
for (String nt : nTok) {
|
||||||
|
if (nt.isBlank()) continue;
|
||||||
|
for (String ht : hTok) {
|
||||||
|
if (ht.equals(nt)) { hit++; break; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (hit == 0) return 0.0;
|
||||||
|
return 0.3 + 0.4 * ((double) hit / Math.max(1, nTok.length));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Maps a version string to the closest matching tag using the repo's configured mapping rules.
|
||||||
|
* Rules are tried in order.
|
||||||
|
*/
|
||||||
|
public Optional<Version> mapVersion(Repository repo, List<Version> versions, String requested) {
|
||||||
|
for (TagPattern rule : repo.versionMappingRules()) {
|
||||||
|
String candidate = switch (rule) {
|
||||||
|
case TagPattern.Exact e -> requested;
|
||||||
|
case TagPattern.VPrefix v -> "v" + stripV(requested);
|
||||||
|
case TagPattern.ReleasePrefix r -> "release-" + stripV(requested);
|
||||||
|
case TagPattern.Custom c -> c.template()
|
||||||
|
.replace("{version}", requested)
|
||||||
|
.replace("{semver}", stripV(requested));
|
||||||
|
case TagPattern.SemverFuzzy s -> null; // handled below
|
||||||
|
};
|
||||||
|
if (candidate == null) continue;
|
||||||
|
Optional<Version> exact = versions.stream()
|
||||||
|
.filter(v -> v.tag().equalsIgnoreCase(candidate))
|
||||||
|
.findFirst();
|
||||||
|
if (exact.isPresent()) return exact;
|
||||||
|
}
|
||||||
|
// Semver fuzzy: pick tag with closest semver distance
|
||||||
|
return semverClosest(versions, requested);
|
||||||
|
}
|
||||||
|
|
||||||
|
private Optional<Version> semverClosest(List<Version> versions, String requested) {
|
||||||
|
int[] r = parseSemver(requested);
|
||||||
|
if (r == null) return Optional.empty();
|
||||||
|
return versions.stream()
|
||||||
|
.map(v -> new Object[] {v, parseSemver(v.tag())})
|
||||||
|
.filter(t -> t[1] != null)
|
||||||
|
.min(Comparator.comparingLong(t -> semverDist((int[]) t[1], r)))
|
||||||
|
.map(t -> (Version) t[0]);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static @Nullable int[] parseSemver(String s) {
|
||||||
|
Matcher m = SEMVER.matcher(s);
|
||||||
|
if (!m.matches()) return null;
|
||||||
|
return new int[] {
|
||||||
|
parseIntOrZero(m.group(1)),
|
||||||
|
parseIntOrZero(m.group(2)),
|
||||||
|
parseIntOrZero(m.group(3))
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private static int parseIntOrZero(String s) {
|
||||||
|
if (s == null || s.isEmpty()) return 0;
|
||||||
|
try { return Integer.parseInt(s); } catch (NumberFormatException e) { return 0; }
|
||||||
|
}
|
||||||
|
|
||||||
|
private static long semverDist(int[] a, int[] b) {
|
||||||
|
long d = 0;
|
||||||
|
d += Math.abs(a[0] - b[0]) * 1_000_000L;
|
||||||
|
d += Math.abs(a[1] - b[1]) * 1_000L;
|
||||||
|
d += Math.abs(a[2] - b[2]);
|
||||||
|
return d;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static String stripV(String s) {
|
||||||
|
return s.startsWith("v") || s.startsWith("V") ? s.substring(1) : s;
|
||||||
|
}
|
||||||
|
|
||||||
|
private void ensureIndexed(RepositoryId repoId, Version v) {
|
||||||
|
if (v.status() == VersionStatus.INDEXED || v.status() == VersionStatus.INDEXING) return;
|
||||||
|
try {
|
||||||
|
log.info("on-demand indexing: repo={} tag={}", repoId, v.tag());
|
||||||
|
indexer.enqueue(repoId, v.id(), false);
|
||||||
|
} catch (Exception e) {
|
||||||
|
log.warn("on-demand indexing enqueue failed: {}", e.toString());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,238 @@
|
|||||||
|
package com.trueref.application.search;
|
||||||
|
|
||||||
|
import com.trueref.domain.error.InvalidSearchRequest;
|
||||||
|
import com.trueref.domain.model.ChunkId;
|
||||||
|
import com.trueref.domain.model.Repository;
|
||||||
|
import com.trueref.domain.model.SearchHit;
|
||||||
|
import com.trueref.domain.model.SearchScope;
|
||||||
|
import com.trueref.domain.model.Version;
|
||||||
|
import com.trueref.domain.port.in.SearchLibraryDocs;
|
||||||
|
import com.trueref.domain.port.out.ChunkStore;
|
||||||
|
import com.trueref.domain.port.out.EmbeddingService;
|
||||||
|
import com.trueref.domain.port.out.RepositoryStore;
|
||||||
|
import com.trueref.domain.port.out.RerankerService;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.HashMap;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.Objects;
|
||||||
|
import java.util.regex.Matcher;
|
||||||
|
import java.util.regex.Pattern;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Hybrid search: BM25 + dense kNN fused by Reciprocal Rank Fusion (RRF), then reranked by a
|
||||||
|
* cross-encoder, then packed to a token budget.
|
||||||
|
*/
|
||||||
|
public final class HybridSearchService implements SearchLibraryDocs {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(HybridSearchService.class);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Matches camelCase identifiers that are likely to be Phaser API method/class names (≥6 chars,
|
||||||
|
* must contain at least one uppercase letter after the first char, not all-caps).
|
||||||
|
* Examples: setCollideWorldBounds, createBitmapMask, addOverlap.
|
||||||
|
*/
|
||||||
|
private static final Pattern CAMEL_IDENT = Pattern.compile(
|
||||||
|
"\\b([a-z][a-zA-Z0-9]{5,})(?=\\b)");
|
||||||
|
|
||||||
|
private final ChunkStore chunks;
|
||||||
|
private final EmbeddingService embedder;
|
||||||
|
private final RerankerService reranker;
|
||||||
|
private final RepositoryStore repos;
|
||||||
|
private final int rrfK;
|
||||||
|
private final int rerankTopK;
|
||||||
|
private final int finalTopK;
|
||||||
|
|
||||||
|
public HybridSearchService(
|
||||||
|
ChunkStore chunks,
|
||||||
|
EmbeddingService embedder,
|
||||||
|
RerankerService reranker,
|
||||||
|
RepositoryStore repos,
|
||||||
|
int rrfK,
|
||||||
|
int rerankTopK,
|
||||||
|
int finalTopK) {
|
||||||
|
this.chunks = chunks;
|
||||||
|
this.embedder = embedder;
|
||||||
|
this.reranker = reranker;
|
||||||
|
this.repos = repos;
|
||||||
|
this.rrfK = rrfK;
|
||||||
|
this.rerankTopK = rerankTopK;
|
||||||
|
this.finalTopK = finalTopK;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Result search(Query q) {
|
||||||
|
if (q.text() == null || q.text().isBlank()) {
|
||||||
|
throw new InvalidSearchRequest("query text must not be blank");
|
||||||
|
}
|
||||||
|
if (q.scope().refs().isEmpty()) {
|
||||||
|
throw new InvalidSearchRequest("search scope must not be empty");
|
||||||
|
}
|
||||||
|
|
||||||
|
String text = rewrite(q.text(), q.topic());
|
||||||
|
// Augment BM25 query with camelCase identifiers found in the text so that the exact
|
||||||
|
// method-name chunk scores higher in BM25 even when it competes with generic mentions.
|
||||||
|
String bm25Text = augmentWithCamelIdents(text);
|
||||||
|
|
||||||
|
List<SearchHit> bm25 = chunks.bm25Search(bm25Text, q.scope(), rerankTopK);
|
||||||
|
float[] vec = embedder.embed(List.of(text)).get(0);
|
||||||
|
List<SearchHit> dense = chunks.denseSearch(vec, q.scope(), rerankTopK);
|
||||||
|
|
||||||
|
List<SearchHit> fused = rrf(bm25, dense);
|
||||||
|
if (fused.size() > rerankTopK) fused = fused.subList(0, rerankTopK);
|
||||||
|
|
||||||
|
// Demote changelog / synthetic-skill / docs paths before the reranker sees them so that
|
||||||
|
// authoritative source-code chunks aren't squeezed out by historical migration notes.
|
||||||
|
List<SearchHit> biased = applyFilePathBias(fused);
|
||||||
|
|
||||||
|
// Enrich with repo name + tag (ChunkStore leaves these empty).
|
||||||
|
List<SearchHit> enriched = enrich(biased);
|
||||||
|
|
||||||
|
List<SearchHit> reranked = reranker.rerank(text, enriched);
|
||||||
|
|
||||||
|
List<SearchHit> packed = packByTokenBudget(reranked, q.tokensBudget(), q.maxHits() > 0 ? q.maxHits() : finalTopK);
|
||||||
|
int totalTokens = packed.stream().mapToInt(h -> estimateTokens(h.content())).sum();
|
||||||
|
return new Result(packed, totalTokens);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ */
|
||||||
|
|
||||||
|
private String rewrite(String text, String topic) {
|
||||||
|
String base = text.trim();
|
||||||
|
if (topic != null && !topic.isBlank()) {
|
||||||
|
return base + " " + topic.trim();
|
||||||
|
}
|
||||||
|
return base;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Returns a copy of {@code text} with each camelCase identifier repeated at the end (once).
|
||||||
|
* This lifts their BM25 term-frequency contribution without altering the semantic meaning
|
||||||
|
* used for the dense embedding query.
|
||||||
|
*
|
||||||
|
* <p>Example: "how to use setCollideWorldBounds" →
|
||||||
|
* "how to use setCollideWorldBounds setCollideWorldBounds"
|
||||||
|
*/
|
||||||
|
private static String augmentWithCamelIdents(String text) {
|
||||||
|
Matcher m = CAMEL_IDENT.matcher(text);
|
||||||
|
StringBuilder extra = new StringBuilder();
|
||||||
|
while (m.find()) {
|
||||||
|
String ident = m.group(1);
|
||||||
|
// Only repeat identifiers that contain at least one uppercase letter
|
||||||
|
// (filters out short common words like "should", "create").
|
||||||
|
if (!ident.equals(ident.toLowerCase())) {
|
||||||
|
extra.append(' ').append(ident);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return extra.isEmpty() ? text : text + extra;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Applies a path-based multiplier to RRF scores before handing candidates to the reranker.
|
||||||
|
* Changelogs and synthetic skill docs are semantically relevant but tend to outrank the
|
||||||
|
* authoritative source-code chunks when the query mentions API migration or breaking changes.
|
||||||
|
* Demoting them here keeps them retrievable while giving source files priority.
|
||||||
|
*
|
||||||
|
* <p>Multipliers (tuned against the phaser_rag_eval suite):
|
||||||
|
* <ul>
|
||||||
|
* <li>{@code changelog/} → ×0.50 — migration notes, not current API reference
|
||||||
|
* <li>{@code skills/} / {@code SKILL.md} → ×0.60 — synthetic summaries, not authoritative
|
||||||
|
* <li>{@code docs/} → ×0.75 — curated docs; useful but prefer source JSDoc
|
||||||
|
* <li>everything else (source, tests, configs) → ×1.0
|
||||||
|
* </ul>
|
||||||
|
*/
|
||||||
|
private static List<SearchHit> applyFilePathBias(List<SearchHit> hits) {
|
||||||
|
boolean anyChanged = false;
|
||||||
|
List<SearchHit> out = new ArrayList<>(hits.size());
|
||||||
|
for (SearchHit h : hits) {
|
||||||
|
double mult = filePathMultiplier(h.filePath());
|
||||||
|
if (mult == 1.0) {
|
||||||
|
out.add(h);
|
||||||
|
} else {
|
||||||
|
out.add(new SearchHit(
|
||||||
|
h.chunkId(), h.repoId(), h.versionId(), h.repoName(), h.tag(),
|
||||||
|
h.filePath(), h.startLine(), h.endLine(), h.language(), h.symbol(),
|
||||||
|
h.content(), h.score() * mult));
|
||||||
|
anyChanged = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!anyChanged) return hits;
|
||||||
|
out.sort(Comparator.comparingDouble(SearchHit::score).reversed());
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static double filePathMultiplier(String filePath) {
|
||||||
|
if (filePath == null || filePath.isEmpty()) return 1.0;
|
||||||
|
String lp = filePath.toLowerCase();
|
||||||
|
if (lp.startsWith("changelog/") || lp.contains("/changelog/")) return 0.50;
|
||||||
|
if (lp.contains("/skills/") || lp.endsWith("skill.md")) return 0.60;
|
||||||
|
if (lp.startsWith("docs/") || lp.contains("/docs/")) return 0.75;
|
||||||
|
return 1.0;
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<SearchHit> rrf(List<SearchHit> a, List<SearchHit> b) {
|
||||||
|
Map<ChunkId, Double> scores = new HashMap<>();
|
||||||
|
Map<ChunkId, SearchHit> firstSeen = new HashMap<>();
|
||||||
|
addRankContribution(a, scores, firstSeen);
|
||||||
|
addRankContribution(b, scores, firstSeen);
|
||||||
|
return scores.entrySet().stream()
|
||||||
|
.sorted(Map.Entry.<ChunkId, Double>comparingByValue().reversed())
|
||||||
|
.map(e -> {
|
||||||
|
SearchHit h = firstSeen.get(e.getKey());
|
||||||
|
return new SearchHit(
|
||||||
|
h.chunkId(), h.repoId(), h.versionId(), h.repoName(), h.tag(),
|
||||||
|
h.filePath(), h.startLine(), h.endLine(), h.language(), h.symbol(),
|
||||||
|
h.content(), e.getValue());
|
||||||
|
})
|
||||||
|
.toList();
|
||||||
|
}
|
||||||
|
|
||||||
|
private void addRankContribution(List<SearchHit> hits, Map<ChunkId, Double> scores, Map<ChunkId, SearchHit> seen) {
|
||||||
|
for (int rank = 0; rank < hits.size(); rank++) {
|
||||||
|
SearchHit h = hits.get(rank);
|
||||||
|
scores.merge(h.chunkId(), 1.0 / (rrfK + rank + 1.0), Double::sum);
|
||||||
|
seen.putIfAbsent(h.chunkId(), h);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<SearchHit> enrich(List<SearchHit> hits) {
|
||||||
|
Map<String, String> repoNameByRepoId = new HashMap<>();
|
||||||
|
Map<String, String> tagByVersionId = new HashMap<>();
|
||||||
|
List<SearchHit> out = new ArrayList<>(hits.size());
|
||||||
|
for (SearchHit h : hits) {
|
||||||
|
String repoName = repoNameByRepoId.computeIfAbsent(
|
||||||
|
h.repoId().toString(),
|
||||||
|
k -> repos.findById(h.repoId()).map(Repository::name).orElse("?"));
|
||||||
|
String tag = tagByVersionId.computeIfAbsent(
|
||||||
|
h.versionId().toString(),
|
||||||
|
k -> repos.findVersion(h.versionId()).map(Version::tag).orElse("?"));
|
||||||
|
out.add(new SearchHit(
|
||||||
|
h.chunkId(), h.repoId(), h.versionId(),
|
||||||
|
repoName, tag,
|
||||||
|
h.filePath(), h.startLine(), h.endLine(), h.language(), h.symbol(),
|
||||||
|
h.content(), h.score()));
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<SearchHit> packByTokenBudget(List<SearchHit> ranked, int tokenBudget, int maxHits) {
|
||||||
|
List<SearchHit> out = new ArrayList<>();
|
||||||
|
int used = 0;
|
||||||
|
for (SearchHit h : ranked) {
|
||||||
|
if (out.size() >= maxHits) break;
|
||||||
|
int t = estimateTokens(h.content());
|
||||||
|
if (used + t > tokenBudget && !out.isEmpty()) break;
|
||||||
|
out.add(h);
|
||||||
|
used += t;
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 4 chars ≈ 1 token — same rule of thumb Context7 uses for packing. */
|
||||||
|
private static int estimateTokens(String s) {
|
||||||
|
return Math.max(1, s.length() / 4);
|
||||||
|
}
|
||||||
|
}
|
||||||
68
trueref-bootstrap/pom.xml
Normal file
68
trueref-bootstrap/pom.xml
Normal file
@@ -0,0 +1,68 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||||
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||||
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||||
|
<modelVersion>4.0.0</modelVersion>
|
||||||
|
|
||||||
|
<parent>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-parent</artifactId>
|
||||||
|
<version>0.1.0-SNAPSHOT</version>
|
||||||
|
</parent>
|
||||||
|
|
||||||
|
<artifactId>trueref-bootstrap</artifactId>
|
||||||
|
<name>trueref-bootstrap</name>
|
||||||
|
<description>Spring Boot entry point. Wires beans across modules. Produces the executable fat JAR.</description>
|
||||||
|
|
||||||
|
<dependencies>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-domain</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-application</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-adapters</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-frontend</artifactId>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.springframework.boot</groupId>
|
||||||
|
<artifactId>spring-boot-starter-actuator</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>io.micrometer</groupId>
|
||||||
|
<artifactId>micrometer-registry-prometheus</artifactId>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.springframework.boot</groupId>
|
||||||
|
<artifactId>spring-boot-starter-test</artifactId>
|
||||||
|
<scope>test</scope>
|
||||||
|
</dependency>
|
||||||
|
</dependencies>
|
||||||
|
|
||||||
|
<build>
|
||||||
|
<finalName>trueref</finalName>
|
||||||
|
<plugins>
|
||||||
|
<plugin>
|
||||||
|
<groupId>org.springframework.boot</groupId>
|
||||||
|
<artifactId>spring-boot-maven-plugin</artifactId>
|
||||||
|
<configuration>
|
||||||
|
<mainClass>com.trueref.bootstrap.TrueRefApplication</mainClass>
|
||||||
|
</configuration>
|
||||||
|
<executions>
|
||||||
|
<execution>
|
||||||
|
<goals><goal>repackage</goal></goals>
|
||||||
|
</execution>
|
||||||
|
</executions>
|
||||||
|
</plugin>
|
||||||
|
</plugins>
|
||||||
|
</build>
|
||||||
|
</project>
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
package com.trueref.bootstrap;
|
||||||
|
|
||||||
|
import com.trueref.application.catalog.CatalogService;
|
||||||
|
import com.trueref.application.ingest.DiscoveryService;
|
||||||
|
import com.trueref.application.ingest.IngestionOrchestrator;
|
||||||
|
import com.trueref.application.observability.InMemoryJobEventBus;
|
||||||
|
import com.trueref.application.observability.JobObservationService;
|
||||||
|
import com.trueref.application.resolve.LibraryResolver;
|
||||||
|
import com.trueref.application.search.HybridSearchService;
|
||||||
|
import com.trueref.domain.port.out.ChunkStore;
|
||||||
|
import com.trueref.domain.port.out.CodeParser;
|
||||||
|
import com.trueref.domain.port.out.EmbeddingCache;
|
||||||
|
import com.trueref.domain.port.out.EmbeddingService;
|
||||||
|
import com.trueref.domain.port.out.GitClient;
|
||||||
|
import com.trueref.domain.port.out.JobStore;
|
||||||
|
import com.trueref.domain.port.out.RepositoryStore;
|
||||||
|
import com.trueref.domain.port.out.RerankerService;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import org.springframework.beans.factory.annotation.Value;
|
||||||
|
import org.springframework.context.annotation.Bean;
|
||||||
|
import org.springframework.context.annotation.Configuration;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Explicit bean wiring for the application layer (which stays Spring-annotation-free).
|
||||||
|
* We expose only concrete beans; Spring resolves interface dependencies against the single
|
||||||
|
* concrete implementation.
|
||||||
|
*/
|
||||||
|
@Configuration
|
||||||
|
public class ApplicationBeans {
|
||||||
|
|
||||||
|
@Bean
|
||||||
|
Path trueRefHome(@Value("${trueref.home:./data}") String home) {
|
||||||
|
return Path.of(home);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Bean
|
||||||
|
InMemoryJobEventBus jobEventBus() {
|
||||||
|
return new InMemoryJobEventBus();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Bean
|
||||||
|
CatalogService catalogService(RepositoryStore store, Path trueRefHome) {
|
||||||
|
return new CatalogService(store, trueRefHome);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Bean
|
||||||
|
DiscoveryService discoveryService(RepositoryStore store, GitClient git) {
|
||||||
|
return new DiscoveryService(store, git);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Bean(destroyMethod = "shutdown")
|
||||||
|
IngestionOrchestrator ingestionOrchestrator(
|
||||||
|
RepositoryStore repoStore,
|
||||||
|
JobStore jobStore,
|
||||||
|
ChunkStore chunkStore,
|
||||||
|
EmbeddingService embeddings,
|
||||||
|
EmbeddingCache embeddingCache,
|
||||||
|
GitClient git,
|
||||||
|
CodeParser parser,
|
||||||
|
InMemoryJobEventBus bus,
|
||||||
|
@Value("${trueref.ingestion.max-parse-jobs:4}") int maxParseJobs,
|
||||||
|
@Value("${trueref.ingestion.embed-queue-capacity:4}") int embedQueueCapacity) {
|
||||||
|
return new IngestionOrchestrator(
|
||||||
|
repoStore, jobStore, chunkStore, embeddings, embeddingCache, git, parser, bus,
|
||||||
|
maxParseJobs, embedQueueCapacity);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Bean
|
||||||
|
LibraryResolver libraryResolver(RepositoryStore store, IngestionOrchestrator indexer) {
|
||||||
|
return new LibraryResolver(store, indexer);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Bean
|
||||||
|
HybridSearchService hybridSearchService(
|
||||||
|
ChunkStore chunks,
|
||||||
|
EmbeddingService embedder,
|
||||||
|
RerankerService reranker,
|
||||||
|
RepositoryStore repos,
|
||||||
|
@Value("${trueref.search.rrf-k:60}") int rrfK,
|
||||||
|
@Value("${trueref.reranker.top-k:50}") int rerankTopK,
|
||||||
|
@Value("${trueref.search.final-top-k:20}") int finalTopK) {
|
||||||
|
return new HybridSearchService(chunks, embedder, reranker, repos, rrfK, rerankTopK, finalTopK);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Bean
|
||||||
|
JobObservationService jobObservationService(JobStore jobs, InMemoryJobEventBus bus) {
|
||||||
|
return new JobObservationService(jobs, bus);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
package com.trueref.bootstrap;
|
||||||
|
|
||||||
|
import com.trueref.application.ingest.DiscoveryService;
|
||||||
|
import com.trueref.domain.model.Repository;
|
||||||
|
import com.trueref.domain.model.Version;
|
||||||
|
import com.trueref.domain.model.VersionStatus;
|
||||||
|
import com.trueref.domain.port.in.IndexVersion;
|
||||||
|
import com.trueref.domain.port.out.RepositoryStore;
|
||||||
|
import java.time.Instant;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.scheduling.annotation.EnableScheduling;
|
||||||
|
import org.springframework.scheduling.annotation.Scheduled;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
/** Periodically fetches tags for registered repos and enqueues indexing for new ones. */
|
||||||
|
@Component
|
||||||
|
@EnableScheduling
|
||||||
|
@ConditionalOnProperty(name = "trueref.ingestion.poller-enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class ScheduledPoller {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(ScheduledPoller.class);
|
||||||
|
|
||||||
|
private final RepositoryStore repoStore;
|
||||||
|
private final DiscoveryService discovery;
|
||||||
|
private final IndexVersion indexer;
|
||||||
|
|
||||||
|
public ScheduledPoller(RepositoryStore repoStore, DiscoveryService discovery, IndexVersion indexer) {
|
||||||
|
this.repoStore = repoStore;
|
||||||
|
this.discovery = discovery;
|
||||||
|
this.indexer = indexer;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Scheduled(fixedDelayString = "${trueref.ingestion.poll-interval-default:PT1H}")
|
||||||
|
public void pollAll() {
|
||||||
|
Instant start = Instant.now();
|
||||||
|
int scanned = 0;
|
||||||
|
int enqueued = 0;
|
||||||
|
for (Repository repo : repoStore.findAll()) {
|
||||||
|
try {
|
||||||
|
discovery.discover(repo.id());
|
||||||
|
for (Version v : repoStore.findVersionsByRepo(repo.id())) {
|
||||||
|
if (v.status() == VersionStatus.DISCOVERED) {
|
||||||
|
indexer.enqueue(repo.id(), v.id(), false);
|
||||||
|
enqueued++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
scanned++;
|
||||||
|
} catch (Exception e) {
|
||||||
|
log.warn("poll failed for repo={}: {}", repo.name(), e.toString());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
log.info("poll completed in {}ms: repos scanned={} jobs enqueued={}",
|
||||||
|
java.time.Duration.between(start, Instant.now()).toMillis(), scanned, enqueued);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
package com.trueref.bootstrap;
|
||||||
|
|
||||||
|
import com.trueref.domain.port.out.JobStore;
|
||||||
|
import com.trueref.domain.port.out.RepositoryStore;
|
||||||
|
import java.time.Instant;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.boot.context.event.ApplicationReadyEvent;
|
||||||
|
import org.springframework.context.event.EventListener;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* On-startup cleanup for stale job state left by a previous crash or SIGKILL.
|
||||||
|
*
|
||||||
|
* <p>Any job that is RUNNING or QUEUED when the application starts must have been orphaned by a
|
||||||
|
* previous JVM exit (clean or unclean). We fail them all atomically before accepting traffic so
|
||||||
|
* the UI and API never show phantom RUNNING jobs. Matching INDEXING versions are also reset to
|
||||||
|
* FAILED so they can be re-queued immediately.
|
||||||
|
*
|
||||||
|
* <p>This fires <em>after</em> Flyway migrations and all beans are initialised, but before the
|
||||||
|
* application starts accepting HTTP requests (ApplicationReadyEvent fires before the embedded
|
||||||
|
* Tomcat connector starts accepting connections).
|
||||||
|
*/
|
||||||
|
@Component
|
||||||
|
class StaleJobCleanupStartup {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(StaleJobCleanupStartup.class);
|
||||||
|
|
||||||
|
private static final String RESTART_REASON = "interrupted by server restart";
|
||||||
|
|
||||||
|
private final JobStore jobStore;
|
||||||
|
private final RepositoryStore repositoryStore;
|
||||||
|
|
||||||
|
StaleJobCleanupStartup(JobStore jobStore, RepositoryStore repositoryStore) {
|
||||||
|
this.jobStore = jobStore;
|
||||||
|
this.repositoryStore = repositoryStore;
|
||||||
|
}
|
||||||
|
|
||||||
|
@EventListener(ApplicationReadyEvent.class)
|
||||||
|
public void cleanupStaleJobs() {
|
||||||
|
Instant now = Instant.now();
|
||||||
|
|
||||||
|
int failedJobs = jobStore.failStaleJobs(now);
|
||||||
|
if (failedJobs > 0) {
|
||||||
|
log.warn(
|
||||||
|
"Startup cleanup: marked {} orphaned job(s) as FAILED (were RUNNING or QUEUED at shutdown).",
|
||||||
|
failedJobs);
|
||||||
|
} else {
|
||||||
|
log.info("Startup cleanup: no stale jobs found.");
|
||||||
|
}
|
||||||
|
|
||||||
|
int failedVersions = repositoryStore.failStaleIndexingVersions(RESTART_REASON);
|
||||||
|
if (failedVersions > 0) {
|
||||||
|
log.warn(
|
||||||
|
"Startup cleanup: reset {} INDEXING version(s) to FAILED (their jobs did not complete).",
|
||||||
|
failedVersions);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
package com.trueref.bootstrap;
|
||||||
|
|
||||||
|
import org.springframework.boot.SpringApplication;
|
||||||
|
import org.springframework.boot.autoconfigure.SpringBootApplication;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Trueref entry point. The only place where Spring component scanning is allowed across the
|
||||||
|
* {@code com.trueref} package tree. Adapters and application modules expose explicit
|
||||||
|
* {@code @Configuration} classes that this class imports via component scanning.
|
||||||
|
*/
|
||||||
|
@SpringBootApplication(scanBasePackages = "com.trueref")
|
||||||
|
public class TrueRefApplication {
|
||||||
|
|
||||||
|
public static void main(String[] args) {
|
||||||
|
SpringApplication.run(TrueRefApplication.class, args);
|
||||||
|
}
|
||||||
|
}
|
||||||
114
trueref-bootstrap/src/main/resources/application.yml
Normal file
114
trueref-bootstrap/src/main/resources/application.yml
Normal file
@@ -0,0 +1,114 @@
|
|||||||
|
spring:
|
||||||
|
application:
|
||||||
|
name: trueref
|
||||||
|
threads:
|
||||||
|
virtual:
|
||||||
|
enabled: true
|
||||||
|
datasource:
|
||||||
|
url: jdbc:h2:file:${trueref.home:./data}/h2/trueref;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE;MV_STORE=TRUE
|
||||||
|
username: sa
|
||||||
|
password: ""
|
||||||
|
driver-class-name: org.h2.Driver
|
||||||
|
hikari:
|
||||||
|
# Embedded H2 serialises writes internally; 8 connections is ample for virtual-thread
|
||||||
|
# workloads. 32 is wasteful and causes unnecessary H2 lock contention.
|
||||||
|
maximum-pool-size: 8
|
||||||
|
minimum-idle: 2
|
||||||
|
flyway:
|
||||||
|
enabled: true
|
||||||
|
locations: classpath:db/migration
|
||||||
|
mvc:
|
||||||
|
async:
|
||||||
|
request-timeout: 0 # SSE streams must not time out
|
||||||
|
# Spring AI MCP server. In Spring AI 1.0.0 the WebMVC transport is SSE-based
|
||||||
|
# (WebMvcSseServerTransportProvider) — the closest available transport to the 2025-03-26
|
||||||
|
# "Streamable HTTP" spec; there is no separate "protocol: streamable" property in this
|
||||||
|
# starter. JSON-RPC POSTs land on `sse-message-endpoint` (/mcp); server-initiated
|
||||||
|
# notifications stream over `sse-endpoint` (/sse). See com.trueref.adapter.in.mcp.
|
||||||
|
ai:
|
||||||
|
mcp:
|
||||||
|
server:
|
||||||
|
enabled: true
|
||||||
|
name: trueref
|
||||||
|
version: 0.1.0
|
||||||
|
type: SYNC
|
||||||
|
sse-message-endpoint: /mcp
|
||||||
|
sse-endpoint: /sse
|
||||||
|
|
||||||
|
server:
|
||||||
|
port: 8080
|
||||||
|
shutdown: graceful
|
||||||
|
|
||||||
|
management:
|
||||||
|
endpoints:
|
||||||
|
web:
|
||||||
|
exposure:
|
||||||
|
include: health,info,metrics,prometheus
|
||||||
|
endpoint:
|
||||||
|
health:
|
||||||
|
show-details: always
|
||||||
|
|
||||||
|
springdoc:
|
||||||
|
api-docs:
|
||||||
|
path: /v3/api-docs
|
||||||
|
swagger-ui:
|
||||||
|
path: /swagger-ui.html
|
||||||
|
|
||||||
|
trueref:
|
||||||
|
home: ${TRUEREF_HOME:./data}
|
||||||
|
ingestion:
|
||||||
|
poll-interval-default: PT1H
|
||||||
|
tag-cap-default: 100
|
||||||
|
max-file-size-bytes-default: 1048576
|
||||||
|
watched-folder: ${trueref.home}/watched
|
||||||
|
# Max parallel parse jobs (FETCH/CLONE → CHECKOUT → DISCOVER → DIFF → PARSE).
|
||||||
|
# Parse is I/O + CPU only — no GPU. 4 is safe on this machine (Ryzen 9 3900X, 62 GB RAM);
|
||||||
|
# increase for repos with small files, decrease if git I/O saturates disk.
|
||||||
|
max-parse-jobs: 4
|
||||||
|
# Max parsed batches buffered between parse workers and the embed worker.
|
||||||
|
# When the embed worker is busy, parse workers block here — natural backpressure.
|
||||||
|
# Total peak in-memory batches = max-parse-jobs + embed-queue-capacity.
|
||||||
|
embed-queue-capacity: 4
|
||||||
|
embedding:
|
||||||
|
model: bge-base-en-v1.5
|
||||||
|
onnx-providers: cuda,directml,cpu
|
||||||
|
session-count: 1
|
||||||
|
batch-size: 32
|
||||||
|
max-seq-len: 512
|
||||||
|
# Which CUDA device to bind ONNX sessions to. Passed directly to ORT's CUDA EP
|
||||||
|
# as the physical device index — ORT uses the CUDA driver/NVML API which can bypass
|
||||||
|
# CUDA_VISIBLE_DEVICES remapping. The ./trueref script sets this to $TRUEREF_GPU (default: 1 = RTX 3060).
|
||||||
|
gpu-device-id: 0
|
||||||
|
# Per-session GPU memory cap in bytes. 0 = unbounded. With session-count=1 there
|
||||||
|
# is no pool contention, so leave this unbounded — capping it risks exhausting the
|
||||||
|
# BFC arena during model-weight loading before inference starts. The ./trueref script
|
||||||
|
# defaults to 0 and can be overridden with TRUEREF_MEM_LIMIT.
|
||||||
|
gpu-mem-limit-bytes: 0
|
||||||
|
# Override download URLs per (model, file). The built-in defaults (in ModelDownloader)
|
||||||
|
# cover bge-base-en-v1.5, ms-marco-MiniLM-L6-v2, bge-m3, and bge-reranker-v2-m3.
|
||||||
|
# Set HF_TOKEN in the environment for higher rate limits or gated models.
|
||||||
|
# model-sources:
|
||||||
|
# bge-base-en-v1.5:
|
||||||
|
# model.onnx:
|
||||||
|
# - https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/onnx/model.onnx
|
||||||
|
reranker:
|
||||||
|
model: ms-marco-MiniLM-L6-v2
|
||||||
|
top-k: 100
|
||||||
|
embedding-cache:
|
||||||
|
# Must match the embedding model's output dimension. Changing this automatically
|
||||||
|
# wipes the stale .f32 files in the cache directory on next startup.
|
||||||
|
dimension: 768
|
||||||
|
search:
|
||||||
|
rrf-k: 60
|
||||||
|
final-top-k: 20
|
||||||
|
mcp:
|
||||||
|
tokens-default: 5000
|
||||||
|
tokens-min: 500
|
||||||
|
tokens-max: 50000
|
||||||
|
|
||||||
|
logging:
|
||||||
|
level:
|
||||||
|
root: INFO
|
||||||
|
com.trueref: INFO
|
||||||
|
org.eclipse.jgit: WARN
|
||||||
|
org.apache.lucene: WARN
|
||||||
62
trueref-bootstrap/src/main/scripts/trueref
Executable file
62
trueref-bootstrap/src/main/scripts/trueref
Executable file
@@ -0,0 +1,62 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# trueref launcher — wraps the fat JAR with the JVM flags required to silence
|
||||||
|
# the FFM (foreign linker) restricted-method warning emitted by JNA-based
|
||||||
|
# tokenizer libraries and to make Lucene's Vector API path readable.
|
||||||
|
#
|
||||||
|
# --enable-native-access=ALL-UNNAMED
|
||||||
|
# Lucene 10 + DJL HuggingFace Tokenizers use the new java.lang.foreign
|
||||||
|
# Linker API; on Java 21 this requires explicit native-access opt-in.
|
||||||
|
# --add-modules jdk.incubator.vector
|
||||||
|
# Lucene 10 ships an incubator-vector codepath that is significantly
|
||||||
|
# faster for cosine/dot-product math but only loads if the module is
|
||||||
|
# made readable from the unnamed module.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# bin/trueref # default settings
|
||||||
|
# bin/trueref --server.port=18080 # forward Spring properties
|
||||||
|
# TRUEREF_JAR=/path/to/trueref.jar bin/trueref
|
||||||
|
#
|
||||||
|
# Environment overrides:
|
||||||
|
# TRUEREF_JAR Path to the fat JAR (default: <script-dir>/../trueref.jar)
|
||||||
|
# JAVA Path to the java binary (default: ${JAVA_HOME:-}/bin/java or `java` on PATH)
|
||||||
|
# JAVA_OPTS Extra JVM flags (e.g. -Xmx16g, -XX:+UseZGC)
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
JAR_DEFAULT="${SCRIPT_DIR}/../trueref.jar"
|
||||||
|
JAR="${TRUEREF_JAR:-$JAR_DEFAULT}"
|
||||||
|
|
||||||
|
if [[ ! -f "$JAR" ]]; then
|
||||||
|
echo "trueref: jar not found at $JAR" >&2
|
||||||
|
echo "trueref: set TRUEREF_JAR or place trueref.jar next to this script" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -n "${JAVA:-}" ]]; then
|
||||||
|
:
|
||||||
|
elif [[ -n "${JAVA_HOME:-}" && -x "${JAVA_HOME}/bin/java" ]]; then
|
||||||
|
JAVA="${JAVA_HOME}/bin/java"
|
||||||
|
else
|
||||||
|
JAVA="$(command -v java || true)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -z "${JAVA:-}" || ! -x "${JAVA}" ]]; then
|
||||||
|
echo "trueref: java not found; set JAVA_HOME or install JDK 21+" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ONNX Runtime CUDA EP needs cuDNN 9 on LD_LIBRARY_PATH. Many distros only ship
|
||||||
|
# cuDNN via the system package manager or via a Python wheel (nvidia-cudnn-cu12).
|
||||||
|
# If the user sets TRUEREF_CUDNN_LIB we trust it; otherwise we leave LD_LIBRARY_PATH
|
||||||
|
# alone and let CUDA fall back to CPU with a logged warning.
|
||||||
|
if [[ -n "${TRUEREF_CUDNN_LIB:-}" ]]; then
|
||||||
|
export LD_LIBRARY_PATH="${TRUEREF_CUDNN_LIB}${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
exec "$JAVA" \
|
||||||
|
--enable-native-access=ALL-UNNAMED \
|
||||||
|
--add-modules=jdk.incubator.vector \
|
||||||
|
${JAVA_OPTS:-} \
|
||||||
|
-jar "$JAR" \
|
||||||
|
"$@"
|
||||||
21
trueref-domain/pom.xml
Normal file
21
trueref-domain/pom.xml
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||||
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||||
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||||
|
<modelVersion>4.0.0</modelVersion>
|
||||||
|
|
||||||
|
<parent>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-parent</artifactId>
|
||||||
|
<version>0.1.0-SNAPSHOT</version>
|
||||||
|
</parent>
|
||||||
|
|
||||||
|
<artifactId>trueref-domain</artifactId>
|
||||||
|
<name>trueref-domain</name>
|
||||||
|
<description>Pure domain model + ports. No Spring, no I/O, no third-party libs beyond JSpecify.</description>
|
||||||
|
|
||||||
|
<!-- Hexagonal contract: domain has ZERO runtime dependencies beyond JSpecify (annotations only). -->
|
||||||
|
<dependencies>
|
||||||
|
<!-- inherits jspecify + test deps from parent -->
|
||||||
|
</dependencies>
|
||||||
|
</project>
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
package com.trueref.domain.error;
|
||||||
|
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
public final class IngestionFailed extends TrueRefException {
|
||||||
|
public IngestionFailed(String message, @Nullable Throwable cause) {
|
||||||
|
super("ingestion_failed", message, cause);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
package com.trueref.domain.error;
|
||||||
|
|
||||||
|
public final class InvalidSearchRequest extends TrueRefException {
|
||||||
|
public InvalidSearchRequest(String message) {
|
||||||
|
super("invalid_search_request", message, null);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
package com.trueref.domain.error;
|
||||||
|
|
||||||
|
public final class RepositoryAlreadyRegistered extends TrueRefException {
|
||||||
|
public RepositoryAlreadyRegistered(String name) {
|
||||||
|
super("repository_already_registered", "Repository already registered: " + name, null);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
package com.trueref.domain.error;
|
||||||
|
|
||||||
|
public final class RepositoryNotFound extends TrueRefException {
|
||||||
|
public RepositoryNotFound(String idOrName) {
|
||||||
|
super("repository_not_found", "Repository not found: " + idOrName, null);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
package com.trueref.domain.error;
|
||||||
|
|
||||||
|
public final class TagNotFound extends TrueRefException {
|
||||||
|
public TagNotFound(String repo, String tag) {
|
||||||
|
super("tag_not_found", "Tag not found in repository: " + repo + "@" + tag, null);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
package com.trueref.domain.error;
|
||||||
|
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
/** Root of all domain errors. Carries a stable string {@link #code()} for client localization. */
|
||||||
|
public abstract sealed class TrueRefException extends RuntimeException
|
||||||
|
permits RepositoryAlreadyRegistered,
|
||||||
|
RepositoryNotFound,
|
||||||
|
VersionNotFound,
|
||||||
|
VersionNotIndexed,
|
||||||
|
TagNotFound,
|
||||||
|
IngestionFailed,
|
||||||
|
InvalidSearchRequest {
|
||||||
|
|
||||||
|
private final String code;
|
||||||
|
|
||||||
|
protected TrueRefException(String code, String message, @Nullable Throwable cause) {
|
||||||
|
super(message, cause);
|
||||||
|
this.code = code;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String code() {
|
||||||
|
return code;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
package com.trueref.domain.error;
|
||||||
|
|
||||||
|
public final class VersionNotFound extends TrueRefException {
|
||||||
|
public VersionNotFound(String repo, String version) {
|
||||||
|
super("version_not_found", "Version not found: " + repo + "@" + version, null);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
package com.trueref.domain.error;
|
||||||
|
|
||||||
|
/** Thrown when a search request targets a known version that has not been indexed yet. */
|
||||||
|
public final class VersionNotIndexed extends TrueRefException {
|
||||||
|
public VersionNotIndexed(String repo, String version) {
|
||||||
|
super("version_not_indexed", "Version not yet indexed: " + repo + "@" + version, null);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
/**
|
||||||
|
* Sealed exception hierarchy for the domain. Adapters translate these to HTTP / JSON-RPC responses.
|
||||||
|
*/
|
||||||
|
@org.jspecify.annotations.NullMarked
|
||||||
|
package com.trueref.domain.error;
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A globally-deduplicated piece of content (function, class, markdown section, sliding-window
|
||||||
|
* fallback). Identified by {@link #contentHash()}: two chunks with the same hash are the same
|
||||||
|
* chunk, regardless of which repo/tag/file they originated from.
|
||||||
|
*
|
||||||
|
* @param symbol AST symbol name when applicable (e.g. function or class), null for prose chunks
|
||||||
|
*/
|
||||||
|
public record Chunk(
|
||||||
|
ChunkId id,
|
||||||
|
String contentHash,
|
||||||
|
String content,
|
||||||
|
String language,
|
||||||
|
@Nullable String symbol,
|
||||||
|
int tokenCount) {}
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
public record ChunkId(java.util.UUID value) {
|
||||||
|
public static ChunkId random() {
|
||||||
|
return new ChunkId(java.util.UUID.randomUUID());
|
||||||
|
}
|
||||||
|
|
||||||
|
public static ChunkId of(String s) {
|
||||||
|
return new ChunkId(java.util.UUID.fromString(s));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public String toString() {
|
||||||
|
return value.toString();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Many-to-many edge between a {@link Chunk} and a {@link Version}. Carries the location of the
|
||||||
|
* chunk inside the version's source tree.
|
||||||
|
*/
|
||||||
|
public record ChunkVersion(
|
||||||
|
ChunkId chunkId, VersionId versionId, String filePath, int startLine, int endLine) {}
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
/** Vector representation of a {@link Chunk}. Dense float vector; sparse channel deferred. */
|
||||||
|
public record Embedding(ChunkId chunkId, float[] vector) {
|
||||||
|
|
||||||
|
public Embedding {
|
||||||
|
// Defensive copy to make the record effectively immutable.
|
||||||
|
vector = vector.clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public float[] vector() {
|
||||||
|
return vector.clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
public int dimension() {
|
||||||
|
return vector.length;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A unit of orchestrated work. One job has many {@link JobStage stages} executed in sequence.
|
||||||
|
*
|
||||||
|
* @param versionId null for repo-level jobs (e.g. {@link JobType#DISCOVER_TAGS})
|
||||||
|
*/
|
||||||
|
public record IngestionJob(
|
||||||
|
JobId id,
|
||||||
|
RepositoryId repoId,
|
||||||
|
@Nullable VersionId versionId,
|
||||||
|
JobType type,
|
||||||
|
JobStatus status,
|
||||||
|
@Nullable Instant startedAt,
|
||||||
|
@Nullable Instant finishedAt,
|
||||||
|
List<JobStage> stages) {
|
||||||
|
|
||||||
|
public IngestionJob {
|
||||||
|
stages = List.copyOf(stages);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
public record JobId(java.util.UUID value) {
|
||||||
|
public static JobId random() {
|
||||||
|
return new JobId(java.util.UUID.randomUUID());
|
||||||
|
}
|
||||||
|
|
||||||
|
public static JobId of(String s) {
|
||||||
|
return new JobId(java.util.UUID.fromString(s));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public String toString() {
|
||||||
|
return value.toString();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
import java.time.Instant;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
/** A single emitted observability event for an ingestion job. Streamed via SSE to the UI. */
|
||||||
|
public record JobLogEvent(
|
||||||
|
JobId jobId,
|
||||||
|
Instant ts,
|
||||||
|
Level level,
|
||||||
|
JobStage.@Nullable StageName stage,
|
||||||
|
String message) {
|
||||||
|
|
||||||
|
public enum Level {
|
||||||
|
DEBUG,
|
||||||
|
INFO,
|
||||||
|
WARN,
|
||||||
|
ERROR
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
import java.time.Instant;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
public record JobStage(
|
||||||
|
JobId jobId,
|
||||||
|
StageName name,
|
||||||
|
StageStatus status,
|
||||||
|
@Nullable Instant startedAt,
|
||||||
|
@Nullable Instant finishedAt,
|
||||||
|
long itemsProcessed,
|
||||||
|
long itemsTotal,
|
||||||
|
long bytesProcessed,
|
||||||
|
@Nullable String errorMessage) {
|
||||||
|
|
||||||
|
public enum StageName {
|
||||||
|
CLONE,
|
||||||
|
FETCH,
|
||||||
|
CHECKOUT,
|
||||||
|
DISCOVER_FILES,
|
||||||
|
DIFF_FILES,
|
||||||
|
PARSE,
|
||||||
|
CHUNK,
|
||||||
|
EMBED,
|
||||||
|
INDEX,
|
||||||
|
COMMIT
|
||||||
|
}
|
||||||
|
|
||||||
|
public enum StageStatus {
|
||||||
|
PENDING,
|
||||||
|
RUNNING,
|
||||||
|
SUCCEEDED,
|
||||||
|
FAILED,
|
||||||
|
SKIPPED
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
public enum JobStatus {
|
||||||
|
QUEUED,
|
||||||
|
RUNNING,
|
||||||
|
SUCCEEDED,
|
||||||
|
FAILED,
|
||||||
|
CANCELLED
|
||||||
|
}
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
public enum JobType {
|
||||||
|
DISCOVER_TAGS,
|
||||||
|
INDEX_VERSION,
|
||||||
|
REFRESH,
|
||||||
|
COMPACT
|
||||||
|
}
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
import java.time.Duration;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A registered git repository (local or remote-cloned). The {@code localPath} is always present;
|
||||||
|
* for remote repositories it points to our managed clone directory and {@code managedClone} is true.
|
||||||
|
*
|
||||||
|
* @param remoteUrl git URL when {@code managedClone} is true; null otherwise
|
||||||
|
* @param ignoreGlobs per-repo globs ANDed with .gitignore + built-in defaults
|
||||||
|
* @param maxFileSizeBytes files larger than this are skipped during ingestion
|
||||||
|
* @param pollInterval scheduled fetch interval; {@link Duration#ZERO} disables polling
|
||||||
|
* @param tagCap max most-recent tags to auto-index; UI/MCP can index more on demand
|
||||||
|
* @param versionMappingRules ordered patterns mapping a client version (e.g. {@code "1.2.3"}) to a tag
|
||||||
|
*/
|
||||||
|
public record Repository(
|
||||||
|
RepositoryId id,
|
||||||
|
String name,
|
||||||
|
@Nullable String remoteUrl,
|
||||||
|
String localPath,
|
||||||
|
boolean managedClone,
|
||||||
|
List<String> ignoreGlobs,
|
||||||
|
long maxFileSizeBytes,
|
||||||
|
Duration pollInterval,
|
||||||
|
int tagCap,
|
||||||
|
List<TagPattern> versionMappingRules,
|
||||||
|
Instant createdAt,
|
||||||
|
Instant updatedAt) {
|
||||||
|
|
||||||
|
public Repository {
|
||||||
|
ignoreGlobs = List.copyOf(ignoreGlobs);
|
||||||
|
versionMappingRules = List.copyOf(versionMappingRules);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
/** Type-safe identifier for a registered repository. */
|
||||||
|
public record RepositoryId(java.util.UUID value) {
|
||||||
|
public static RepositoryId random() {
|
||||||
|
return new RepositoryId(java.util.UUID.randomUUID());
|
||||||
|
}
|
||||||
|
|
||||||
|
public static RepositoryId of(String s) {
|
||||||
|
return new RepositoryId(java.util.UUID.fromString(s));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public String toString() {
|
||||||
|
return value.toString();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
/** A single ranked snippet returned from a search. */
|
||||||
|
public record SearchHit(
|
||||||
|
ChunkId chunkId,
|
||||||
|
RepositoryId repoId,
|
||||||
|
VersionId versionId,
|
||||||
|
String repoName,
|
||||||
|
String tag,
|
||||||
|
String filePath,
|
||||||
|
int startLine,
|
||||||
|
int endLine,
|
||||||
|
String language,
|
||||||
|
@Nullable String symbol,
|
||||||
|
String content,
|
||||||
|
double score) {}
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Defines the (repo, version) scope of a search request. Multiple scopes can be ORed together so a
|
||||||
|
* single query may span "spring-boot v3.5.4" and "spring-boot v3.4.0", for example.
|
||||||
|
*/
|
||||||
|
public record SearchScope(List<RepoVersionRef> refs) {
|
||||||
|
|
||||||
|
public SearchScope {
|
||||||
|
refs = List.copyOf(refs);
|
||||||
|
}
|
||||||
|
|
||||||
|
public record RepoVersionRef(RepositoryId repoId, VersionId versionId) {}
|
||||||
|
}
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Strategy for mapping a client-supplied version string to a git tag in a repository. Patterns are
|
||||||
|
* tried in order; the first match wins. Built-in patterns: EXACT, V_PREFIX, RELEASE_PREFIX,
|
||||||
|
* SEMVER_FUZZY. CUSTOM allows a user-supplied template like {@code "release-{semver}"}.
|
||||||
|
*/
|
||||||
|
public sealed interface TagPattern {
|
||||||
|
|
||||||
|
/** {@code "1.2.3"} → tag {@code "1.2.3"}. */
|
||||||
|
record Exact() implements TagPattern {}
|
||||||
|
|
||||||
|
/** {@code "1.2.3"} → tag {@code "v1.2.3"}. */
|
||||||
|
record VPrefix() implements TagPattern {}
|
||||||
|
|
||||||
|
/** {@code "1.2.3"} → tag {@code "release-1.2.3"}. */
|
||||||
|
record ReleasePrefix() implements TagPattern {}
|
||||||
|
|
||||||
|
/** Any tag whose semver is closest to the requested version. */
|
||||||
|
record SemverFuzzy() implements TagPattern {}
|
||||||
|
|
||||||
|
/** Custom template containing {@code {version}} or {@code {semver}} placeholders. */
|
||||||
|
record Custom(String template) implements TagPattern {}
|
||||||
|
}
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
import java.time.Instant;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
/** A specific git tag (or branch) of a {@link Repository} that may be indexed independently. */
|
||||||
|
public record Version(
|
||||||
|
VersionId id,
|
||||||
|
RepositoryId repoId,
|
||||||
|
String tag,
|
||||||
|
String commitSha,
|
||||||
|
VersionStatus status,
|
||||||
|
@Nullable Instant indexedAt,
|
||||||
|
int chunkCount,
|
||||||
|
@Nullable String errorMessage) {}
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
public record VersionId(java.util.UUID value) {
|
||||||
|
public static VersionId random() {
|
||||||
|
return new VersionId(java.util.UUID.randomUUID());
|
||||||
|
}
|
||||||
|
|
||||||
|
public static VersionId of(String s) {
|
||||||
|
return new VersionId(java.util.UUID.fromString(s));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public String toString() {
|
||||||
|
return value.toString();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
package com.trueref.domain.model;
|
||||||
|
|
||||||
|
public enum VersionStatus {
|
||||||
|
/** Tag known but not yet indexed. */
|
||||||
|
DISCOVERED,
|
||||||
|
/** Indexing job currently running. */
|
||||||
|
INDEXING,
|
||||||
|
/** Successfully indexed and queryable. */
|
||||||
|
INDEXED,
|
||||||
|
/** Last indexing attempt failed; see {@link Version#errorMessage()}. */
|
||||||
|
FAILED,
|
||||||
|
/** Tag no longer exists upstream; chunks reclaimable by compaction. */
|
||||||
|
INACTIVE
|
||||||
|
}
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
/**
|
||||||
|
* Pure domain model for trueref. Contains records and enums describing repositories, versions,
|
||||||
|
* chunks, ingestion jobs, and search results. <strong>Must remain free of any I/O, Spring,
|
||||||
|
* Jackson, or other framework concerns.</strong> JSpecify nullability annotations are allowed.
|
||||||
|
*/
|
||||||
|
@org.jspecify.annotations.NullMarked
|
||||||
|
package com.trueref.domain.model;
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
package com.trueref.domain.port.in;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.RepositoryId;
|
||||||
|
import com.trueref.domain.model.Version;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
/** Use case: discover/refresh git tags of a repository. */
|
||||||
|
public interface DiscoverVersions {
|
||||||
|
|
||||||
|
/** Performs git fetch (if managed) + tag enumeration. Returns the now-known versions. */
|
||||||
|
List<Version> discover(RepositoryId repoId);
|
||||||
|
}
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
package com.trueref.domain.port.in;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.JobId;
|
||||||
|
import com.trueref.domain.model.RepositoryId;
|
||||||
|
import com.trueref.domain.model.VersionId;
|
||||||
|
|
||||||
|
/** Use case: schedule indexing of a specific (repo, tag/version). */
|
||||||
|
public interface IndexVersion {
|
||||||
|
|
||||||
|
/** Enqueues an INDEX_VERSION job. Returns immediately with the job id. */
|
||||||
|
JobId enqueue(RepositoryId repoId, VersionId versionId, boolean force);
|
||||||
|
}
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
package com.trueref.domain.port.in;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.IngestionJob;
|
||||||
|
import com.trueref.domain.model.JobId;
|
||||||
|
import com.trueref.domain.model.JobLogEvent;
|
||||||
|
import com.trueref.domain.model.JobStatus;
|
||||||
|
import com.trueref.domain.model.RepositoryId;
|
||||||
|
import com.trueref.domain.model.VersionId;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Optional;
|
||||||
|
import java.util.function.Consumer;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
/** Use case: read jobs and subscribe to job/log streams (for SSE in the UI). */
|
||||||
|
public interface ObserveJobs {
|
||||||
|
|
||||||
|
Optional<IngestionJob> findJob(JobId id);
|
||||||
|
|
||||||
|
List<IngestionJob> listJobs(
|
||||||
|
@Nullable RepositoryId repoId, @Nullable VersionId versionId, @Nullable JobStatus status, int limit);
|
||||||
|
|
||||||
|
/** Subscribes to live status updates of all jobs. Returns an unsubscribe handle. */
|
||||||
|
AutoCloseable subscribeJobs(Consumer<IngestionJob> listener);
|
||||||
|
|
||||||
|
/** Subscribes to log events of a single job. Returns an unsubscribe handle. */
|
||||||
|
AutoCloseable subscribeLogs(JobId jobId, Consumer<JobLogEvent> listener);
|
||||||
|
}
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
package com.trueref.domain.port.in;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.Repository;
|
||||||
|
import com.trueref.domain.model.RepositoryId;
|
||||||
|
import com.trueref.domain.model.Version;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Optional;
|
||||||
|
|
||||||
|
/** Use case: read-only access to repositories and their versions. */
|
||||||
|
public interface QueryCatalog {
|
||||||
|
|
||||||
|
List<Repository> listRepositories();
|
||||||
|
|
||||||
|
Optional<Repository> findRepository(RepositoryId id);
|
||||||
|
|
||||||
|
List<Version> listVersions(RepositoryId repoId);
|
||||||
|
}
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
package com.trueref.domain.port.in;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.Repository;
|
||||||
|
import com.trueref.domain.model.RepositoryId;
|
||||||
|
import com.trueref.domain.model.TagPattern;
|
||||||
|
import java.time.Duration;
|
||||||
|
import java.util.List;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
/** Use case: register a new repository (local path or remote URL). */
|
||||||
|
public interface RegisterRepository {
|
||||||
|
|
||||||
|
Repository register(Command cmd);
|
||||||
|
|
||||||
|
record Command(
|
||||||
|
String name,
|
||||||
|
@Nullable String remoteUrl,
|
||||||
|
@Nullable String localPath,
|
||||||
|
List<String> ignoreGlobs,
|
||||||
|
@Nullable Long maxFileSizeBytes,
|
||||||
|
@Nullable Duration pollInterval,
|
||||||
|
@Nullable Integer tagCap,
|
||||||
|
List<TagPattern> versionMappingRules) {
|
||||||
|
|
||||||
|
public Command {
|
||||||
|
ignoreGlobs = List.copyOf(ignoreGlobs);
|
||||||
|
versionMappingRules = List.copyOf(versionMappingRules);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void unregister(RepositoryId id);
|
||||||
|
}
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
package com.trueref.domain.port.in;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.RepositoryId;
|
||||||
|
import com.trueref.domain.model.VersionId;
|
||||||
|
import com.trueref.domain.model.VersionStatus;
|
||||||
|
import java.util.List;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Use case: turn a fuzzy library name (and optional version) into one or more concrete (repo,
|
||||||
|
* version) handles, ranked by relevance. Mirrors Context7's {@code resolve-library-id}.
|
||||||
|
*/
|
||||||
|
public interface ResolveLibraryId {
|
||||||
|
|
||||||
|
Result resolve(Query query);
|
||||||
|
|
||||||
|
record Query(String libraryName, @Nullable String query, @Nullable String version) {}
|
||||||
|
|
||||||
|
record Result(List<Match> matches) {
|
||||||
|
public Result {
|
||||||
|
matches = List.copyOf(matches);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
record Match(
|
||||||
|
RepositoryId repoId,
|
||||||
|
String libraryId, // "/owner/repo[/version]"
|
||||||
|
String name,
|
||||||
|
@Nullable String description,
|
||||||
|
int snippetCount,
|
||||||
|
List<VersionRef> availableVersions,
|
||||||
|
double score) {
|
||||||
|
|
||||||
|
public Match {
|
||||||
|
availableVersions = List.copyOf(availableVersions);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
record VersionRef(VersionId versionId, String tag, VersionStatus status) {}
|
||||||
|
}
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
package com.trueref.domain.port.in;
|
||||||
|
|
||||||
|
import com.trueref.domain.model.SearchHit;
|
||||||
|
import com.trueref.domain.model.SearchScope;
|
||||||
|
import java.util.List;
|
||||||
|
import org.jspecify.annotations.Nullable;
|
||||||
|
|
||||||
|
/** Use case: hybrid (BM25 + dense) search with rerank, scoped to specific (repo, version) pairs. */
|
||||||
|
public interface SearchLibraryDocs {
|
||||||
|
|
||||||
|
Result search(Query query);
|
||||||
|
|
||||||
|
record Query(
|
||||||
|
String text,
|
||||||
|
@Nullable String topic,
|
||||||
|
SearchScope scope,
|
||||||
|
int tokensBudget,
|
||||||
|
int maxHits) {}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @param hits ranked snippets, packed to fit within {@link Query#tokensBudget()}
|
||||||
|
* @param totalTokensReturned cumulative token count of returned snippets
|
||||||
|
*/
|
||||||
|
record Result(List<SearchHit> hits, int totalTokensReturned) {
|
||||||
|
|
||||||
|
public Result {
|
||||||
|
hits = List.copyOf(hits);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
/**
|
||||||
|
* Driving ports — interfaces implemented by the application layer and called by adapters
|
||||||
|
* (REST controllers, MCP tool handlers, scheduled tasks, etc.).
|
||||||
|
*/
|
||||||
|
@org.jspecify.annotations.NullMarked
|
||||||
|
package com.trueref.domain.port.in;
|
||||||
63
trueref-frontend/pom.xml
Normal file
63
trueref-frontend/pom.xml
Normal file
@@ -0,0 +1,63 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||||
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||||
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||||
|
<modelVersion>4.0.0</modelVersion>
|
||||||
|
|
||||||
|
<parent>
|
||||||
|
<groupId>com.trueref</groupId>
|
||||||
|
<artifactId>trueref-parent</artifactId>
|
||||||
|
<version>0.1.0-SNAPSHOT</version>
|
||||||
|
</parent>
|
||||||
|
|
||||||
|
<artifactId>trueref-frontend</artifactId>
|
||||||
|
<name>trueref-frontend</name>
|
||||||
|
<description>SvelteKit static UI built with frontend-maven-plugin and packaged as a resource jar.</description>
|
||||||
|
<packaging>jar</packaging>
|
||||||
|
|
||||||
|
<build>
|
||||||
|
<resources>
|
||||||
|
<!-- Point directly at the SvelteKit build output so resources:resources
|
||||||
|
(bound to process-resources) finds the files that npm-build already
|
||||||
|
created in generate-resources. The intermediate copy-frontend-build
|
||||||
|
step used target/frontend-dist as the resource directory, but that
|
||||||
|
directory is only populated later in process-resources, causing an
|
||||||
|
empty JAR on clean builds. -->
|
||||||
|
<resource>
|
||||||
|
<directory>web/build</directory>
|
||||||
|
<targetPath>static</targetPath>
|
||||||
|
</resource>
|
||||||
|
</resources>
|
||||||
|
<plugins>
|
||||||
|
<plugin>
|
||||||
|
<groupId>com.github.eirslett</groupId>
|
||||||
|
<artifactId>frontend-maven-plugin</artifactId>
|
||||||
|
<configuration>
|
||||||
|
<workingDirectory>web</workingDirectory>
|
||||||
|
<installDirectory>${project.build.directory}</installDirectory>
|
||||||
|
<nodeVersion>${node.version}</nodeVersion>
|
||||||
|
<npmVersion>${npm.version}</npmVersion>
|
||||||
|
</configuration>
|
||||||
|
<executions>
|
||||||
|
<execution>
|
||||||
|
<id>install-node-and-npm</id>
|
||||||
|
<goals><goal>install-node-and-npm</goal></goals>
|
||||||
|
<phase>generate-resources</phase>
|
||||||
|
</execution>
|
||||||
|
<execution>
|
||||||
|
<id>npm-install</id>
|
||||||
|
<goals><goal>npm</goal></goals>
|
||||||
|
<phase>generate-resources</phase>
|
||||||
|
<configuration><arguments>install</arguments></configuration>
|
||||||
|
</execution>
|
||||||
|
<execution>
|
||||||
|
<id>npm-build</id>
|
||||||
|
<goals><goal>npm</goal></goals>
|
||||||
|
<phase>generate-resources</phase>
|
||||||
|
<configuration><arguments>run build</arguments></configuration>
|
||||||
|
</execution>
|
||||||
|
</executions>
|
||||||
|
</plugin>
|
||||||
|
</plugins>
|
||||||
|
</build>
|
||||||
|
</project>
|
||||||
9
trueref-frontend/web/.gitignore
vendored
Normal file
9
trueref-frontend/web/.gitignore
vendored
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
node_modules
|
||||||
|
/build
|
||||||
|
/.svelte-kit
|
||||||
|
/package
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
|
!.env.example
|
||||||
|
.DS_Store
|
||||||
|
*.log
|
||||||
1
trueref-frontend/web/.npmrc
Normal file
1
trueref-frontend/web/.npmrc
Normal file
@@ -0,0 +1 @@
|
|||||||
|
engine-strict=false
|
||||||
9
trueref-frontend/web/.prettierrc
Normal file
9
trueref-frontend/web/.prettierrc
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
{
|
||||||
|
"useTabs": false,
|
||||||
|
"tabWidth": 2,
|
||||||
|
"singleQuote": true,
|
||||||
|
"trailingComma": "none",
|
||||||
|
"printWidth": 100,
|
||||||
|
"plugins": ["prettier-plugin-svelte"],
|
||||||
|
"overrides": [{ "files": "*.svelte", "options": { "parser": "svelte" } }]
|
||||||
|
}
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user