mozempk/trueref
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: trueref v0.1.0-SNAPSHOT
Some checks failed
Build and publish Docker image / Build and push (push) Failing after 1m27s
Details

Browse Source 
Java 21 / Spring Boot 3.5.3 multi-module Maven project.
Hybrid BM25+HNSW search with RRF, cross-encoder reranker,
ONNX Runtime 1.22.0 (CPU + CUDA 12 GPU variants).
This commit is contained in:
moze
2026-05-06 00:49:16 +02:00
commit c5f950c2c0
132 changed files with 11287 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

85
.gitea/workflows/docker.yml Normal file
View File

@@ -0,0 +1,85 @@
name: Build and publish Docker image
on:
push:
branches:
- main
- master
tags:
- 'v*.*.*'
workflow_dispatch:
jobs:
docker:
name: Build and push
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
# Set up Docker Buildx for efficient layer caching.
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
# Log in to the Gitea container registry.
# The built-in GITEA_TOKEN is injected automatically by Gitea Actions and
# has write access to packages in the same organisation/user namespace.
- name: Log in to Gitea registry
uses: docker/login-action@v3
with:
registry: git.sal.giize.com
username: ${{ gitea.actor }}
password: ${{ secrets.GITEA_TOKEN }}
# ── Determine tags ───────────────────────────────────────────────────
# On a version tag (v1.2.3): latest, cpu, cpu-1.2.3, 1.2.3
# On branch push (main/master): latest, cpu
- name: Docker metadata (CPU)
id: meta_cpu
uses: docker/metadata-action@v5
with:
images: git.sal.giize.com/mozempk/trueref
flavor: |
latest=auto
tags: |
type=raw,value=latest,enable={{is_default_branch}}
type=raw,value=cpu,enable={{is_default_branch}}
type=semver,pattern={{version}},prefix=cpu-
type=semver,pattern={{version}}
- name: Docker metadata (GPU)
id: meta_gpu
uses: docker/metadata-action@v5
with:
images: git.sal.giize.com/mozempk/trueref
flavor: |
latest=false
tags: |
type=raw,value=gpu,enable={{is_default_branch}}
type=semver,pattern={{version}},prefix=gpu-
# ── CPU image ────────────────────────────────────────────────────────
- name: Build and push CPU image
uses: docker/build-push-action@v5
with:
context: .
file: Dockerfile
push: true
tags: ${{ steps.meta_cpu.outputs.tags }}
labels: ${{ steps.meta_cpu.outputs.labels }}
cache-from: type=gha,scope=cpu
cache-to: type=gha,mode=max,scope=cpu
# ── GPU image ────────────────────────────────────────────────────────
# Built from the same source; only the runtime base image differs.
- name: Build and push GPU image
uses: docker/build-push-action@v5
with:
context: .
file: Dockerfile.gpu
push: true
tags: ${{ steps.meta_gpu.outputs.tags }}
labels: ${{ steps.meta_gpu.outputs.labels }}
cache-from: type=gha,scope=gpu
cache-to: type=gha,mode=max,scope=gpu

30
.gitignore vendored Normal file
View File

@@ -0,0 +1,30 @@
target/
build/
out/
.idea/
.vscode/
*.iml
*.ipr
*.iws
.DS_Store
# Maven
.mvn/wrapper/maven-wrapper.jar
# trueref runtime data (models, DB, index — too large / machine-specific)
data/
data-onnx-smoke/
logs/
# cuDNN and other large native runtime libraries
runtime/
# JVM crash dumps
hs_err_pid*.log
core.*
# Frontend
trueref-frontend/web/node_modules/
trueref-frontend/web/build/
trueref-frontend/web/.svelte-kit/
node_modules/

428
ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,428 @@
# trueref — Architecture
> Self-hosted, fat-JAR, Java-21 clone of [Context7](https://github.com/upstash/context7) ingestion + retrieval, with first-class differential per-tag indexing, embedded vector + BM25 store, ONNX-accelerated embeddings/rerank, Streamable-HTTP MCP server, REST + OpenAPI, and a SvelteKit UI.
## 1. Goals & Non-Goals
### Goals
- **Functional parity with Context7** ingestion outcome (own chunk schema).
- **Differential per-tag indexing**: every git tag of every registered repo is independently queryable.
- **Embedded everything**: single fat JAR runnable on a workstation/server. No external Postgres/Qdrant.
- **GPU-accelerated retrieval** via ONNX Runtime (CUDA Linux/Win, DirectML Win, CPU fallback).
- **MCP Streamable-HTTP server** exposing exactly two tools: `resolve-library-id`, `get-library-docs` — drop-in for any MCP client.
- **Full observability** of ingestion pipelines surfaced in the UI (live progress, log tail, history, timings, resource usage).
- **REST + OpenAPI/Swagger** for programmatic and UI use.
- **SvelteKit UI** for repo registration, indexing control, monitoring, and ad-hoc query.
- **Hexagonal architecture** so vector store, embedder, parser, persistence, etc. are swappable.
### Non-Goals
- No public hosted SaaS — self-host only.
- No model fine-tuning.
- No mobile app.
- No generative LLM in the pipeline (retrieval-only, like Context7).
- No multi-tenancy / auth (LAN-only deployment).
---
## 2. Tech Stack (locked)
| Concern | Choice | Rationale |
|---|---|---|
| Language / runtime | **Java 21 LTS** | Virtual threads stable; Spring Boot 3.5 supported. (Java 25 dropped — Boot 3.5 supports up to 23.) |
| Framework | **Spring Boot 3.5.x** + **Spring AI 1.0.x** | Web MVC + virtual-thread executor; Spring AI for embedding/MCP abstractions. |
| Build | **Maven** | Stable, ubiquitous, Spring-Boot first-class. |
| Metadata store | **H2 (MVCC mode, file-based)** + Flyway | Zero ops, JDBC, MVCC concurrency, fits fat JAR. |
| Vector + lexical store | **Apache Lucene 9.x** | Pure JVM. BM25 + HNSW kNN in one index. Collapses two stores. |
| Embedding model | **BAAI/bge-m3** (ONNX) | Multilingual, 8k context, dense+sparse capable. MIT-like license. |
| Reranker | **BAAI/bge-reranker-v2-m3** (ONNX) | Cross-encoder, Apache 2.0. |
| ML runtime | **ONNX Runtime** (`onnxruntime_gpu` Linux CUDA / `onnxruntime-directml` Win / `onnxruntime` CPU) | In-JVM via official Java bindings. |
| Git | **JGit** | Pure Java; clone, fetch, tag enumeration, diff. |
| Code parsing | **Pure-Java heuristic chunker** (markdown-aware, brace-balanced for C-family, indent-based for Python, sliding-window fallback) | No native deps; preserves fat-JAR purity. Tree-sitter is a documented future swap (see FINDINGS §F11). |
| Job orchestration | **Custom virtual-thread orchestrator** + H2-backed durable state | Fast, no Spring Batch overhead. |
| MCP server | **Spring AI MCP Server (Streamable HTTP)** | Spec 2025-03-26, single `/mcp` endpoint. |
| REST docs | **springdoc-openapi** | OpenAPI 3 + Swagger UI auto-generated. |
| Observability | **Micrometer + OpenTelemetry**, exposed via REST/SSE for UI. **Prometheus + Grafana optional** via `/actuator/prometheus`. | UI-first; Prom/Graf attach later. |
| Frontend | **SvelteKit + `@sveltejs/adapter-static`** | Built into `bootstrap/src/main/resources/static/`, served by Spring as part of fat JAR. |
| Packaging | **Single fat JAR** via `spring-boot-maven-plugin` | One artifact, embedded everything. |
---
## 3. Hexagonal Layout (Maven multi-module)
Direction of dependencies is enforced by Maven coordinates alone — no ArchUnit needed.
```
trueref-parent/ (pom; BOM + plugin management)
├── trueref-domain pure Java; records, sealed types, port interfaces. ZERO deps.
├── trueref-application use-case impls; depends on: domain
├── trueref-adapters ALL adapters live here; depends on: domain, application
│ └── com.trueref.adapter
│ ├── in
│ │ ├── rest @RestController + DTOs + OpenAPI + SSE
│ │ └── mcp MCP tool defs (Spring AI MCP server)
│ └── out
│ ├── persistence.h2 JdbcClient + Flyway, RepositoryStore impl
│ ├── vectorstore.lucene Lucene BM25 + HNSW kNN, ChunkStore impl
│ ├── embedding.onnx ONNX bge-m3 + bge-reranker-v2-m3
│ ├── git.jgit GitClient impl
│ ├── parsing.treesitter CodeParser impl
│ └── cache.disk EmbeddingCache (file-per-hash)
├── trueref-frontend SvelteKit; built via frontend-maven-plugin into static jar
└── trueref-bootstrap @SpringBootApplication; wires beans; produces fat JAR
depends on: domain, application, adapters, frontend
```
**Dependency rule (Maven-enforced):**
- `domain` → nothing.
- `application` → `domain`.
- `adapters` → `domain` + `application`.
- `frontend` → none (resource-only jar).
- `bootstrap` → all of the above (the only place wiring lives).
> All packages live under `com.trueref.*` regardless of module. Module boundaries enforce dependency direction; package layout inside `adapters` mirrors the in/out hexagonal convention.
---
## 4. Core Domain Model
```
Repository {
id: UUID
name: String // "spring-projects/spring-boot"
remoteUrl: String? // null if local-only
localPath: Path // either user-provided or our managed clone dir
managedClone: bool // true if WE clone/fetch
ignoreGlobs: List<String> // per-repo overrides
maxFileSizeBytes: long // default 1MB
pollIntervalSec: long // default 3600; 0 disables polling
versionMappingRules: List<TagPattern> // exact, v-prefix, release-prefix, regex
createdAt, updatedAt
}
Version {
id: UUID
repoId: UUID
tag: String // "v1.2.3" or branch name
commitSha: String
status: enum { DISCOVERED, INDEXING, INDEXED, FAILED, INACTIVE }
indexedAt: Instant?
chunkCount: int
errorMessage: String?
}
Chunk { // global, deduplicated by content_hash
id: UUID
contentHash: String // sha256 of canonicalized content
content: String // the snippet text
language: String // "java", "python", "markdown", ...
symbol: String? // function/class name if AST-extracted
tokenCount: int
// dense + sparse vectors stored in Lucene index, not here
}
ChunkVersion { // many-to-many: which versions contain which chunks
chunkId: UUID
versionId: UUID
filePath: String
startLine: int
endLine: int
// PK (chunkId, versionId, filePath, startLine)
}
IngestionJob {
id: UUID
repoId: UUID
versionId: UUID? // null = repo-level (e.g. discovery)
type: enum { DISCOVER_TAGS, INDEX_VERSION, COMPACT, REFRESH }
status: enum { QUEUED, RUNNING, SUCCEEDED, FAILED, CANCELLED }
startedAt, finishedAt
stages: List<JobStage>
}
JobStage {
jobId: UUID
name: enum { CLONE, FETCH, CHECKOUT, DISCOVER_FILES, PARSE, CHUNK, EMBED, INDEX, COMMIT }
status: enum { PENDING, RUNNING, SUCCEEDED, FAILED, SKIPPED }
startedAt, finishedAt
itemsProcessed: long
itemsTotal: long
bytesProcessed: long
errorMessage: String?
}
JobLogEvent { // ring-buffered + persisted; streamed via SSE
jobId: UUID
ts: Instant
level: enum { DEBUG, INFO, WARN, ERROR }
stage: JobStage.name?
message: String
}
```
---
## 5. Ingestion Pipeline
```
┌────────────────────────────────────────────────────────┐
│ IngestionOrchestrator (virtual-thread per stage) │
└────────────────────────────────────────────────────────┘
│
┌──────────────────────────┼──────────────────────────────────────────┐
▼ ▼ ▼
[CLONE/FETCH] [DISCOVER_TAGS] [INDEX_VERSION job]
JGit pull/clone git tag list ∩ (per (repo,tag))
version mapping
rules
│
┌────────────────────────────┤
▼ ▼
[CHECKOUT worktree] (parallel tags up to N)
│
▼
[DISCOVER_FILES]
respect .gitignore +
defaults + per-repo globs +
max file size
│
▼
[GIT_DIFF vs prev indexed tag]
→ if exists, only changed
files reach PARSE
│
▼
[PARSE] heuristic chunker
(markdown sections; brace-balanced;
indent-based; sliding-window fallback)
│
▼
[CHUNK] AST-aware splits +
sliding-window fallback
│
▼
[HASH + DEDUPE]
content_hash lookup → existing
chunkId reused
│
▼
[EMBED] ONNX bge-m3
NEW chunks only
(GPU semaphore-gated batch)
│
▼
[INDEX] Lucene upsert:
- chunk doc with vector
- chunk_version doc
│
▼
[COMMIT] Lucene commit +
H2 transaction
│
▼
Version.status = INDEXED
```
### Key invariants
1. **Embeddings are computed at most once per `content_hash`.** Persistent disk cache keyed by hash → vector bytes.
2. **A tag's chunks = union of (a) reused chunks via hash and (b) newly-embedded chunks.** This makes re-indexing a near-identical tag almost free.
3. **Git-diff fast path:** if a tag's parent (nearest previously indexed tag in semver order) exists, only files changed in `git diff parent..tag` are re-parsed. Unchanged files contribute their parent's chunk_versions verbatim with new line offsets adjusted by diff (or fully re-parsed if rename detection is ambiguous).
4. **Per-stage virtual-thread pools.** Threads themselves are unbounded (per user spec), but a **GPU semaphore** (default `permits = ortSessionCount`) gates ONNX inference to avoid GPU OOM. Lucene writer is single-thread (its own queue).
---
## 6. Search Pipeline
```
query ─► [Query Rewrite] rule-based: lowercase, dedupe stop tokens,
│ optional library-id-aware expansion
▼
[BM25 search] [Dense kNN search]
Lucene similarity Lucene HNSW (bge-m3 dense)
│ │
└─────────────► [RRF fusion] ◄──────┘
│
▼
top-K candidates (default 50)
│
▼
[Cross-encoder rerank]
ONNX bge-reranker-v2-m3
(GPU semaphore)
│
▼
[Token-budget assemble]
pack snippets up to `tokens` param
(default 5000, min 500, max 50000)
│
▼
ranked snippets w/ citations
(file path, repo, tag, lines)
```
All searches are **scoped** to `(repoId, versionId)` filter clauses on the Lucene index using `chunk_versions` join semantics.
---
## 7. MCP Server (Streamable HTTP)
- Single endpoint: `POST /mcp` (JSON-RPC over HTTP) with optional SSE upgrade per request, per [MCP 2025-03-26 spec](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/).
- **Two tools, exactly matching Context7 schema:**
### `resolve-library-id`
```json
{
"name": "resolve-library-id",
"description": "Resolves a library/package name to a trueref-compatible library ID...",
"inputSchema": {
"type": "object",
"required": ["libraryName"],
"properties": {
"libraryName": { "type": "string" },
"query": { "type": "string", "description": "optional, ranks results by relevance" }
}
}
}
```
Returns ranked candidate library IDs (`/{owner}/{repo}` style) with metadata (description, snippet count, available versions, source reputation).
### `get-library-docs`
```json
{
"name": "get-library-docs",
"inputSchema": {
"type": "object",
"required": ["libraryId"],
"properties": {
"libraryId": { "type": "string", "description": "/org/project[/version]" },
"topic": { "type": "string" },
"tokens": { "type": "integer", "minimum": 500, "maximum": 50000, "default": 5000 }
}
}
}
```
### On-demand indexing flow
- If `libraryId` includes a version that maps to a known git tag but is **not yet indexed**:
1. Enqueue `INDEX_VERSION` job immediately.
2. Return a **partial** response built from the **nearest indexed tag** (semver-closest) plus a status block: `{ "indexing": { "status": "in_progress", "version": "1.2.3", "retryAfterSec": 30 } }`.
- If version maps to **no** tag: return error `version_not_found` with the list of candidate tags discovered.
---
## 8. REST API Surface
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/repos` | List registered repos |
| POST | `/api/repos` | Register (local path or remote URL) |
| GET | `/api/repos/{id}` | Repo detail + version summary |
| DELETE | `/api/repos/{id}` | Unregister + soft-delete versions |
| POST | `/api/repos/{id}/discover` | Force tag discovery |
| GET | `/api/repos/{id}/versions` | All known versions + status |
| POST | `/api/repos/{id}/versions/{tag}/index` | Index a specific tag |
| POST | `/api/repos/{id}/versions/{tag}/reindex` | Force re-index |
| GET | `/api/jobs` | List jobs (filter by repo/version/status) |
| GET | `/api/jobs/{id}` | Job detail with stages |
| GET | `/api/jobs/{id}/log` (SSE) | Live log stream |
| GET | `/api/jobs/stream` (SSE) | Live job-status events for the dashboard |
| POST | `/api/search` | Hybrid search across one or more (repo, version) scopes |
| GET | `/api/resolve?q=react` | Library-ID resolution preview |
| GET | `/api/observability/metrics` | UI-friendly aggregated metrics JSON |
| GET | `/api/observability/resources` | Heap, GPU mem (via NVML when present), index size |
| GET | `/swagger-ui/index.html` | Swagger UI |
| GET | `/v3/api-docs` | OpenAPI JSON |
| ANY | `/mcp` | MCP Streamable HTTP endpoint |
| GET | `/actuator/prometheus` | Prometheus scrape (optional) |
| GET | `/**` | SPA fallback to `index.html` |
---
## 9. Concurrency & Performance
- **Virtual threads everywhere** for I/O (HTTP, JGit, file I/O, Lucene reads).
- **`Tomcat` configured with virtual-thread executor** (`spring.threads.virtual.enabled=true`).
- **Per-stage logical pools** are unbounded virtual-thread executors per orchestrator instance.
- **GPU access gated by a `Semaphore`** with permits = number of ONNX sessions (configurable, default = 2).
- **Lucene writer**: single `IndexWriter` instance protected by a queue; readers use a refresh-on-search `SearcherManager`.
- **Embedding cache**: file-per-hash on disk under `data/embedding-cache/`; hot LRU in memory.
- **Tag concurrency**: not capped (per spec), but each tag job awaits the GPU semaphore — natural backpressure.
---
## 10. Observability
- **Metrics** via Micrometer (`MeterRegistry`):
- Counters: chunks_embedded, chunks_reused, files_skipped, jobs_succeeded/failed.
- Timers: stage durations per stage name.
- Gauges: active_jobs, gpu_semaphore_available, lucene_index_size_bytes, heap_used.
- **OpenTelemetry traces** for every job (one trace per `IngestionJob`, span per `JobStage`).
- **JobEventBus**: in-process pub/sub. SSE controllers subscribe and push events to UI.
- **UI dashboards** (no Grafana required):
- "Live" tab: progress bars per running (repo, tag), per-stage throughput, log tail.
- "History" tab: paginated jobs table.
- "Stats" tab: per-stage timing histograms, chunk counts per repo/version, chunk dedupe ratio.
- "Resources" tab: heap, GPU memory (NVML where available), index size on disk.
- **Prometheus** scraping is opt-in (Actuator endpoint).
---
## 11. Storage Layout (on disk)
```
$TRUEREF_HOME/ # default: ./data
├── h2/ # H2 database files
├── lucene/ # single index dir; one Lucene writer
├── repos/ # managed clones (when managedClone=true)
│ └── <repoId>/...
├── embedding-cache/ # one file per content_hash → fp16 vector bytes
├── models/ # ONNX model files (auto-downloaded on first run)
│ ├── bge-m3/
│ └── bge-reranker-v2-m3/
└── logs/
```
---
## 12. Configuration (excerpt)
```yaml
trueref:
home: ${TRUEREF_HOME:./data}
ingestion:
poll-interval-default: 1h
tag-cap-default: 100 # most-recent N tags by semver/date
max-file-size-bytes-default: 1048576
embedding:
model: bge-m3
onnx-providers: [cuda, directml, cpu] # tried in order
session-count: 2 # = GPU semaphore permits
batch-size: 32
reranker:
model: bge-reranker-v2-m3
top-k: 50
search:
rrf-k: 60
final-top-k: 20
mcp:
tokens-default: 5000
tokens-min: 500
tokens-max: 50000
spring:
threads.virtual.enabled: true
```
---
## 13. Out-of-the-box behaviors locked from clarifications
- **Auth**: none (LAN-only) on REST and MCP.
- **Tag selection**: default cap 100 most-recent; on-demand index of any tag via UI search OR via MCP when an unindexed version is requested.
- **Differential indexing**: dedupe by `content_hash` AND skip unchanged files via `git diff parent..tag`.
- **Repo input**: UI-add (local path or remote URL) AND watched folder `./data/watched/` for bare repos.
- **Re-index trigger**: on-demand + scheduled `git fetch` poll (default 1h per repo).
- **Stale tag cleanup**: soft delete via `Version.status=INACTIVE`; compaction job reclaims orphan chunks.
- **Embedding cache**: persistent on disk, keyed by `content_hash`.
- **Concurrency**: unbounded virtual threads, GPU semaphore-gated.
See [FINDINGS.md](FINDINGS.md) for research backing each choice.

91
CODE_STYLE.md Normal file
View File

@@ -0,0 +1,91 @@
# trueref — Code Style
## 1. Language & Toolchain
- **Java 21**, source/target 21.
- **Maven** with `spring-boot-maven-plugin` for the fat JAR.
- **Spotless** with **Palantir Java Format** for formatting (4-space indent, 120 col).
- **ErrorProne** + **NullAway** for static analysis. NullAway annotations: `@org.jspecify.annotations.Nullable` / `@NonNull`.
- Hexagonal boundaries are enforced by **Maven module dependencies** (no ArchUnit). See ARCHITECTURE §3.
## 2. Records, Sealed Types, Pattern Matching
- Prefer **records** for DTOs, value objects, and `port.in`/`port.out` parameter/result types.
- Use **sealed interfaces** for closed result/event hierarchies (`sealed interface IngestionEvent permits ...`).
- Use **pattern matching** (`switch` expressions, `instanceof`) over visitor pattern.
## 3. Nullability & Optional
- All API surfaces (public methods on ports, REST DTOs) are **non-null by default**; mark nullable explicitly with `@Nullable`.
- Use `Optional<T>` **only** as a return type from query-style methods. Never as a field, never as a parameter.
## 4. Concurrency
- Spawn virtual threads via `Thread.ofVirtual().start(...)` or `Executors.newVirtualThreadPerTaskExecutor()`. **Never** call `Thread.sleep` inside a synchronized block.
- Shared mutable state is forbidden in `domain` and discouraged in `application`. When unavoidable, use `java.util.concurrent.atomic.*` or a `ReentrantLock`.
- GPU work goes through `GpuSemaphore.acquire()` (a thin wrapper around `Semaphore`).
- Long-running orchestration uses **structured concurrency** (`StructuredTaskScope`) where it improves cancellation safety.
## 5. Error Handling
- **Domain errors** are sealed exception hierarchies rooted at `TrueRefException`. Adapters translate them to HTTP/JSON-RPC errors centrally (REST: `@ControllerAdvice`; MCP: dedicated translator).
- **No checked exceptions** at port boundaries. Wrap third-party checked exceptions at the adapter edge.
- Validation errors carry a stable `code` (string) so the UI can localize.
- Never `catch (Exception e)` and swallow. Either log + rethrow as a domain exception or let it propagate.
## 6. Logging
- **SLF4J** with parameterized messages: `log.info("indexed tag {} of repo {}", tag, repoName);` — never string-concatenate.
- Structured fields via MDC: `repoId`, `versionId`, `jobId`, `stage`. Cleared in a try/finally.
- Log levels:
- `ERROR`: unrecoverable, requires operator attention.
- `WARN`: degraded, automatic recovery in progress.
- `INFO`: lifecycle events (job started/finished, repo registered).
- `DEBUG`: per-file, per-chunk detail. Off by default.
## 7. Naming
- Use cases (`port.in`): imperative verb phrases — `IndexVersion`, `ResolveLibraryId`.
- SPIs (`port.out`): noun-ish role names — `EmbeddingService`, `ChunkStore`, `GitClient`.
- Adapter classes: `<Tech><Role>` — `LuceneChunkStore`, `OnnxEmbeddingService`, `JGitClient`.
- DTOs: `<Resource><Action>Request` / `<Resource>Response`.
- Records' field names are camelCase (no Hungarian, no `_` prefixes).
## 8. Package & File Discipline
- One public type per file.
- Internal helpers are package-private. Avoid `public` unless used across packages.
- Domain packages export **only** records and interfaces. No Spring annotations, no Lombok, no Jackson annotations.
- Adapter packages may use Spring stereotypes (`@Component`, `@Repository`, `@RestController`) but adapters depend on **port interfaces only** when interacting with the application.
## 9. Spring Wiring
- Wiring lives in `bootstrap`. Each adapter package may define a `@Configuration` (constructor-injected `@Bean` factories) but **does not** auto-`@ComponentScan` itself; bootstrap explicitly imports.
- Use `@ConfigurationProperties` records for typed config; never raw `@Value`.
- Prefer constructor injection. **No field injection.**
## 10. Persistence (H2)
- Migrations under `src/main/resources/db/migration` named `V<N>__<snake_case>.sql`. Flyway runs at startup.
- All access via Spring **`JdbcClient`** (Spring Boot 3.2+, fluent JDBC). **No JPA/Hibernate**, no `JdbcTemplate` directly.
- Mappers are explicit `RowMapper<T>` lambdas, not reflection-based.
- SQL lives next to the repository class, either as `static final String` constants or in `*.sql` files loaded via `ClassPathResource` for non-trivial queries.
## 11. REST
- Controllers in `adapter.in.rest`. They depend only on `port.in` interfaces and DTO records.
- DTOs are **separate** from domain records. Mapping via plain `static of(...)` factories. **No MapStruct.**
- All endpoints documented with `@Operation`, `@ApiResponses`, `@Schema` (springdoc).
- Request validation via `jakarta.validation` annotations on DTOs.
- SSE endpoints return `SseEmitter`; subscribe to `JobEventBus`, unsubscribe on completion/timeout.
## 12. MCP
- Tool definitions are **records** decorated to produce JSON Schema via Spring AI's MCP support. Schema strings stay verbatim (1:1 with Context7) so LLMs see identical contracts.
- Tool handlers depend only on `port.in` (`SearchLibraryDocs`, `ResolveLibraryId`).
## 13. Tests
- **JUnit 5** + **AssertJ** + **Mockito** (sparingly).
- Unit tests live next to the package they test. Integration tests under `src/test/java/.../it/` and use `@SpringBootTest`.
- Use **Testcontainers** only when truly required (we mostly avoid it via embedded stores).
- ArchUnit test suite is mandatory and runs in CI.
## 14. Dependency Hygiene
- BOM-managed versions only. Add a dependency only if it provides clear value over JDK + Spring Boot + already-included libs.
- No Lombok, no Guava (use JDK 21 equivalents), no Reactor (we use virtual threads + blocking).
- No Kotlin, no Scala.
## 15. Documentation
- All architectural decisions go in **ARCHITECTURE.md**.
- All research notes go in **FINDINGS.md** with sources.
- All conventions go in **this file**.
- Per-package `package-info.java` may exist for non-trivial packages, summarizing role and exported types.
- No README sprawl: `README.md` is a quickstart only and links to the three docs above.

51
Dockerfile Normal file
View File

@@ -0,0 +1,51 @@
# ─── Build stage ──────────────────────────────────────────────────────────────
# eclipse-temurin:21-jdk-jammy ships JDK 21 + Maven-compatible toolchain.
# frontend-maven-plugin downloads Node/npm automatically, so no explicit
# Node install is needed in the build stage.
FROM eclipse-temurin:21-jdk-jammy AS builder
RUN apt-get update \
&& apt-get install -y --no-install-recommends maven \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /build
COPY . .
RUN mvn -q package -DskipTests -T 1C
# ─── Runtime stage (CPU-only) ─────────────────────────────────────────────────
FROM eclipse-temurin:21-jre-jammy
LABEL org.opencontainers.image.title="TrueRef"
LABEL org.opencontainers.image.description="Self-hosted documentation retrieval platform for AI coding assistants (CPU variant)"
LABEL org.opencontainers.image.url="https://git.sal.giize.com/mozempk/trueref"
LABEL org.opencontainers.image.source="https://git.sal.giize.com/mozempk/trueref"
WORKDIR /app
COPY --from=builder /build/trueref-bootstrap/target/trueref.jar /app/trueref.jar
# /data is the default trueref.home: H2 DB, Lucene index, embedding cache and
# downloaded models all live here. Mount a volume to persist between restarts.
VOLUME /data
ENV TRUEREF_HOME=/data \
TRUEREF_PORT=18080 \
JAVA_OPTS=""
EXPOSE 18080
# JVM flags required by trueref:
# --enable-native-access silences FFM Linker warning from DJL tokenizers
# --add-modules enables Lucene 10 SIMD codepath (incubator.vector)
# Spring properties are passed via CMD so users can override them at runtime.
ENTRYPOINT ["sh", "-c", \
"exec java \
--enable-native-access=ALL-UNNAMED \
--add-modules=jdk.incubator.vector \
${JAVA_OPTS} \
-jar /app/trueref.jar \
--server.port=${TRUEREF_PORT} \
--trueref.home=${TRUEREF_HOME} \
--trueref.embedding.onnx-providers=cpu \
\"$@\"", "--"]

69
Dockerfile.gpu Normal file
View File

@@ -0,0 +1,69 @@
# ─── Build stage ──────────────────────────────────────────────────────────────
FROM eclipse-temurin:21-jdk-jammy AS builder
RUN apt-get update \
&& apt-get install -y --no-install-recommends maven \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /build
COPY . .
RUN mvn -q package -DskipTests -T 1C
# ─── Runtime stage (NVIDIA GPU / CUDA 12 + cuDNN 9) ──────────────────────────
# nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04 ships:
# - CUDA 12.4 runtime libs (libcuda.so, libcublas, etc.)
# - cuDNN 9 (cu12 build) required by ONNX Runtime CUDA execution provider
#
# Prerequisites on the Docker host:
# - NVIDIA GPU driver ≥ 550 (CUDA 12.4 compatible)
# - nvidia-container-toolkit installed and configured
#
# Run with: docker run --gpus all --device /dev/nvidia0 ...
FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04
LABEL org.opencontainers.image.title="TrueRef (GPU)"
LABEL org.opencontainers.image.description="Self-hosted documentation retrieval platform for AI coding assistants (NVIDIA GPU / CUDA 12 variant)"
LABEL org.opencontainers.image.url="https://git.sal.giize.com/mozempk/trueref"
LABEL org.opencontainers.image.source="https://git.sal.giize.com/mozempk/trueref"
# Install Eclipse Temurin 21 JRE onto the CUDA base image.
RUN apt-get update \
&& apt-get install -y --no-install-recommends wget apt-transport-https gnupg \
&& wget -q -O - https://packages.adoptium.net/artifactory/api/gpg/key/public \
| gpg --dearmor -o /usr/share/keyrings/adoptium.gpg \
&& echo "deb [signed-by=/usr/share/keyrings/adoptium.gpg] https://packages.adoptium.net/artifactory/deb jammy main" \
> /etc/apt/sources.list.d/adoptium.list \
&& apt-get update \
&& apt-get install -y --no-install-recommends temurin-21-jre \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY --from=builder /build/trueref-bootstrap/target/trueref.jar /app/trueref.jar
VOLUME /data
ENV TRUEREF_HOME=/data \
TRUEREF_PORT=18080 \
# Physical GPU index visible inside the container (0 after --gpus all remapping).
TRUEREF_GPU=0 \
# 0 = unbounded arena; set to e.g. 8589934592 (8 GiB) on shared hosts.
TRUEREF_MEM_LIMIT=0 \
JAVA_OPTS="" \
# CUDA_DEVICE_ORDER ensures nvidia-smi numbering matches CUDA runtime numbering.
CUDA_DEVICE_ORDER=PCI_BUS_ID
EXPOSE 18080
ENTRYPOINT ["sh", "-c", \
"exec java \
--enable-native-access=ALL-UNNAMED \
--add-modules=jdk.incubator.vector \
${JAVA_OPTS} \
-jar /app/trueref.jar \
--server.port=${TRUEREF_PORT} \
--trueref.home=${TRUEREF_HOME} \
--trueref.embedding.gpu-device-id=${TRUEREF_GPU} \
--trueref.embedding.gpu-mem-limit-bytes=${TRUEREF_MEM_LIMIT} \
\"$@\"", "--"]

207
FINDINGS.md Normal file
View File

@@ -0,0 +1,207 @@
# trueref — Findings
Research notes backing the choices in [ARCHITECTURE.md](ARCHITECTURE.md). Each section ends with a verdict and follow-up questions if any.
---
## F1. Context7 ingestion behavior (what we replicate functionally)
- Context7 ingests git repositories and crawls associated docs sites driven by a `context7.json` manifest at the repo root, plus an optional `llms.txt` index.
- It produces snippets shaped roughly as `{ title, description, source, code, language }` and serves them via two MCP tools: `resolve-library-id` and `get-library-docs`.
- The `get-library-docs` API accepts `topic` and `tokens` parameters; topic biases retrieval, tokens caps the response size (defaults observed in client docs: ~5000).
- Source: upstash/context7 GitHub repo & MCP docs.
**Verdict:** functional parity is achievable without copying the manifest schema. Our chunk model captures the same fields under different names (`symbol`/`content`/`filePath`/`language`). MCP tool signatures are kept **byte-identical** for LLM compatibility.
---
## F2. Embedded vector store choice — Lucene 9 over Qdrant
- Qdrant is a Rust binary; embedding it in a fat JAR requires extracting & spawning a child process, contradicting the "single JAR, embedded everything" goal.
- **Apache Lucene ≥9.0** ships HNSW kNN (`KnnFloatVectorField`) alongside BM25 in a single index segment. Pure JVM, no native deps.
- Lucene supports **filtered kNN** (`KnnFloatVectorQuery` with a `BooleanQuery` filter), which we need for `(repoId, versionId)` scoping.
- Trade-off: Lucene HNSW lacks Qdrant's payload-rich filtering tricks (e.g. quantization presets, named vectors). Acceptable for our scale; we get BM25 in the same store for free.
**Verdict:** Lucene 9 (we'll target the latest 9.x). One `IndexWriter`, refresh-on-search via `SearcherManager`.
---
## F3. Embedding model — bge-m3
- BAAI/bge-m3: 568M params, 8192 ctx, multilingual (100+ langs), trained on multi-functionality (dense + sparse + colbert).
- ONNX export available (BAAI provides it; community variants on HuggingFace).
- License: MIT-style (model weights), works for self-hosted commercial use.
- Vector dim: 1024 (dense). Sparse vocab compatible with Lucene if we want SPLADE-like sparse — out of scope for v1.
**Verdict:** bge-m3 (dense only for v1). Sparse channel deferred.
---
## F4. Reranker — bge-reranker-v2-m3
- Cross-encoder, scores (query, passage) pairs.
- Same family as embedder: balanced quality/cost, ONNX-exportable.
- Apache 2.0 license.
**Verdict:** bge-reranker-v2-m3. Top-K candidates from RRF fed in, top-N (default 20) returned.
---
## F5. ML runtime — ONNX Runtime (Java bindings)
- ONNX Runtime has **official Java bindings** (`com.microsoft.onnxruntime:onnxruntime` + `onnxruntime_gpu`).
- Execution providers we will support:
- **CUDA** (`onnxruntime_gpu`): Linux + Windows with NVIDIA driver ≥ matching CUDA 12.x.
- **DirectML** (`onnxruntime-directml`): Windows, any DX12 GPU.
- **CPU**: always-on fallback.
- ONNX Runtime has **no Vulkan execution provider**. Our earlier "Vulkan fallback" wish is not satisfiable in this stack — we drop it.
- Generative LLMs in ONNX (e.g. Phi-3.5-mini) are possible but awkward (KV cache management, tokenizer differences). Since we picked **retrieval-only**, no generative model is needed.
**Verdict:** ONNX Runtime, providers tried in order: cuda → directml → cpu. Vulkan dropped (documented).
---
## F6. Java version — 21 LTS, not 25
- Spring Boot 3.5.x officially supports Java 17–23.
- Spring AI 1.0.x targets the same range.
- Java 25 is supported by neither at time of writing; risking obscure reflection/MR-JAR issues with downstream libs (JGit, Lucene, ONNX bindings).
- Java 21 is LTS and has stable virtual threads + structured concurrency (`StructuredTaskScope` was preview through 23, finalizing soon — we'll guard usage behind a thin wrapper to ease later upgrade).
**Verdict:** Java 21 LTS. Re-evaluate to 25 once Spring Boot certifies it.
---
## F7. Differential indexing scheme
- We chose **dedupe-by-content-hash** AND **git-diff-driven file skipping**.
- The hash dedupe alone gives constant-cost embeddings for unchanged code across tags.
- The git-diff path additionally avoids parsing/chunking unchanged files, which dominates ingest CPU on large repos.
- Storage model:
- `chunks`: one row per unique `content_hash`. Vector lives in Lucene keyed by `chunkId`.
- `chunk_versions`: many-to-many; one row per `(chunk, version, file, line range)`.
- Search: `BooleanQuery(filter=chunk_versions.version_id IN scope)` joined to vector field.
- The chunk dedupe ratio is reported as a UI metric — it's the most intuitive measure of "differential" effectiveness.
**Verdict:** confirmed; both mechanisms compose without conflict.
---
## F8. MCP transport — Streamable HTTP
- The current MCP spec (revision 2025-03-26) defines **Streamable HTTP**: a single `POST /mcp` endpoint that may upgrade to SSE for long-lived/streamed responses; replaces the deprecated 2024-11-05 SSE transport.
- Spring AI 1.0 ships an MCP server module that supports Streamable HTTP via Spring MVC.
- We expose **only** Streamable HTTP, no SSE-only legacy endpoint (per user spec).
**Verdict:** Streamable HTTP only at `/mcp`.
---
## F9. Embedded SQL store — H2 (MVCC)
- H2 in MVCC mode supports concurrent readers and a single writer with row-level locking. Good enough for our metadata write rates (jobs, versions, chunk_versions).
- File-based, single JAR dependency, JDBC.
- Considered & rejected:
- **DuckDB**: column-store, slower OLTP, no good Flyway story.
- **SQLite**: poor concurrency under write load.
- **Embedded Postgres (zonky)**: pulls a 100+ MB native binary per OS — fights the fat JAR goal.
**Verdict:** H2 file-based, MVCC=true, with Flyway migrations.
---
## F10. Job orchestration — custom virtual-thread orchestrator
- Spring Batch is feature-rich but requires a JobRepository (typically Postgres or H2) and adds startup cost we don't need.
- Our jobs are **per-tag**, **simple linear stage sequences**, with persistence-of-status as the only durability requirement.
- Custom orchestrator: each `IngestionJob` runs on a virtual thread; stages execute sequentially; stage transitions are durably written to H2 in a transaction; `JobEventBus` emits events for SSE.
- Crash recovery: on startup, scan jobs in `RUNNING` status, mark them `FAILED` (or resume specific resumable stages — v2).
**Verdict:** custom orchestrator. Spring Batch deferred unless we hit a ceiling.
---
## F11. Code parser — pure-Java heuristic for v1, tree-sitter pluggable for v2
The Java tree-sitter ecosystem in 2026 is fragmented:
- **`io.github.tree-sitter:jtreesitter`** uses Project Panama FFI → requires **Java 22+**. We target Java 21 LTS, so this is out.
- **`io.github.bonede:tree-sitter`** is JNI-based and works on Java 21, but bundling per-OS (linux/windows/mac × x64/arm64) native grammar binaries for many languages bloats the fat JAR significantly and creates a packaging matrix we don't want to maintain in v1.
- **`ai.serenade.treesitter:java-tree-sitter`** is unmaintained.
**Decision (v1):** ship a pure-Java heuristic `CodeParser` adapter. Strategies, tried in order per file:
1. **Markdown / `.txt` / `.rst`**: split by ATX/Setext headings; large sections further split by paragraph.
2. **Brace-balanced languages** (java, c, c++, c#, go, rust, js, ts, kotlin, scala, swift): walk the file tracking brace depth + line-based heuristics (function signatures, top-level declarations) to extract chunks of complete top-level constructs. Symbol name extracted via a tiny regex per language.
3. **Indent-based languages** (python, yaml, ruby): split on top-level `def`/`class`/`module` boundaries; symbol name from the declaration line.
4. **Fallback** (any text file): sliding-window of N lines (default 80) with M lines overlap (default 10).
The `CodeParser` port is unchanged. A future tree-sitter implementation (when JDK upgrade or upstream packaging matures) can be swapped in by providing an alternate `@Component` and toggling a config flag — that's exactly what hexagonal architecture buys us.
**Verdict:** pure-Java heuristic parser for v1; tree-sitter remains a documented future enhancement.
---
## F12. Concurrency caps & GPU contention
- User chose **unbounded virtual threads**. This is safe for I/O-bound stages.
- ONNX inference is GPU-bound; calling the same `OrtSession` from many threads concurrently is unsupported. Two mitigations:
1. A **session pool** of size N (config `embedding.session-count`, default 2).
2. A **`Semaphore(N)`** acquired by any caller before invoking inference. Pool & semaphore sizes match.
- This means tag-level parallelism is naturally throttled by GPU capacity without explicit per-tag limits.
**Verdict:** session pool + semaphore. Document the knob clearly in `application.yml`.
---
## F13. Frontend in fat JAR
- SvelteKit `@sveltejs/adapter-static` produces a fully static bundle (HTML/CSS/JS). We build it as a Maven sub-step (frontend-maven-plugin) and copy `frontend/build/` to `bootstrap/src/main/resources/static/`. Spring serves it by default.
- SPA fallback: a `WebMvcConfigurer` maps all unmatched non-API paths to `index.html` so client-side routing works.
**Verdict:** static adapter + Spring static-resource serving. Single artifact preserved.
---
## F14. Open questions / future work
1. **Sparse channel** (bge-m3 sparse / SPLADE) for stronger lexical recall — deferred to v2.
2. **Per-language reranker fine-tuning** — out of scope (no fine-tuning, per spec).
3. **Compaction job** to truly delete orphan chunks (currently soft-delete on versions). Schedule TBD.
4. **Watched-folder** auto-discovery semantics: how often do we rescan `./data/watched/`? Default proposal: every 5 min + on filesystem watch event (Java NIO `WatchService`).
5. **Repo size cap**: do we need a maximum total cloned size to prevent runaway disk use? Currently unlimited; could add per-repo and global caps in v2.
6. **GPU memory introspection**: Linux NVML via JNI (`jnvml`) for GPU mem gauges; on Windows + DirectML we surface only "available/in-use" booleans.
---
## F15. References (for re-checking when libraries bump)
- Context7 repo & MCP tool surface — to sanity-check schema fidelity on releases.
- Spring AI 1.0.x release notes — verify MCP server Streamable HTTP module name & API.
- Spring Boot 3.5.x release notes — confirm Java version compatibility window.
- Lucene 9.x kNN docs — confirm filtered vector query API surface.
- ONNX Runtime Java release notes — confirm CUDA/DirectML EP availability per version.
- BAAI/bge-m3 model card — confirm ONNX export availability/format.
- MCP spec 2025-03-26 — Streamable HTTP transport requirements.
> Use the Context7 MCP lookup skill before bumping any of the above to fetch fresh, version-specific docs.
---
## F16. Smoke-test log (2026-04-21)
End-to-end smoke after first assembly:
- `mvn -pl trueref-bootstrap -am package` → BUILD SUCCESS, fat JAR ~582 MB.
- `mvn test` → **16 tests pass** (parser 6, pooling 5, disk cache 5), **0 failures**.
- `java -jar trueref-bootstrap/target/trueref.jar --trueref.embedding.session-count=0` — started in 3.6 s.
- `GET /actuator/health` → `UP` (db H2, disk, ping, ssl).
- `POST /api/repos` + `GET /api/repos` — round-trips a repo.
- `GET /swagger-ui.html` → 302 redirect (to `/swagger-ui/index.html`), `GET /v3/api-docs` → 200.
- `GET /` → 200 (SvelteKit SPA served from Spring static resources).
- `POST /mcp` one-shot JSON-RPC returns HTTP 500 — expected, the WebMVC MCP transport requires an SSE session established by `GET /sse` first; MCP clients that implement the Streamable-HTTP spec do this automatically. Verified MCP tools register: `tools/list` handler is reached (error thrown is transport-level session lookup, not bean wiring).
Fixes landed during smoke:
- `V1__init_schema.sql`: H2 in PostgreSQL mode rejects `AUTO_INCREMENT`. Switched `job_log_events.id` to `BIGINT GENERATED BY DEFAULT AS IDENTITY` and removed the explicit `NULL` constraint.
- `OnnxProperties.sessionCount` can now be 0 (disables the ONNX stack, for environments where models aren't available); `GpuSemaphore` accepts 0 permits by internally using 1 (never acquired in disabled mode).
- `OnnxEmbeddingService` / `OnnxRerankerService` short-circuit in disabled mode; reranker pass-through preserves input order.
- `ApplicationBeans` exposes only concrete beans (not both the class and its interface) to avoid ambiguous autowiring.

21
README.md Normal file
View File

@@ -0,0 +1,21 @@
# trueref
Self-hosted [Context7](https://github.com/upstash/context7) clone in Java 21 + Spring Boot 3.5: indexes git repositories per tag, exposes a Streamable-HTTP MCP server, REST + Swagger, and a SvelteKit dashboard for ingestion observability and querying.
See:
- [ARCHITECTURE.md](ARCHITECTURE.md) — design, hexagonal layout, pipelines, MCP/REST surfaces.
- [CODE_STYLE.md](CODE_STYLE.md) — conventions.
- [FINDINGS.md](FINDINGS.md) — research notes backing every choice.
## Quickstart
```bash
./mvnw -DskipTests package
java -jar trueref-bootstrap/target/trueref.jar
```
Browse:
- UI: http://localhost:8080/
- Swagger: http://localhost:8080/swagger-ui.html
- MCP endpoint: http://localhost:8080/mcp
- Actuator: http://localhost:8080/actuator

198
pom.xml Normal file
View File

@@ -0,0 +1,198 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.trueref</groupId>
<artifactId>trueref-parent</artifactId>
<version>0.1.0-SNAPSHOT</version>
<packaging>pom</packaging>
<name>trueref</name>
<description>Self-hosted Context7-style library docs indexer + MCP server</description>
<modules>
<module>trueref-domain</module>
<module>trueref-application</module>
<module>trueref-adapters</module>
<module>trueref-frontend</module>
<module>trueref-bootstrap</module>
</modules>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.5.3</version>
<relativePath/>
</parent>
<properties>
<java.version>21</java.version>
<maven.compiler.release>21</maven.compiler.release>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<spring-ai.version>1.0.0</spring-ai.version>
<springdoc.version>2.8.6</springdoc.version>
<jgit.version>7.3.0.202506031305-r</jgit.version>
<lucene.version>10.4.0</lucene.version>
<onnxruntime.version>1.22.0</onnxruntime.version>
<huggingface-tokenizers.version>0.33.0</huggingface-tokenizers.version>
<h2.version>2.3.232</h2.version>
<flyway.version>11.8.2</flyway.version>
<jspecify.version>1.0.0</jspecify.version>
<assertj.version>3.26.3</assertj.version>
<!-- Plugins -->
<spotless.version>2.43.0</spotless.version>
<frontend-maven-plugin.version>1.15.1</frontend-maven-plugin.version>
<node.version>v20.18.0</node.version>
<npm.version>10.8.2</npm.version>
</properties>
<dependencyManagement>
<dependencies>
<!-- Internal modules -->
<dependency>
<groupId>com.trueref</groupId>
<artifactId>trueref-domain</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.trueref</groupId>
<artifactId>trueref-application</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.trueref</groupId>
<artifactId>trueref-adapters</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.trueref</groupId>
<artifactId>trueref-frontend</artifactId>
<version>${project.version}</version>
</dependency>
<!-- Spring AI BOM -->
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-bom</artifactId>
<version>${spring-ai.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<!-- 3rd-party -->
<dependency>
<groupId>org.springdoc</groupId>
<artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
<version>${springdoc.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.jgit</groupId>
<artifactId>org.eclipse.jgit</artifactId>
<version>${jgit.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.jgit</groupId>
<artifactId>org.eclipse.jgit.ssh.apache</artifactId>
<version>${jgit.version}</version>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-core</artifactId>
<version>${lucene.version}</version>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-analysis-common</artifactId>
<version>${lucene.version}</version>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-queryparser</artifactId>
<version>${lucene.version}</version>
</dependency>
<dependency>
<groupId>com.microsoft.onnxruntime</groupId>
<artifactId>onnxruntime</artifactId>
<version>${onnxruntime.version}</version>
</dependency>
<dependency>
<groupId>com.microsoft.onnxruntime</groupId>
<artifactId>onnxruntime_gpu</artifactId>
<version>${onnxruntime.version}</version>
</dependency>
<dependency>
<groupId>ai.djl.huggingface</groupId>
<artifactId>tokenizers</artifactId>
<version>${huggingface-tokenizers.version}</version>
</dependency>
<dependency>
<groupId>org.jspecify</groupId>
<artifactId>jspecify</artifactId>
<version>${jspecify.version}</version>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<version>${assertj.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.jspecify</groupId>
<artifactId>jspecify</artifactId>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>com.diffplug.spotless</groupId>
<artifactId>spotless-maven-plugin</artifactId>
<version>${spotless.version}</version>
<configuration>
<java>
<palantirJavaFormat/>
<removeUnusedImports/>
<importOrder/>
<trimTrailingWhitespace/>
<endWithNewline/>
</java>
</configuration>
</plugin>
<plugin>
<groupId>com.github.eirslett</groupId>
<artifactId>frontend-maven-plugin</artifactId>
<version>${frontend-maven-plugin.version}</version>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<release>${java.version}</release>
<parameters>true</parameters>
</configuration>
</plugin>
</plugins>
</build>
</project>

611
tests/quality/phaser_rag_eval.py Normal file
View File

@@ -0,0 +1,611 @@
#!/usr/bin/env python3
"""
Phaser RAG Quality Evaluation Suite
====================================
Simulates an LLM querying TrueRef for Phaser documentation and guidance.
Tests are designed to be hard and objective: each defines exact expected content
fragments and/or expected source files that MUST appear in the top-k results.
Scoring metrics per test:
file@1 - expected file appeared as hit #1
file@3 - expected file appeared in hits 1-3
file@5 - expected file appeared in hits 1-5
content@5 - at least one expected content fragment found across the top-5 hits combined
content@1 - expected content fragment found in hit #1
Overall suite scores:
MRR - Mean Reciprocal Rank (file position)
P@1..5 - Precision@k for file hits
C@5 - Content recall across top-5
Run:
python3 phaser_rag_eval.py [--base-url http://localhost:18080] [--verbose]
"""
import argparse
import json
import sys
import time
from dataclasses import dataclass, field
from typing import Optional
import urllib.request
import urllib.error
# ---------------------------------------------------------------------------
# Config
# ---------------------------------------------------------------------------
REPO_ID = "50010965-aa3f-45f4-bb8d-72a0d50bf0db"
# Version IDs pinned to specific tags (fetched at startup if not found)
VERSIONS = {
"v4.1.0": "6c6a00f5-0945-4fd7-b62c-c0e69f14effe",
"v3.88.0": "d032d4d4-e6bc-4c9d-9c3c-8853e4a1cdc9",
"v3.85.2": "d1cf906e-54b9-416f-bd5b-9432d69d9935",
"v3.60.0": "95d0a8e2-9071-4986-85d4-59ae97893353",
}
# ---------------------------------------------------------------------------
# Test definition
# ---------------------------------------------------------------------------
@dataclass
class TestCase:
id: str
name: str
query: str
version: str # key into VERSIONS
topic: Optional[str] = None
expected_files: list[str] = field(default_factory=list) # substrings of filePath
expected_content: list[str] = field(default_factory=list) # substrings that MUST appear
required_content: list[str] = field(default_factory=list) # ALL of these must appear (stricter)
max_hits: int = 10
tokens_budget: int = 6000
# Optional: minimum rerank score the top hit should exceed
min_score: Optional[float] = None
# ---------------------------------------------------------------------------
# Test definitions — 25 hard, objective cases
# ---------------------------------------------------------------------------
TESTS: list[TestCase] = [
# ── 1. Tween system: basic config properties ──────────────────────────
TestCase(
id="T01",
name="Tween config: yoyo/hold/repeatDelay properties",
query="What properties can I set in a TweenBuilderConfig to make a tween yoyo with a hold and repeat delay?",
version="v4.1.0",
topic="tweens",
expected_files=["tweens/builders/TweenBuilder.js", "tweens/typedefs"],
expected_content=["yoyo", "hold", "repeatDelay"],
required_content=["yoyo", "repeatDelay"],
),
# ── 2. Tween system: onComplete / onUpdate callbacks ──────────────────
TestCase(
id="T02",
name="Tween callbacks: onComplete and onUpdate signatures",
query="How do I use onComplete and onUpdate callbacks in a Phaser tween? What arguments do they receive?",
version="v4.1.0",
topic="tweens",
expected_files=["tweens/"],
expected_content=["onComplete", "onUpdate", "onStart"],
required_content=["onComplete"],
),
# ── 3. Arcade physics: setCollideWorldBounds signature ────────────────
TestCase(
id="T03",
name="Arcade physics: setCollideWorldBounds signature",
query="What are the parameters of setCollideWorldBounds in Phaser Arcade physics? Can I pass bounceX and bounceY to set bounce on world edges?",
version="v4.1.0",
topic="physics",
expected_files=["physics/arcade/Body.js"],
expected_content=["setCollideWorldBounds", "bounceX", "bounceY", "onWorldBounds"],
required_content=["setCollideWorldBounds", "bounceX"],
),
# ── 4. Arcade physics: addCollider vs addOverlap ──────────────────────
TestCase(
id="T04",
name="Arcade physics: addCollider vs addOverlap difference",
query="What is the difference between addCollider and addOverlap in Phaser's Arcade physics World? How do I add a callback?",
version="v4.1.0",
topic="physics",
expected_files=["physics/arcade/World.js"],
expected_content=["addCollider", "addOverlap", "collideCallback", "processCallback"],
required_content=["addCollider", "addOverlap"],
),
# ── 5. Camera: shake parameters ───────────────────────────────────────
TestCase(
id="T05",
name="Camera shake: duration, intensity, force, callback",
query="How do I make the camera shake in Phaser? What parameters does camera.shake accept?",
version="v4.1.0",
topic="camera",
expected_files=["cameras/2d/Camera.js"],
expected_content=["shake", "duration", "intensity", "force", "callback"],
required_content=["shake", "intensity"],
),
# ── 6. Camera: startFollow with lerp ─────────────────────────────────
TestCase(
id="T06",
name="Camera follow: startFollow lerpX lerpY parameters",
query="How do I make the Phaser camera follow a player with smooth lerp? What are the lerpX and lerpY parameters?",
version="v4.1.0",
topic="camera",
expected_files=["cameras/2d/Camera.js"],
expected_content=["startFollow", "lerpX", "lerpY", "roundPixels"],
required_content=["startFollow", "lerpX"],
),
# ── 7. Camera: setDeadzone ────────────────────────────────────────────
TestCase(
id="T07",
name="Camera deadzone: setDeadzone width/height",
query="How does camera deadzone work in Phaser? How do I create a rectangular deadzone so the camera only moves when the player exits it?",
version="v4.1.0",
topic="camera",
expected_files=["cameras/2d/Camera.js"],
expected_content=["setDeadzone", "deadzone"],
required_content=["setDeadzone"],
),
# ── 8. Scene: pass data when starting another scene ───────────────────
TestCase(
id="T08",
name="Scene management: pass data on scene.start",
query="How do I pass data to another scene when calling scene.start() or scene.launch()? How does the init method receive it?",
version="v4.1.0",
topic="scenes",
expected_files=["scene/"],
expected_content=["init", "data", "start", "launch"],
required_content=["init"],
),
# ── 9. Animation system: chaining animations ──────────────────────────
TestCase(
id="T09",
name="Animation chaining: chain() and playAfterRepeat()",
query="How can I chain multiple animations so one plays after another finishes in Phaser? What is the chain() method?",
version="v4.1.0",
topic="animations",
expected_files=["gameobjects/sprite/Sprite.js", "animations/"],
expected_content=["chain", "playAfterRepeat", "playAfterDelay"],
required_content=["chain"],
),
# ── 10. Animation system: events ─────────────────────────────────────
TestCase(
id="T10",
name="Animation events: ANIMATION_COMPLETE, ANIMATION_START",
query="What events does the Phaser animation system emit? How do I listen for when an animation completes on a specific sprite?",
version="v4.1.0",
topic="animations",
expected_files=["animations/events/"],
expected_content=["ANIMATION_COMPLETE", "ANIMATION_START", "ANIMATION_STOP"],
required_content=["ANIMATION_COMPLETE"],
),
# ── 11. Input: pointer events ─────────────────────────────────────────
TestCase(
id="T11",
name="Input: setInteractive + pointerdown/pointerover events",
query="How do I call setInteractive on a game object and listen for pointerdown and pointerover events in Phaser?",
version="v4.1.0",
topic="input",
expected_files=["input/"],
expected_content=["pointerdown", "pointerover", "pointerout", "setInteractive"],
required_content=["setInteractive", "pointerdown"],
),
# ── 12. Input: keyboard cursor keys ──────────────────────────────────
TestCase(
id="T12",
name="Input: createCursorKeys and keyboard key states",
query="How do I read arrow key input in Phaser? How does createCursorKeys() work and how do I check if a key is down?",
version="v4.1.0",
topic="input",
expected_files=["input/keyboard/"],
expected_content=["createCursorKeys", "isDown", "up", "down", "left", "right"],
required_content=["createCursorKeys"],
),
# ── 13. Loader: atlas and texture keys ───────────────────────────────
TestCase(
id="T13",
name="Loader: load.atlas config object and frame keys",
query="How do I load a texture atlas in Phaser? What are the arguments to this.load.atlas() and how do I use frame keys?",
version="v4.1.0",
topic="loader",
expected_files=["loader/filetypes/AtlasJSONFile.js", "loader/"],
expected_content=["atlas", "textureURL", "atlasURL", "frameConfig"],
required_content=["atlas"],
min_score=0.7,
),
# ── 14. Tilemaps: setCollisionBetween ────────────────────────────────
TestCase(
id="T14",
name="Tilemap: setCollisionBetween start/stop parameters",
query="How do I set collision on a range of tile indices in a Phaser tilemap? What does setCollisionBetween do?",
version="v4.1.0",
topic="tilemaps",
expected_files=["tilemaps/Tilemap.js", "tilemaps/"],
expected_content=["setCollisionBetween", "start", "stop", "collides", "recalculateFaces"],
required_content=["setCollisionBetween"],
),
# ── 15. Tilemaps: createFromObjects ──────────────────────────────────
TestCase(
id="T15",
name="Tilemap: createFromObjects from Tiled object layer",
query="How do I convert Tiled object layer objects into Phaser game objects? How does createFromObjects work?",
version="v4.1.0",
topic="tilemaps",
expected_files=["tilemaps/Tilemap.js"],
expected_content=["createFromObjects", "objectLayerName"],
required_content=["createFromObjects"],
),
# ── 16. RenderTexture: beginDraw / endDraw (v3 API) ──────────────────
TestCase(
id="T16",
name="RenderTexture v3: beginDraw / batchDraw / endDraw pattern",
query="How do I use beginDraw and endDraw on a Phaser RenderTexture for batch drawing? What is the workflow?",
version="v3.85.2",
topic="rendering",
expected_files=["textures/DynamicTexture.js"],
expected_content=["beginDraw", "endDraw", "batchDraw", "batchDrawFrame"],
required_content=["beginDraw", "endDraw"],
),
# ── 17. Masking: BitmapMask vs GeometryMask (v3 API) ──────────────────
TestCase(
id="T17",
name="Masking v3: createBitmapMask vs createGeometryMask",
query="What is the difference between a BitmapMask and a GeometryMask in Phaser? How do I create and apply them?",
version="v3.85.2",
topic="rendering",
expected_files=["gameobjects/components/Mask.js", "display/mask/"],
expected_content=["createBitmapMask", "createGeometryMask", "setMask", "BitmapMask", "GeometryMask"],
required_content=["BitmapMask", "GeometryMask"],
),
# ── 18. Groups: getFirstDead / getFirstAlive pool pattern ─────────────
TestCase(
id="T18",
name="Group: object pool with getFirstDead / getFirstAlive",
query="How do I implement an object pool in Phaser using a Group? What are getFirstDead and getFirstAlive?",
version="v4.1.0",
topic="gameobjects",
expected_files=["gameobjects/group/Group.js"],
expected_content=["getFirstDead", "getFirstAlive", "createIfNull", "countActive"],
required_content=["getFirstDead", "getFirstAlive"],
),
# ── 19. Matter.js: fromVertices custom body shape ─────────────────────
TestCase(
id="T19",
name="Matter.js: custom body shape with fromVertices",
query="How do I create a custom polygon physics body from vertices in Phaser's Matter.js physics?",
version="v4.1.0",
topic="physics",
expected_files=["physics/matter-js/Factory.js", "physics/matter-js/"],
expected_content=["fromVertices", "vertexSets", "options"],
required_content=["fromVertices"],
),
# ── 20. Game config: FPS limit / target ───────────────────────────────
TestCase(
id="T20",
name="Game config: fps.target and fps.limit settings",
query="How do I configure the target frame rate and FPS limit in the Phaser game config? What is the difference between target and limit?",
version="v4.1.0",
topic="core",
expected_files=["core/TimeStep.js", "core/Config.js"],
expected_content=["targetFps", "fpsLimit", "target", "fps"],
required_content=["targetFps"],
),
# ── 21. Scale Manager: ScaleModes ────────────────────────────────────
TestCase(
id="T21",
name="Scale Manager: FIT vs ENVELOP scale modes",
query="What scale modes are available in Phaser's Scale Manager? How does FIT differ from ENVELOP? How do I make a responsive game?",
version="v4.1.0",
topic="scale",
expected_files=["scale/"],
expected_content=["FIT", "ENVELOP", "ScaleManager", "autoCenter"],
required_content=["FIT"],
),
# ── 22. Data Manager: set/get/events ──────────────────────────────────
TestCase(
id="T22",
name="Data Manager: set/get and CHANGE_DATA event",
query="How does the Phaser Data Manager work? How do I watch for data changes using events on a game object's data?",
version="v4.1.0",
topic="data",
expected_files=["data/DataManager.js", "data/"],
expected_content=["CHANGE_DATA", "set", "get", "events"],
required_content=["CHANGE_DATA"],
),
# ── 23. Depth sort: setDepth and displayList ──────────────────────────
TestCase(
id="T23",
name="Depth sorting: setDepth and display list ordering",
query="How does Phaser handle rendering order (z-order)? How do I use setDepth to control which objects render on top?",
version="v4.1.0",
topic="rendering",
expected_files=["gameobjects/"],
expected_content=["setDepth", "depth", "displayList"],
required_content=["setDepth"],
),
# ── 24. Version diff: v3.60 TweenChain (new in 3.60) ─────────────────
TestCase(
id="T24",
name="Version-specific: TweenChain introduced in v3.60",
query="How do I create a sequence of tweens that play one after another using TweenChain in Phaser 3.60+?",
version="v3.60.0",
topic="tweens",
expected_files=["tweens/"],
expected_content=["TweenChain", "chain"],
required_content=["TweenChain"],
),
# ── 25. Hard adversarial: camera.ignore() ─────────────────────────────
TestCase(
id="T25",
name="Camera: ignore() to exclude game objects from a camera",
query="How do I make a game object invisible to a specific camera in Phaser while remaining visible to others? What is camera.ignore()?",
version="v4.1.0",
topic="camera",
expected_files=["cameras/2d/"],
expected_content=["ignore", "camera"],
required_content=["ignore"],
),
]
# ---------------------------------------------------------------------------
# HTTP helpers
# ---------------------------------------------------------------------------
def post_json(url: str, payload: dict) -> dict:
body = json.dumps(payload).encode()
req = urllib.request.Request(
url, data=body,
headers={"Content-Type": "application/json", "Accept": "application/json"},
method="POST",
)
with urllib.request.urlopen(req, timeout=30) as resp:
return json.loads(resp.read().decode())
def get_json(url: str) -> dict | list:
req = urllib.request.Request(url, headers={"Accept": "application/json"})
with urllib.request.urlopen(req, timeout=15) as resp:
return json.loads(resp.read().decode())
# ---------------------------------------------------------------------------
# Evaluation logic
# ---------------------------------------------------------------------------
@dataclass
class TestResult:
test: TestCase
hits: list[dict]
elapsed_ms: float
error: Optional[str] = None
# Computed below
file_rank: Optional[int] = None # 1-based rank of first expected-file match
content_ranks: list[int] = field(default_factory=list) # 1-based ranks where content found
required_found: bool = False
top_score: Optional[float] = None
def file_at(self, k: int) -> bool:
return self.file_rank is not None and self.file_rank <= k
def content_at(self, k: int) -> bool:
return any(r <= k for r in self.content_ranks)
def mrr(self) -> float:
if self.file_rank is None:
return 0.0
return 1.0 / self.file_rank
def summary_line(self) -> str:
f1 = "✓" if self.file_at(1) else "·"
f3 = "✓" if self.file_at(3) else "·"
f5 = "✓" if self.file_at(5) else "·"
c5 = "✓" if self.content_at(5) else "·"
req = "✓" if self.required_found else "✗"
rank_str = f"rank={self.file_rank}" if self.file_rank else "NOT FOUND"
score_str = f"score={self.top_score:.3f}" if self.top_score else ""
ms_str = f"{self.elapsed_ms:.0f}ms"
return (
f"[{self.test.id}] {self.test.name[:52]:<52} "
f"f@1={f1} f@3={f3} f@5={f5} c@5={c5} req={req} "
f"{rank_str:>12} {score_str} {ms_str}"
)
def evaluate(result: TestResult, verbose: bool = False) -> None:
hits = result.hits
if not hits:
return
result.top_score = hits[0].get("score") if hits else None
# File rank: position of first hit whose filePath matches any expected_files substring
for i, hit in enumerate(hits):
fp = hit.get("filePath", "")
if any(ef in fp for ef in result.test.expected_files):
result.file_rank = i + 1
break
# Content rank: for each expected_content fragment, find the first hit that contains it
combined_content = {i: (hit.get("content") or "") for i, hit in enumerate(hits)}
for fragment in result.test.expected_content:
for i, content in combined_content.items():
if fragment.lower() in content.lower():
result.content_ranks.append(i + 1)
break
# Required content: ALL required fragments must appear somewhere in top-10
all_content = " ".join(combined_content.values()).lower()
result.required_found = all(
r.lower() in all_content for r in result.test.required_content
)
if verbose:
print(f"\n{'─'*80}")
print(f"[{result.test.id}] {result.test.name}")
print(f" Query: {result.test.query}")
print(f" Expected files: {result.test.expected_files}")
print(f" Expected content: {result.test.expected_content}")
print(f" Top hits:")
for i, hit in enumerate(hits[:5]):
fp = hit.get("filePath", "?")
score = hit.get("score", 0.0)
snip = (hit.get("content") or "")[:100].replace("\n", " ")
marker = " ← FILE MATCH" if any(ef in fp for ef in result.test.expected_files) else ""
print(f" [{i+1}] score={score:.3f} {fp}{marker}")
print(f" {snip}")
# ---------------------------------------------------------------------------
# Main runner
# ---------------------------------------------------------------------------
def run(base_url: str, verbose: bool) -> None:
base_url = base_url.rstrip("/")
search_url = f"{base_url}/api/search"
versions_url = f"{base_url}/api/repos/{REPO_ID}/versions"
print(f"TrueRef Phaser RAG Evaluation Suite")
print(f"Server : {base_url}")
print(f"Tests : {len(TESTS)}")
print()
# Resolve version IDs from server (in case they differ)
try:
all_versions = get_json(versions_url)
live_map = {v["tag"]: v["id"] for v in all_versions if v.get("status") == "INDEXED"}
for tag in list(VERSIONS.keys()):
if tag in live_map:
VERSIONS[tag] = live_map[tag]
except Exception as e:
print(f"WARN: could not refresh version IDs: {e}")
results: list[TestResult] = []
for tc in TESTS:
version_id = VERSIONS.get(tc.version)
if not version_id:
print(f"SKIP [{tc.id}]: version {tc.version} not available")
continue
payload = {
"text": tc.query,
"scope": [{"repoId": REPO_ID, "versionId": version_id}],
"maxHits": tc.max_hits,
"tokensBudget": tc.tokens_budget,
}
if tc.topic:
payload["topic"] = tc.topic
t0 = time.time()
try:
resp = post_json(search_url, payload)
elapsed = (time.time() - t0) * 1000
hits = resp.get("hits", [])
tr = TestResult(test=tc, hits=hits, elapsed_ms=elapsed)
evaluate(tr, verbose=verbose)
except Exception as e:
elapsed = (time.time() - t0) * 1000
tr = TestResult(test=tc, hits=[], elapsed_ms=elapsed, error=str(e))
print(f"ERROR [{tc.id}]: {e}")
results.append(tr)
# ── Summary table ─────────────────────────────────────────────────────
print()
print("=" * 110)
print(f"{'TEST ID + NAME':<56} {'f@1':>4} {'f@3':>4} {'f@5':>4} {'c@5':>4} {'req':>4} {'file rank':>12} {'score':>10} {'ms':>6}")
print("=" * 110)
for tr in results:
if tr.error:
print(f"[{tr.test.id}] {'ERROR: ' + tr.test.name[:45]:<52} ERROR: {tr.error[:40]}")
else:
print(tr.summary_line())
# ── Aggregate metrics ─────────────────────────────────────────────────
valid = [tr for tr in results if not tr.error]
n = len(valid)
if n == 0:
print("\nNo valid results.")
return
mrr = sum(tr.mrr() for tr in valid) / n
p_at_1 = sum(1 for tr in valid if tr.file_at(1)) / n
p_at_3 = sum(1 for tr in valid if tr.file_at(3)) / n
p_at_5 = sum(1 for tr in valid if tr.file_at(5)) / n
content_at5 = sum(1 for tr in valid if tr.content_at(5)) / n
req_recall = sum(1 for tr in valid if tr.required_found) / n
avg_ms = sum(tr.elapsed_ms for tr in valid) / n
print("=" * 110)
print()
print("Aggregate metrics:")
print(f" MRR (file) : {mrr:.4f} ({mrr*100:.1f}%)")
print(f" Precision@1 (file) : {p_at_1:.4f} ({p_at_1*100:.1f}%)")
print(f" Precision@3 (file) : {p_at_3:.4f} ({p_at_3*100:.1f}%)")
print(f" Precision@5 (file) : {p_at_5:.4f} ({p_at_5*100:.1f}%)")
print(f" Content recall@5 : {content_at5:.4f} ({content_at5*100:.1f}%)")
print(f" Required recall : {req_recall:.4f} ({req_recall*100:.1f}%) ← hardest: ALL required fragments in top-10")
print(f" Avg query latency : {avg_ms:.0f} ms")
print()
# ── Failure analysis ──────────────────────────────────────────────────
failures = [tr for tr in valid if not tr.file_at(5) or not tr.required_found]
if failures:
print(f"Improvement targets ({len(failures)} tests below par):")
for tr in failures:
issues = []
if not tr.file_at(5):
issues.append(f"file not in top-5 (rank={tr.file_rank})")
if not tr.required_found:
missing = [r for r in tr.test.required_content
if r.lower() not in " ".join(h.get("content","") for h in tr.hits).lower()]
issues.append(f"required content missing: {missing}")
print(f" [{tr.test.id}] {tr.test.name}: {'; '.join(issues)}")
else:
print("All tests passed file@5 and required-content checks.")
# Exit code: 0 if MRR ≥ 0.5 AND required recall ≥ 0.8, else 1
if mrr >= 0.5 and req_recall >= 0.8:
print("\nResult: PASS")
sys.exit(0)
else:
print(f"\nResult: FAIL (MRR={mrr:.3f} threshold=0.5, req_recall={req_recall:.3f} threshold=0.8)")
sys.exit(1)
# ---------------------------------------------------------------------------
# Entry point
# ---------------------------------------------------------------------------
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Phaser RAG quality evaluation")
parser.add_argument("--base-url", default="http://localhost:18080",
help="TrueRef server base URL")
parser.add_argument("--verbose", action="store_true",
help="Print per-test hit details")
args = parser.parse_args()
run(args.base_url, args.verbose)

115
trueref Executable file
View File

@@ -0,0 +1,115 @@
#!/usr/bin/env bash
# trueref launcher (workspace root)
#
# Wraps the fat JAR with:
# - --enable-native-access=ALL-UNNAMED (silences FFM Linker warning from DJL tokenizers)
# - --add-modules=jdk.incubator.vector (enables Lucene 10 SIMD codepath)
# - cuDNN 9 (cu12 build) on LD_LIBRARY_PATH so ONNX Runtime CUDA EP loads
# - CUDA_VISIBLE_DEVICES isolation so ORT BFC arena doesn't trip over the second card
# - per-session GPU memory cap so embedder + reranker fit on one card
#
# Defaults are tuned for this machine (LMDE 7, CUDA 12.4, RTX 2080 SUPER + RTX 3060).
# Override anything via env vars or by appending Spring properties to the command line.
#
# Usage:
# ./trueref # run with defaults (port 18080)
# ./trueref --server.port=8080 # forward Spring properties
# TRUEREF_GPU=0 ./trueref # use the 2080 SUPER instead# TRUEREF_GPU=cpu ./trueref # disable CUDA, run on CPU
# TRUEREF_HOME=/data/trueref ./trueref # custom data dir
#
# Env vars:
# TRUEREF_GPU GPU index (matches `nvidia-smi -L`) or "cpu". Default: 1
# TRUEREF_HOME Data directory. Default: ./data
# TRUEREF_PORT HTTP port. Default: 18080
# TRUEREF_MEM_LIMIT Per-session GPU mem cap in bytes. Default: 0 (unbounded).
# With session-count=1 there is no multi-session contention, so the BFC
# arena can grow freely — capping it risks running out of budget during
# model-weight loading (~1.5-2 GB) before inference even starts.
# Set to e.g. 8589934592 (8 GiB) only if you run multiple pools on one card.
# TRUEREF_CUDNN_LIB Directory containing libcudnn.so.9. Default: ./runtime/cudnn/nvidia/cudnn/lib
# TRUEREF_JAR Path to the fat JAR. Default: ./trueref-bootstrap/target/trueref.jar
# JAVA java binary. Default: $JAVA_HOME/bin/java or `java` on PATH
# JAVA_OPTS Extra JVM flags (e.g. -Xmx16g)
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
JAR="${TRUEREF_JAR:-$ROOT/trueref-bootstrap/target/trueref.jar}"
GPU="${TRUEREF_GPU:-1}"
HOME_DIR="${TRUEREF_HOME:-$ROOT/data}"
PORT="${TRUEREF_PORT:-18080}"
MEM_LIMIT="${TRUEREF_MEM_LIMIT:-0}"
CUDNN_LIB="${TRUEREF_CUDNN_LIB:-$ROOT/runtime/cudnn/nvidia/cudnn/lib}"
if [[ ! -f "$JAR" ]]; then
echo "trueref: jar not found at $JAR" >&2
echo "trueref: build it first with: mvn -DskipTests -pl trueref-bootstrap -am package" >&2
exit 1
fi
# Resolve java
if [[ -n "${JAVA:-}" ]]; then
:
elif [[ -n "${JAVA_HOME:-}" && -x "${JAVA_HOME}/bin/java" ]]; then
JAVA="${JAVA_HOME}/bin/java"
else
JAVA="$(command -v java || true)"
fi
if [[ -z "${JAVA:-}" || ! -x "${JAVA}" ]]; then
echo "trueref: java not found; install JDK 21+ or set JAVA_HOME" >&2
exit 1
fi
mkdir -p "$HOME_DIR"
SPRING_ARGS=(
"--server.port=$PORT"
"--trueref.home=$HOME_DIR"
)
# CUDA setup. "cpu" disables CUDA entirely; otherwise pass the physical GPU index
# directly to ORT. ORT's CUDA EP uses the physical device index regardless of
# CUDA_VISIBLE_DEVICES remapping — so we pass the physical index and explicitly
# unset CUDA_VISIBLE_DEVICES to avoid the two-layer renumbering problem where
# CUDA runtime remaps N→0 but ORT still expects the physical N.
if [[ "$GPU" == "cpu" || "$GPU" == "CPU" ]]; then
echo "trueref: GPU disabled (TRUEREF_GPU=cpu) — embedder/reranker will run on CPU"
SPRING_ARGS+=(
"--trueref.embedding.onnx-providers=cpu"
)
else
if [[ -d "$CUDNN_LIB" ]]; then
export LD_LIBRARY_PATH="${CUDNN_LIB}${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
else
echo "trueref: TRUEREF_CUDNN_LIB=$CUDNN_LIB not found — CUDA EP will fall back to CPU" >&2
echo "trueref: download cu12 cuDNN with:" >&2
echo " mkdir -p runtime/cudnn && cd runtime/cudnn && \\" >&2
echo " pip download --no-deps --only-binary=:all: --python-version 3.12 \\" >&2
echo " --platform manylinux2014_x86_64 'nvidia-cudnn-cu12<10' -d . && \\" >&2
echo " unzip -q -o nvidia_cudnn_cu12-*.whl 'nvidia/cudnn/lib/*' && rm *.whl" >&2
fi
# CUDA runtime respects CUDA_VISIBLE_DEVICES for all allocations (cudaMalloc, BFC arena,
# etc.). By restricting CUDA's view to exactly the target GPU, we prevent the runtime from
# creating a default context on device 0 before ORT's cudaSetDevice() takes effect.
# We always pass gpu-device-id=0 to ORT because CUDA_VISIBLE_DEVICES makes the target
# card the ONLY visible device (index 0 in the runtime's view).
#
# CUDA_DEVICE_ORDER=PCI_BUS_ID ensures CUDA runtime numbering matches nvidia-smi numbering.
# Without it, the default FASTEST_FIRST ordering can rank GPUs differently from nvidia-smi,
# so CUDA_VISIBLE_DEVICES=N would expose a different physical card than nvidia-smi GPU N.
export CUDA_DEVICE_ORDER="PCI_BUS_ID"
export CUDA_VISIBLE_DEVICES="$GPU"
SPRING_ARGS+=(
"--trueref.embedding.gpu-device-id=0"
"--trueref.embedding.gpu-mem-limit-bytes=$MEM_LIMIT"
)
fi
exec "$JAVA" \
--enable-native-access=ALL-UNNAMED \
--add-modules=jdk.incubator.vector \
${JAVA_OPTS:-} \
-jar "$JAR" \
"${SPRING_ARGS[@]}" \
"$@"

106
trueref-adapters/pom.xml Normal file
View File

@@ -0,0 +1,106 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.trueref</groupId>
<artifactId>trueref-parent</artifactId>
<version>0.1.0-SNAPSHOT</version>
</parent>
<artifactId>trueref-adapters</artifactId>
<name>trueref-adapters</name>
<description>All driving (REST, MCP) and driven (H2, Lucene, ONNX, JGit, tree-sitter, disk cache) adapters.</description>
<dependencies>
<dependency>
<groupId>com.trueref</groupId>
<artifactId>trueref-domain</artifactId>
</dependency>
<dependency>
<groupId>com.trueref</groupId>
<artifactId>trueref-application</artifactId>
</dependency>
<!-- Spring Web + JDBC + validation (no auto-config; bootstrap controls @ComponentScan) -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
</dependency>
<!-- OpenAPI / Swagger UI -->
<dependency>
<groupId>org.springdoc</groupId>
<artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
</dependency>
<!-- Spring AI MCP server (Streamable HTTP) -->
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-mcp-server-webmvc</artifactId>
</dependency>
<!-- H2 + Flyway -->
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>${h2.version}</version>
</dependency>
<dependency>
<groupId>org.flywaydb</groupId>
<artifactId>flyway-core</artifactId>
<version>${flyway.version}</version>
</dependency>
<!-- Lucene -->
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-core</artifactId>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-analysis-common</artifactId>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-queryparser</artifactId>
</dependency>
<!-- ONNX Runtime: GPU jar contains both CUDA and CPU providers; DirectML jar is added at runtime via classifier on Windows. -->
<dependency>
<groupId>com.microsoft.onnxruntime</groupId>
<artifactId>onnxruntime_gpu</artifactId>
</dependency>
<dependency>
<groupId>ai.djl.huggingface</groupId>
<artifactId>tokenizers</artifactId>
</dependency>
<!-- JGit -->
<dependency>
<groupId>org.eclipse.jgit</groupId>
<artifactId>org.eclipse.jgit</artifactId>
</dependency>
<dependency>
<groupId>org.eclipse.jgit</groupId>
<artifactId>org.eclipse.jgit.ssh.apache</artifactId>
</dependency>
<!-- Test -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>

24
trueref-adapters/src/main/java/com/trueref/adapter/in/mcp/McpConfig.java Normal file
View File

@@ -0,0 +1,24 @@
package com.trueref.adapter.in.mcp;
import org.springframework.ai.tool.ToolCallbackProvider;
import org.springframework.ai.tool.method.MethodToolCallbackProvider;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
/**
* Registers the trueref MCP tool callbacks with Spring AI's MCP WebMVC auto-configuration. The
* {@link MethodToolCallbackProvider} scans {@link TrueRefMcpTools} for methods annotated with
* {@link org.springframework.ai.tool.annotation.Tool} and publishes them on the MCP endpoint
* configured in {@code application.yml} (POST {@code /mcp} via
* {@code spring.ai.mcp.server.sse-message-endpoint}).
*/
@Configuration
@EnableConfigurationProperties(McpProperties.class)
public class McpConfig {
@Bean
public ToolCallbackProvider trueRefMcpToolCallbacks(TrueRefMcpTools tools) {
return MethodToolCallbackProvider.builder().toolObjects(tools).build();
}
}

22
trueref-adapters/src/main/java/com/trueref/adapter/in/mcp/McpProperties.java Normal file
View File

@@ -0,0 +1,22 @@
package com.trueref.adapter.in.mcp;
import org.springframework.boot.context.properties.ConfigurationProperties;
/**
* Token-budget defaults for {@code get-library-docs}. Matches Context7 semantics: clients may
* request an explicit token budget per call; unspecified calls use {@link #tokensDefault}. All
* requests are clamped to {@code [tokensMin, tokensMax]}.
*/
@ConfigurationProperties(prefix = "trueref.mcp")
public record McpProperties(int tokensDefault, int tokensMin, int tokensMax) {
public McpProperties {
if (tokensDefault <= 0) tokensDefault = 5000;
if (tokensMin <= 0) tokensMin = 500;
if (tokensMax <= 0) tokensMax = 50_000;
}
public int clamp(int requested) {
return Math.max(tokensMin, Math.min(tokensMax, requested));
}
}

297
trueref-adapters/src/main/java/com/trueref/adapter/in/mcp/TrueRefMcpTools.java Normal file
View File

@@ -0,0 +1,297 @@
package com.trueref.adapter.in.mcp;
import com.trueref.application.resolve.LibraryResolver;
import com.trueref.domain.model.Repository;
import com.trueref.domain.model.SearchHit;
import com.trueref.domain.model.SearchScope;
import com.trueref.domain.model.Version;
import com.trueref.domain.model.VersionStatus;
import com.trueref.domain.port.in.IndexVersion;
import com.trueref.domain.port.in.QueryCatalog;
import com.trueref.domain.port.in.ResolveLibraryId;
import com.trueref.domain.port.in.SearchLibraryDocs;
import java.util.Comparator;
import java.util.List;
import java.util.Optional;
import org.jspecify.annotations.Nullable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.tool.annotation.ToolParam;
import org.springframework.stereotype.Service;
/**
* Context7-compatible MCP tool handlers. The two tool names, parameter names, and response
* shapes are intentionally 1:1 with upstream Context7 so that any MCP client written against
* Context7 works against trueref unchanged.
*
* <p>Ranking and version→tag mapping are delegated to the application layer
* ({@link ResolveLibraryId}, {@link LibraryResolver}); hybrid search goes through
* {@link SearchLibraryDocs}; on-demand indexing is enqueued via {@link IndexVersion}.
*/
@Service
public class TrueRefMcpTools {
private static final Logger log = LoggerFactory.getLogger(TrueRefMcpTools.class);
private static final int DEFAULT_MAX_HITS = 50;
/** Matches Context7's banner retry hint (see ARCHITECTURE §7 on-demand indexing flow). */
private static final int INDEXING_RETRY_AFTER_SEC = 30;
private final ResolveLibraryId resolver;
private final LibraryResolver libraryResolver;
private final QueryCatalog catalog;
private final SearchLibraryDocs search;
private final IndexVersion indexer;
private final McpProperties props;
public TrueRefMcpTools(
ResolveLibraryId resolver,
LibraryResolver libraryResolver,
QueryCatalog catalog,
SearchLibraryDocs search,
IndexVersion indexer,
McpProperties props) {
this.resolver = resolver;
this.libraryResolver = libraryResolver;
this.catalog = catalog;
this.search = search;
this.indexer = indexer;
this.props = props;
}
@Tool(
name = "resolve-library-id",
description =
"Resolves a package/product name to a trueref-compatible library ID and "
+ "returns matching libraries. Context7-compatible. Each result "
+ "includes: Title, Context7-compatible library ID (format "
+ "/owner/repo[/version]), Description, Code Snippets, Versions, "
+ "and a relevance Score.")
public String resolveLibraryId(
@ToolParam(
description =
"Library name to search for and retrieve a Context7-"
+ "compatible library ID.")
String libraryName,
@ToolParam(
required = false,
description =
"Optional natural-language query used to rank matching "
+ "libraries by relevance.")
@Nullable String query) {
ResolveLibraryId.Result result =
resolver.resolve(new ResolveLibraryId.Query(libraryName, query, null));
if (result.matches().isEmpty()) {
return "No matching libraries found for: " + libraryName;
}
StringBuilder sb = new StringBuilder();
for (ResolveLibraryId.Match m : result.matches()) {
appendMatchBlock(sb, m);
}
return sb.toString();
}
@Tool(
name = "get-library-docs",
description =
"Fetches up-to-date documentation for a library. You MUST call "
+ "'resolve-library-id' first to obtain the exact library ID "
+ "required to use this tool, UNLESS the user explicitly provides "
+ "a library ID in the format /org/project or /org/project/version.")
public String getLibraryDocs(
@ToolParam(
description =
"Exact trueref-compatible library ID (format: "
+ "/org/project or /org/project/version) "
+ "retrieved from 'resolve-library-id'.")
String libraryId,
@ToolParam(
required = false,
description =
"Topic to focus the documentation on (e.g. "
+ "'routing', 'hooks', 'authentication').")
@Nullable String topic,
@ToolParam(
required = false,
description =
"Max number of tokens to return. Clamped to "
+ "[500, 50000]; defaults to 5000.")
@Nullable Integer tokens) {
ParsedId parsed = parseLibraryId(libraryId);
if (parsed == null) {
return "Invalid libraryId: " + libraryId
+ ". Expected format: /org/project or /org/project/version.";
}
Optional<Repository> repoOpt = catalog.listRepositories().stream()
.filter(r -> r.name().equalsIgnoreCase(parsed.repoName()))
.findFirst();
if (repoOpt.isEmpty()) {
return "No matching library found for ID: " + libraryId;
}
Repository repo = repoOpt.get();
List<Version> versions = catalog.listVersions(repo.id());
SelectedVersion selected = selectVersion(repo, versions, parsed.version());
if (selected.searchTarget() == null) {
return "No indexed version available for /" + repo.name()
+ ". Indexing has been enqueued; retry in ~"
+ INDEXING_RETRY_AFTER_SEC + " seconds.";
}
int budget = props.clamp(tokens == null ? props.tokensDefault() : tokens);
String text = (topic == null || topic.isBlank()) ? repo.name() : topic;
SearchLibraryDocs.Query q = new SearchLibraryDocs.Query(
text,
topic,
new SearchScope(List.of(new SearchScope.RepoVersionRef(repo.id(), selected.searchTarget().id()))),
budget,
DEFAULT_MAX_HITS);
SearchLibraryDocs.Result res;
try {
res = search.search(q);
} catch (Exception e) {
log.warn("MCP search failed for {}: {}", libraryId, e.toString());
return "Search failed for " + libraryId + ": " + e.getMessage();
}
return formatDocs(res.hits(), selected.banner());
}
// --- helpers -----------------------------------------------------------
private void appendMatchBlock(StringBuilder sb, ResolveLibraryId.Match m) {
sb.append("----------\n");
sb.append("- Title: ").append(m.name()).append('\n');
sb.append("- Context7-compatible library ID: ").append(m.libraryId()).append('\n');
if (m.description() != null && !m.description().isBlank()) {
sb.append("- Description: ").append(m.description()).append('\n');
}
sb.append("- Code Snippets: ").append(m.snippetCount()).append('\n');
if (!m.availableVersions().isEmpty()) {
sb.append("- Versions: ");
for (int i = 0; i < m.availableVersions().size(); i++) {
if (i > 0) sb.append(", ");
ResolveLibraryId.VersionRef v = m.availableVersions().get(i);
sb.append(v.tag()).append(" (").append(v.status()).append(')');
}
sb.append('\n');
}
sb.append("- Score: ").append(String.format("%.2f", m.score())).append('\n');
}
private SelectedVersion selectVersion(
Repository repo, List<Version> versions, @Nullable String requestedVersion) {
if (requestedVersion == null || requestedVersion.isBlank()) {
// No version in libraryId: prefer most-recent INDEXED; else nearest DISCOVERED +
// enqueue indexing + banner.
Optional<Version> latestIndexed = versions.stream()
.filter(v -> v.status() == VersionStatus.INDEXED)
.max(Comparator.comparing(Version::tag));
if (latestIndexed.isPresent()) {
return new SelectedVersion(latestIndexed.get(), null);
}
Optional<Version> latestDiscovered = versions.stream()
.filter(v -> v.status() == VersionStatus.DISCOVERED)
.max(Comparator.comparing(Version::tag));
if (latestDiscovered.isPresent()) {
Version v = latestDiscovered.get();
enqueueSafely(repo, v);
return new SelectedVersion(null, indexingBanner(v.tag(), v.tag()));
}
return new SelectedVersion(null, null);
}
// Explicit version requested: use application-layer mapper.
Optional<Version> mapped = libraryResolver.mapVersion(repo, versions, requestedVersion);
if (mapped.isEmpty()) {
// Fall back to nearest INDEXED; if any, show banner for the requested version.
Optional<Version> nearest = versions.stream()
.filter(v -> v.status() == VersionStatus.INDEXED)
.max(Comparator.comparing(Version::tag));
return new SelectedVersion(
nearest.orElse(null), nearest.map(v -> indexingBanner(requestedVersion, v.tag())).orElse(null));
}
Version target = mapped.get();
if (target.status() == VersionStatus.INDEXED) {
return new SelectedVersion(target, null);
}
enqueueSafely(repo, target);
Optional<Version> nearestIndexed = versions.stream()
.filter(v -> v.status() == VersionStatus.INDEXED)
.max(Comparator.comparing(Version::tag));
return new SelectedVersion(
nearestIndexed.orElse(null),
indexingBanner(target.tag(), nearestIndexed.map(Version::tag).orElse("none")));
}
private void enqueueSafely(Repository repo, Version v) {
if (v.status() == VersionStatus.INDEXING) return;
try {
indexer.enqueue(repo.id(), v.id(), false);
} catch (Exception e) {
log.warn("MCP on-demand indexing enqueue failed for {}@{}: {}", repo.name(), v.tag(), e.toString());
}
}
private String indexingBanner(String requestedTag, String fallbackTag) {
return "[indexing] version " + requestedTag
+ " is being indexed now; showing nearest indexed version " + fallbackTag
+ " (retryAfterSec=" + INDEXING_RETRY_AFTER_SEC + ")";
}
private String formatDocs(List<SearchHit> hits, @Nullable String banner) {
StringBuilder sb = new StringBuilder();
if (banner != null) {
sb.append(banner).append('\n').append('\n');
}
sb.append("================\n");
sb.append("CODE SNIPPETS\n");
sb.append("================\n");
if (hits.isEmpty()) {
sb.append("(no matching snippets)\n");
return sb.toString();
}
for (SearchHit h : hits) {
sb.append("TITLE: ")
.append(h.filePath())
.append(':')
.append(h.startLine())
.append('-')
.append(h.endLine())
.append(" (")
.append(h.language())
.append(")\n");
sb.append("SOURCE: ")
.append(h.repoName())
.append('@')
.append(h.tag())
.append(" — ")
.append(h.filePath())
.append("\n\n");
sb.append("```").append(h.language()).append('\n');
sb.append(h.content());
if (!h.content().endsWith("\n")) sb.append('\n');
sb.append("```\n");
sb.append("----------------------------------------\n");
}
return sb.toString();
}
static @Nullable ParsedId parseLibraryId(String raw) {
if (raw == null || raw.isBlank()) return null;
String s = raw.startsWith("/") ? raw.substring(1) : raw;
String[] parts = s.split("/");
if (parts.length == 2) {
return new ParsedId(parts[0] + "/" + parts[1], null);
}
if (parts.length == 3) {
return new ParsedId(parts[0] + "/" + parts[1], parts[2]);
}
return null;
}
record ParsedId(String repoName, @Nullable String version) {}
private record SelectedVersion(@Nullable Version searchTarget, @Nullable String banner) {}
}

16
trueref-adapters/src/main/java/com/trueref/adapter/in/mcp/package-info.java Normal file
View File

@@ -0,0 +1,16 @@
/**
* Driving adapter: Model Context Protocol (MCP) server exposing the two Context7-compatible
* tools ({@code resolve-library-id}, {@code get-library-docs}) over Spring AI's MCP WebMVC
* transport. The HTTP message endpoint is wired to {@code POST /mcp} via
* {@code spring.ai.mcp.server.sse-message-endpoint}.
*
* <p>Spring AI 1.0.0 ships the SSE-based WebMVC transport
* ({@code WebMvcSseServerTransportProvider}); the 2025-03-26 "Streamable HTTP" transport is
* not a separate selectable property in this version. Clients that POST JSON-RPC to the
* configured message endpoint receive JSON-RPC responses; the server additionally opens an
* SSE stream on the configured {@code sse-endpoint} for server-initiated notifications. This
* is the closest equivalent Spring AI 1.0.0 provides to Streamable HTTP and is the
* intended/only transport of this adapter.
*/
@org.jspecify.annotations.NullMarked
package com.trueref.adapter.in.mcp;

16
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/ErrorResponse.java Normal file
View File

@@ -0,0 +1,16 @@
package com.trueref.adapter.in.rest;
import io.swagger.v3.oas.annotations.media.Schema;
import java.util.List;
/** Uniform error envelope returned by {@link GlobalExceptionHandler}. */
@Schema(description = "Error response envelope.")
public record ErrorResponse(String code, String message, List<FieldError> fieldErrors) {
public static ErrorResponse of(String code, String message) {
return new ErrorResponse(code, message, List.of());
}
@Schema(description = "A single field-level validation error.")
public record FieldError(String field, String message) {}
}

120
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/GlobalExceptionHandler.java Normal file
View File

@@ -0,0 +1,120 @@
package com.trueref.adapter.in.rest;
import com.trueref.domain.error.IngestionFailed;
import com.trueref.domain.error.InvalidSearchRequest;
import com.trueref.domain.error.RepositoryAlreadyRegistered;
import com.trueref.domain.error.RepositoryNotFound;
import com.trueref.domain.error.TagNotFound;
import com.trueref.domain.error.TrueRefException;
import com.trueref.domain.error.VersionNotFound;
import com.trueref.domain.error.VersionNotIndexed;
import jakarta.validation.ConstraintViolationException;
import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.validation.FieldError;
import org.springframework.web.bind.MethodArgumentNotValidException;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.server.ResponseStatusException;
import org.springframework.web.servlet.resource.NoResourceFoundException;
/** Central translator from domain / validation exceptions to HTTP + {@link ErrorResponse} JSON. */
@RestControllerAdvice
public class GlobalExceptionHandler {
private static final Logger log = LoggerFactory.getLogger(GlobalExceptionHandler.class);
@ExceptionHandler({RepositoryNotFound.class, VersionNotFound.class, TagNotFound.class})
public ResponseEntity<ErrorResponse> handleNotFound(TrueRefException ex) {
return status(HttpStatus.NOT_FOUND, ex);
}
@ExceptionHandler(RepositoryAlreadyRegistered.class)
public ResponseEntity<ErrorResponse> handleConflict(RepositoryAlreadyRegistered ex) {
return status(HttpStatus.CONFLICT, ex);
}
@ExceptionHandler(VersionNotIndexed.class)
public ResponseEntity<ErrorResponse> handleNotIndexed(VersionNotIndexed ex) {
return status(HttpStatus.CONFLICT, ex);
}
@ExceptionHandler(InvalidSearchRequest.class)
public ResponseEntity<ErrorResponse> handleInvalidSearch(InvalidSearchRequest ex) {
return status(HttpStatus.BAD_REQUEST, ex);
}
@ExceptionHandler(MethodArgumentNotValidException.class)
public ResponseEntity<ErrorResponse> handleValidation(MethodArgumentNotValidException ex) {
List<ErrorResponse.FieldError> fieldErrors = ex.getBindingResult().getFieldErrors().stream()
.map(this::toFieldError)
.toList();
ErrorResponse body = new ErrorResponse("validation_failed", "Request validation failed", fieldErrors);
return ResponseEntity.status(HttpStatus.BAD_REQUEST).body(body);
}
@ExceptionHandler(ConstraintViolationException.class)
public ResponseEntity<ErrorResponse> handleConstraintViolation(ConstraintViolationException ex) {
List<ErrorResponse.FieldError> fieldErrors = ex.getConstraintViolations().stream()
.map(v -> new ErrorResponse.FieldError(v.getPropertyPath().toString(), v.getMessage()))
.toList();
ErrorResponse body = new ErrorResponse("validation_failed", "Request validation failed", fieldErrors);
return ResponseEntity.status(HttpStatus.BAD_REQUEST).body(body);
}
@ExceptionHandler(IllegalArgumentException.class)
public ResponseEntity<ErrorResponse> handleIllegalArgument(IllegalArgumentException ex) {
ErrorResponse body = new ErrorResponse("invalid_request", safeMessage(ex), List.of());
return ResponseEntity.status(HttpStatus.BAD_REQUEST).body(body);
}
@ExceptionHandler(ResponseStatusException.class)
public ResponseEntity<ErrorResponse> handleResponseStatus(ResponseStatusException ex) {
HttpStatus resolved = HttpStatus.resolve(ex.getStatusCode().value());
HttpStatus status = resolved == null ? HttpStatus.INTERNAL_SERVER_ERROR : resolved;
String code = ex.getReason() == null ? status.name().toLowerCase() : ex.getReason();
return ResponseEntity.status(status).body(ErrorResponse.of(code, code));
}
@ExceptionHandler(NoResourceFoundException.class)
public ResponseEntity<Void> handleNoResource(NoResourceFoundException ex) {
// Static resource not found (e.g. browser/.well-known probes). Return 404 without
// logging — these are not application errors and would flood the log at ERROR level.
return ResponseEntity.notFound().build();
}
@ExceptionHandler(IngestionFailed.class)
public ResponseEntity<ErrorResponse> handleIngestionFailed(IngestionFailed ex) {
log.error("ingestion failed", ex);
return status(HttpStatus.INTERNAL_SERVER_ERROR, ex);
}
@ExceptionHandler(TrueRefException.class)
public ResponseEntity<ErrorResponse> handleDomain(TrueRefException ex) {
log.error("unhandled domain error code={}", ex.code(), ex);
return status(HttpStatus.INTERNAL_SERVER_ERROR, ex);
}
@ExceptionHandler(Exception.class)
public ResponseEntity<ErrorResponse> handleUnexpected(Exception ex) {
log.error("unexpected error", ex);
ErrorResponse body = new ErrorResponse("internal_error", "An unexpected error occurred", List.of());
return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(body);
}
private ResponseEntity<ErrorResponse> status(HttpStatus status, TrueRefException ex) {
return ResponseEntity.status(status).body(ErrorResponse.of(ex.code(), safeMessage(ex)));
}
private ErrorResponse.FieldError toFieldError(FieldError fe) {
String msg = fe.getDefaultMessage() == null ? "invalid" : fe.getDefaultMessage();
return new ErrorResponse.FieldError(fe.getField(), msg);
}
private static String safeMessage(Throwable t) {
return t.getMessage() == null ? t.getClass().getSimpleName() : t.getMessage();
}
}

145
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/JobController.java Normal file
View File

@@ -0,0 +1,145 @@
package com.trueref.adapter.in.rest;
import com.trueref.adapter.in.rest.dto.JobDto;
import com.trueref.adapter.in.rest.dto.JobLogEventDto;
import com.trueref.domain.model.JobId;
import com.trueref.domain.model.JobStatus;
import com.trueref.domain.model.RepositoryId;
import com.trueref.domain.model.VersionId;
import com.trueref.domain.port.in.ObserveJobs;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import java.io.IOException;
import java.util.List;
import org.jspecify.annotations.Nullable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.server.ResponseStatusException;
import org.springframework.web.servlet.mvc.method.annotation.SseEmitter;
/** REST + SSE resource: {@code /api/jobs}. */
@RestController
@RequestMapping("/api/jobs")
@Tag(name = "jobs", description = "Inspect ingestion jobs and stream live progress via SSE.")
public class JobController {
private static final Logger log = LoggerFactory.getLogger(JobController.class);
private final ObserveJobs observeJobs;
public JobController(ObserveJobs observeJobs) {
this.observeJobs = observeJobs;
}
@Operation(summary = "List jobs, optionally filtered by repo / version / status.")
@GetMapping
public List<JobDto> list(
@RequestParam(value = "repoId", required = false) @Nullable String repoId,
@RequestParam(value = "versionId", required = false) @Nullable String versionId,
@RequestParam(value = "status", required = false) @Nullable JobStatus status,
@RequestParam(value = "limit", defaultValue = "100") int limit) {
RepositoryId repo = repoId == null ? null : RepositoryId.of(repoId);
VersionId ver = versionId == null ? null : VersionId.of(versionId);
return observeJobs.listJobs(repo, ver, status, limit).stream()
.map(JobDto::of)
.toList();
}
@Operation(summary = "Get a single job by id.")
@GetMapping("/{id}")
public JobDto detail(@PathVariable("id") String id) {
JobId jobId = JobId.of(id);
return observeJobs
.findJob(jobId)
.map(JobDto::of)
.orElseThrow(() ->
new ResponseStatusException(org.springframework.http.HttpStatus.NOT_FOUND, "job_not_found"));
}
@Operation(summary = "Server-Sent Events stream of log events for a single job.")
@GetMapping(value = "/{id}/log", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public SseEmitter logStream(@PathVariable("id") String id) {
JobId jobId = JobId.of(id);
SseEmitter emitter = new SseEmitter(0L);
AutoCloseable subscription = observeJobs.subscribeLogs(jobId, event -> {
try {
emitter.send(SseEmitter.event().name("log").data(JobLogEventDto.of(event)));
} catch (IOException ex) {
emitter.completeWithError(ex);
}
});
attachCleanup(emitter, subscription, "job-log " + id);
return emitter;
}
@Operation(summary = "Server-Sent Events stream of status updates for all jobs.")
@GetMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public SseEmitter stream() {
SseEmitter emitter = new SseEmitter(0L);
// Send an immediate ping so Tomcat flushes the response headers to the client.
// Without this, the response buffer may not be flushed until the first job event
// arrives, keeping the EventSource in CONNECTING state and never firing 'open'.
try {
emitter.send(SseEmitter.event().name("ping").data(""));
} catch (IOException e) {
emitter.completeWithError(e);
return emitter;
}
AutoCloseable subscription = observeJobs.subscribeJobs(job -> {
try {
emitter.send(SseEmitter.event().name("job").data(JobDto.of(job)));
} catch (IOException ex) {
emitter.completeWithError(ex);
}
});
// Keepalive: send a ping every 20 s to keep the connection alive through idle
// periods and detect disconnected clients promptly.
Thread keepalive = Thread.startVirtualThread(() -> {
try {
while (!Thread.currentThread().isInterrupted()) {
Thread.sleep(20_000);
emitter.send(SseEmitter.event().name("ping").data(""));
}
} catch (InterruptedException ignored) {
// normal shutdown
} catch (Exception ignored) {
// emitter already completed or errored; exit
}
});
Runnable cleanup = () -> {
keepalive.interrupt();
try {
subscription.close();
} catch (Exception ex) {
log.debug("failed to close SSE subscription job-stream: {}", ex.toString());
}
};
emitter.onCompletion(cleanup);
emitter.onTimeout(cleanup);
emitter.onError(e -> cleanup.run());
return emitter;
}
private static void attachCleanup(SseEmitter emitter, AutoCloseable subscription, String label) {
Runnable cleanup = () -> {
try {
subscription.close();
} catch (Exception ex) {
log.debug("failed to close SSE subscription {}: {}", label, ex.toString());
}
};
emitter.onCompletion(cleanup);
emitter.onTimeout(cleanup);
emitter.onError(e -> cleanup.run());
}
}

200
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/ObservabilityController.java Normal file
View File

@@ -0,0 +1,200 @@
package com.trueref.adapter.in.rest;
import com.trueref.adapter.in.rest.dto.JobDto;
import com.trueref.adapter.out.embedding.onnx.OnnxProperties;
import com.trueref.domain.model.IngestionJob;
import com.trueref.domain.model.JobStatus;
import com.trueref.domain.model.Repository;
import com.trueref.domain.model.Version;
import com.trueref.domain.model.VersionStatus;
import com.trueref.domain.port.in.ObserveJobs;
import com.trueref.domain.port.in.QueryCatalog;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.EnumMap;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.jspecify.annotations.Nullable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
/** REST resource: {@code /api/observability}. */
@RestController
@RequestMapping("/api/observability")
@Tag(name = "observability", description = "UI-friendly aggregates: metrics + resource usage.")
public class ObservabilityController {
private static final Logger log = LoggerFactory.getLogger(ObservabilityController.class);
private static final int JOB_SAMPLE_LIMIT = 10_000;
private final ObserveJobs observeJobs;
private final QueryCatalog queryCatalog;
private final Path trueRefHome;
private final boolean embedderAvailable;
private final boolean rerankerAvailable;
private final int gpuDeviceId;
public ObservabilityController(
ObserveJobs observeJobs,
QueryCatalog queryCatalog,
OnnxProperties onnxProperties,
@Value("${trueref.home:./data}") String trueRefHome,
@Value("${trueref.embedder.available:true}") boolean embedderAvailable,
@Value("${trueref.reranker.available:true}") boolean rerankerAvailable) {
this.observeJobs = observeJobs;
this.queryCatalog = queryCatalog;
this.trueRefHome = Path.of(trueRefHome);
this.embedderAvailable = embedderAvailable;
this.rerankerAvailable = rerankerAvailable;
this.gpuDeviceId = onnxProperties.gpuDeviceId();
}
@Operation(summary = "Aggregated metrics for the dashboard (job counts, totals, availability).")
@GetMapping("/metrics")
public Map<String, Object> metrics() {
Map<JobStatus, Long> jobsByStatus = new EnumMap<>(JobStatus.class);
for (JobStatus status : JobStatus.values()) {
jobsByStatus.put(status, 0L);
}
List<IngestionJob> jobs = observeJobs.listJobs(null, null, null, JOB_SAMPLE_LIMIT);
for (IngestionJob job : jobs) {
jobsByStatus.merge(job.status(), 1L, Long::sum);
}
long totalChunks = 0L;
long totalVersionsIndexed = 0L;
long totalRepos = 0L;
for (Repository repo : queryCatalog.listRepositories()) {
totalRepos++;
for (Version v : queryCatalog.listVersions(repo.id())) {
totalChunks += v.chunkCount();
if (v.status() == VersionStatus.INDEXED) {
totalVersionsIndexed++;
}
}
}
Map<String, Object> result = new HashMap<>();
result.put("jobsByStatus", toStringKeys(jobsByStatus));
result.put("jobsSampled", jobs.size());
result.put("jobsSampleLimit", JOB_SAMPLE_LIMIT);
result.put("totalRepositories", totalRepos);
result.put("totalChunks", totalChunks);
result.put("totalVersionsIndexed", totalVersionsIndexed);
result.put("embedderAvailable", embedderAvailable);
result.put("rerankerAvailable", rerankerAvailable);
return result;
}
@Operation(summary = "Heap / index-size / cache-size snapshot.")
@GetMapping("/resources")
public Map<String, Object> resources() {
Runtime runtime = Runtime.getRuntime();
long heapMax = runtime.maxMemory();
long heapTotal = runtime.totalMemory();
long heapFree = runtime.freeMemory();
long heapUsed = heapTotal - heapFree;
long luceneBytes = directorySizeBytes(trueRefHome.resolve("lucene"));
long cacheBytes = directorySizeBytes(trueRefHome.resolve("embedding-cache"));
Map<String, Object> heap = new HashMap<>();
heap.put("usedBytes", heapUsed);
heap.put("totalBytes", heapTotal);
heap.put("maxBytes", heapMax);
Map<String, Object> result = new HashMap<>();
result.put("heap", heap);
result.put("luceneIndexBytes", luceneBytes);
result.put("embeddingCacheBytes", cacheBytes);
result.put("trueRefHome", trueRefHome.toAbsolutePath().toString());
result.put("gpu", queryGpuMemory(gpuDeviceId));
return result;
}
/**
* Queries {@code nvidia-smi} for memory stats of the given GPU device index.
* Returns {@code null} when nvidia-smi is absent or the device index is out of range.
*/
private @Nullable Map<String, Object> queryGpuMemory(int deviceId) {
try {
Process proc = new ProcessBuilder(
"nvidia-smi",
"--query-gpu=memory.used,memory.free,memory.total",
"--format=csv,noheader,nounits",
"-i",
String.valueOf(deviceId))
.redirectErrorStream(true)
.start();
String line;
try (BufferedReader br =
new BufferedReader(new InputStreamReader(proc.getInputStream(), StandardCharsets.UTF_8))) {
line = br.readLine();
}
int exit = proc.waitFor();
if (exit != 0 || line == null || line.isBlank()) {
return null;
}
// Output: "usedMiB, freeMiB, totalMiB"
String[] parts = line.split(",");
if (parts.length < 3) return null;
long usedMiB = Long.parseLong(parts[0].trim());
long freeMiB = Long.parseLong(parts[1].trim());
long totalMiB = Long.parseLong(parts[2].trim());
long mibToBytes = 1024L * 1024L;
Map<String, Object> gpu = new HashMap<>();
gpu.put("deviceId", deviceId);
gpu.put("usedBytes", usedMiB * mibToBytes);
gpu.put("freeBytes", freeMiB * mibToBytes);
gpu.put("totalBytes", totalMiB * mibToBytes);
return gpu;
} catch (IOException | InterruptedException | NumberFormatException e) {
log.debug("nvidia-smi query failed: {}", e.toString());
return null;
}
}
private static Map<String, Long> toStringKeys(Map<JobStatus, Long> in) {
Map<String, Long> out = new HashMap<>();
in.forEach((k, v) -> out.put(k.name(), v));
return out;
}
private static long directorySizeBytes(Path dir) {
if (!Files.isDirectory(dir)) {
return 0L;
}
long[] total = {0L};
try {
Files.walkFileTree(dir, new SimpleFileVisitor<>() {
@Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) {
total[0] += attrs.size();
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult visitFileFailed(Path file, IOException exc) {
return FileVisitResult.CONTINUE;
}
});
} catch (IOException e) {
log.warn("failed to walk {}: {}", dir, e.toString());
}
return total[0];
}
}

25
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/OpenApiConfig.java Normal file
View File

@@ -0,0 +1,25 @@
package com.trueref.adapter.in.rest;
import io.swagger.v3.oas.models.OpenAPI;
import io.swagger.v3.oas.models.info.Info;
import io.swagger.v3.oas.models.servers.Server;
import java.util.List;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
/** Springdoc OpenAPI customization — title, version, summary and default local server. */
@Configuration
public class OpenApiConfig {
@Bean
public OpenAPI trueRefOpenApi() {
Info info = new Info()
.title("trueref API")
.version("0.1.0")
.summary("Self-hosted Context7-compatible ingestion + retrieval HTTP API.")
.description(
"REST endpoints for repository registration, ingestion orchestration, hybrid search and library resolution.");
Server local = new Server().url("http://localhost:8080").description("Local development server");
return new OpenAPI().info(info).servers(List.of(local));
}
}

156
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/RepositoryController.java Normal file
View File

@@ -0,0 +1,156 @@
package com.trueref.adapter.in.rest;
import com.trueref.adapter.in.rest.dto.RegisterRepositoryRequest;
import com.trueref.adapter.in.rest.dto.RepositoryDto;
import com.trueref.adapter.in.rest.dto.VersionDto;
import com.trueref.domain.error.RepositoryNotFound;
import com.trueref.domain.error.TagNotFound;
import com.trueref.domain.model.Repository;
import com.trueref.domain.model.RepositoryId;
import com.trueref.domain.model.Version;
import com.trueref.domain.port.in.DiscoverVersions;
import com.trueref.domain.port.in.IndexVersion;
import com.trueref.domain.port.in.QueryCatalog;
import com.trueref.domain.port.in.RegisterRepository;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.validation.Valid;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseStatus;
import org.springframework.web.bind.annotation.RestController;
/** REST resource: {@code /api/repos}. */
@RestController
@RequestMapping("/api/repos")
@Tag(name = "repositories", description = "Register, list, and index git repositories.")
public class RepositoryController {
private static final Logger log = LoggerFactory.getLogger(RepositoryController.class);
private final RegisterRepository registerRepository;
private final QueryCatalog queryCatalog;
private final DiscoverVersions discoverVersions;
private final IndexVersion indexVersion;
public RepositoryController(
RegisterRepository registerRepository,
QueryCatalog queryCatalog,
DiscoverVersions discoverVersions,
IndexVersion indexVersion) {
this.registerRepository = registerRepository;
this.queryCatalog = queryCatalog;
this.discoverVersions = discoverVersions;
this.indexVersion = indexVersion;
}
@Operation(summary = "List all registered repositories.")
@GetMapping
public List<RepositoryDto> list() {
return queryCatalog.listRepositories().stream().map(RepositoryDto::of).toList();
}
@Operation(summary = "Register a new repository (remote URL or local path).")
@PostMapping
@ResponseStatus(HttpStatus.CREATED)
public RepositoryDto register(@Valid @RequestBody RegisterRepositoryRequest request) {
log.info("registering repository name={}", request.name());
Repository repo = registerRepository.register(request.toCommand());
return RepositoryDto.of(repo);
}
@Operation(summary = "Get details of a single repository.")
@GetMapping("/{id}")
public RepositoryDto detail(@PathVariable("id") String id) {
RepositoryId repoId = parseRepoId(id);
return queryCatalog.findRepository(repoId).map(RepositoryDto::of).orElseThrow(() -> new RepositoryNotFound(id));
}
@Operation(summary = "Unregister a repository and soft-delete its versions.")
@DeleteMapping("/{id}")
@ResponseStatus(HttpStatus.NO_CONTENT)
public void unregister(@PathVariable("id") String id) {
RepositoryId repoId = parseRepoId(id);
log.info("unregistering repository id={}", id);
registerRepository.unregister(repoId);
}
@Operation(summary = "Force tag discovery (git fetch + enumerate tags). Returns the resulting versions.")
@PostMapping("/{id}/discover")
public List<VersionDto> discover(@PathVariable("id") String id) {
RepositoryId repoId = parseRepoId(id);
log.info("discovering versions for repository id={}", id);
return discoverVersions.discover(repoId).stream().map(VersionDto::of).toList();
}
@Operation(summary = "List all versions (git tags) known for this repository.")
@GetMapping("/{id}/versions")
public List<VersionDto> versions(@PathVariable("id") String id) {
RepositoryId repoId = parseRepoId(id);
// Ensure 404 if the repo does not exist.
queryCatalog.findRepository(repoId).orElseThrow(() -> new RepositoryNotFound(id));
return queryCatalog.listVersions(repoId).stream().map(VersionDto::of).toList();
}
@Operation(summary = "Enqueue indexing of a specific tag. If the tag is unknown, discovery runs first.")
@PostMapping("/{id}/versions/{tag}/index")
public ResponseEntity<Map<String, String>> index(
@PathVariable("id") String id,
@PathVariable("tag") String tag,
@RequestBody(required = false) IndexBody body) {
boolean force = body != null && Boolean.TRUE.equals(body.force());
return enqueueIndex(id, tag, force);
}
@Operation(summary = "Force re-indexing of a specific tag (equivalent to index with force=true).")
@PostMapping("/{id}/versions/{tag}/reindex")
public ResponseEntity<Map<String, String>> reindex(
@PathVariable("id") String id, @PathVariable("tag") String tag) {
return enqueueIndex(id, tag, true);
}
private ResponseEntity<Map<String, String>> enqueueIndex(String id, String tag, boolean force) {
RepositoryId repoId = parseRepoId(id);
Repository repo = queryCatalog.findRepository(repoId).orElseThrow(() -> new RepositoryNotFound(id));
Optional<Version> existing = findByTag(repoId, tag);
if (existing.isEmpty()) {
log.info("tag {} unknown for repo {}, triggering discovery", tag, repo.name());
discoverVersions.discover(repoId);
existing = findByTag(repoId, tag);
}
Version version = existing.orElseThrow(() -> new TagNotFound(repo.name(), tag));
log.info("enqueueing index job repo={} tag={} force={}", repo.name(), tag, force);
var jobId = indexVersion.enqueue(repoId, version.id(), force);
return ResponseEntity.accepted().body(Map.of("jobId", jobId.toString()));
}
private Optional<Version> findByTag(RepositoryId repoId, String tag) {
return queryCatalog.listVersions(repoId).stream()
.filter(v -> v.tag().equals(tag))
.findFirst();
}
private static RepositoryId parseRepoId(String id) {
try {
return RepositoryId.of(id);
} catch (IllegalArgumentException e) {
throw new RepositoryNotFound(id);
}
}
/** Body of {@code POST /api/repos/{id}/versions/{tag}/index}. */
public record IndexBody(Boolean force) {}
}

41
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/ResolveController.java Normal file
View File

@@ -0,0 +1,41 @@
package com.trueref.adapter.in.rest;
import com.trueref.adapter.in.rest.dto.ResolveMatchDto;
import com.trueref.adapter.in.rest.dto.ResolveRequest;
import com.trueref.adapter.in.rest.dto.ResolveResponse;
import com.trueref.domain.port.in.ResolveLibraryId;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import org.jspecify.annotations.Nullable;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
/** REST resource: {@code /api/resolve}. */
@RestController
@RequestMapping("/api/resolve")
@Tag(name = "resolve", description = "Turn a fuzzy library name (and optional version) into concrete (repo, version) handles.")
public class ResolveController {
private final ResolveLibraryId resolveLibraryId;
public ResolveController(ResolveLibraryId resolveLibraryId) {
this.resolveLibraryId = resolveLibraryId;
}
@Operation(summary = "Preview library ID resolution for the given name / version.")
@GetMapping
public ResolveResponse resolve(
@RequestParam("libraryName") String libraryName,
@RequestParam(value = "version", required = false) @Nullable String version,
@RequestParam(value = "query", required = false) @Nullable String query) {
if (libraryName == null || libraryName.isBlank()) {
throw new IllegalArgumentException("libraryName must not be blank");
}
ResolveRequest req = new ResolveRequest(libraryName, query, version);
ResolveLibraryId.Result result = resolveLibraryId.resolve(req.toQuery());
return new ResolveResponse(
result.matches().stream().map(ResolveMatchDto::of).toList());
}
}

47
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/SearchController.java Normal file
View File

@@ -0,0 +1,47 @@
package com.trueref.adapter.in.rest;
import com.trueref.adapter.in.rest.dto.SearchHitDto;
import com.trueref.adapter.in.rest.dto.SearchRequest;
import com.trueref.adapter.in.rest.dto.SearchResponse;
import com.trueref.domain.port.in.SearchLibraryDocs;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.validation.Valid;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
/** REST resource: {@code /api/search}. */
@RestController
@RequestMapping("/api/search")
@Tag(name = "search", description = "Hybrid BM25 + dense search with cross-encoder rerank.")
public class SearchController {
private static final Logger log = LoggerFactory.getLogger(SearchController.class);
private final SearchLibraryDocs searchLibraryDocs;
public SearchController(SearchLibraryDocs searchLibraryDocs) {
this.searchLibraryDocs = searchLibraryDocs;
}
@Operation(summary = "Hybrid search scoped to one or more (repo, version) pairs.")
@PostMapping
public SearchResponse search(@Valid @RequestBody SearchRequest request) {
log.debug(
"search text='{}' topic={} scopes={} tokensBudget={} maxHits={}",
request.text(),
request.topic(),
request.scope().size(),
request.tokensBudget(),
request.maxHits());
SearchLibraryDocs.Result result = searchLibraryDocs.search(request.toQuery());
return new SearchResponse(
result.hits().stream().map(SearchHitDto::of).toList(),
result.totalTokensReturned(),
request.topic());
}
}

54
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/WebConfig.java Normal file
View File

@@ -0,0 +1,54 @@
package com.trueref.adapter.in.rest;
import java.io.IOException;
import java.util.Set;
import org.jspecify.annotations.Nullable;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.Resource;
import org.springframework.web.servlet.config.annotation.ResourceHandlerRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
import org.springframework.web.servlet.resource.PathResourceResolver;
/**
* SPA fallback: any unmatched request that looks like a client-side route (no file extension in
* the final path segment) is served {@code index.html}. API, MCP, springdoc and actuator paths are
* explicitly excluded — Spring routes those first anyway, but we exclude them defensively so the
* resource resolver does not attempt a fallback for them.
*/
@Configuration
public class WebConfig implements WebMvcConfigurer {
private static final Set<String> EXCLUDED_PREFIXES =
Set.of("api/", "mcp", "swagger-ui/", "v3/api-docs", "actuator/");
@Override
public void addResourceHandlers(ResourceHandlerRegistry registry) {
registry.addResourceHandler("/**")
.addResourceLocations("classpath:/static/")
.resourceChain(true)
.addResolver(new SpaFallbackResolver());
}
static final class SpaFallbackResolver extends PathResourceResolver {
@Override
protected @Nullable Resource getResource(String resourcePath, Resource location) throws IOException {
for (String prefix : EXCLUDED_PREFIXES) {
if (resourcePath.equals(prefix) || resourcePath.startsWith(prefix)) {
return null;
}
}
Resource requested = location.createRelative(resourcePath);
if (requested.exists() && requested.isReadable()) {
return requested;
}
// Fallback to index.html only for client-side route-like paths (no extension on last segment).
int lastSlash = resourcePath.lastIndexOf('/');
String lastSegment = lastSlash < 0 ? resourcePath : resourcePath.substring(lastSlash + 1);
if (!lastSegment.isEmpty() && lastSegment.contains(".")) {
return null;
}
Resource indexHtml = location.createRelative("index.html");
return indexHtml.exists() && indexHtml.isReadable() ? indexHtml : null;
}
}
}

33
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/dto/JobDto.java Normal file
View File

@@ -0,0 +1,33 @@
package com.trueref.adapter.in.rest.dto;
import com.trueref.domain.model.IngestionJob;
import com.trueref.domain.model.JobStatus;
import com.trueref.domain.model.JobType;
import io.swagger.v3.oas.annotations.media.Schema;
import java.time.Instant;
import java.util.List;
import org.jspecify.annotations.Nullable;
@Schema(description = "An orchestrated ingestion job with its stages.")
public record JobDto(
String id,
String repoId,
@Nullable String versionId,
JobType type,
JobStatus status,
@Nullable Instant startedAt,
@Nullable Instant finishedAt,
List<JobStageDto> stages) {
public static JobDto of(IngestionJob j) {
return new JobDto(
j.id().toString(),
j.repoId().toString(),
j.versionId() == null ? null : j.versionId().toString(),
j.type(),
j.status(),
j.startedAt(),
j.finishedAt(),
j.stages().stream().map(JobStageDto::of).toList());
}
}

16
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/dto/JobLogEventDto.java Normal file
View File

@@ -0,0 +1,16 @@
package com.trueref.adapter.in.rest.dto;
import com.trueref.domain.model.JobLogEvent;
import com.trueref.domain.model.JobStage;
import io.swagger.v3.oas.annotations.media.Schema;
import java.time.Instant;
import org.jspecify.annotations.Nullable;
@Schema(description = "A single log event emitted by an ingestion job.")
public record JobLogEventDto(
String jobId, Instant ts, JobLogEvent.Level level, JobStage.@Nullable StageName stage, String message) {
public static JobLogEventDto of(JobLogEvent e) {
return new JobLogEventDto(e.jobId().toString(), e.ts(), e.level(), e.stage(), e.message());
}
}

32
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/dto/JobStageDto.java Normal file
View File

@@ -0,0 +1,32 @@
package com.trueref.adapter.in.rest.dto;
import com.trueref.domain.model.JobStage;
import io.swagger.v3.oas.annotations.media.Schema;
import java.time.Instant;
import org.jspecify.annotations.Nullable;
@Schema(description = "A single stage within an ingestion job.")
public record JobStageDto(
String jobId,
JobStage.StageName name,
JobStage.StageStatus status,
@Nullable Instant startedAt,
@Nullable Instant finishedAt,
long itemsProcessed,
long itemsTotal,
long bytesProcessed,
@Nullable String errorMessage) {
public static JobStageDto of(JobStage s) {
return new JobStageDto(
s.jobId().toString(),
s.name(),
s.status(),
s.startedAt(),
s.finishedAt(),
s.itemsProcessed(),
s.itemsTotal(),
s.bytesProcessed(),
s.errorMessage());
}
}

45
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/dto/RegisterRepositoryRequest.java Normal file
View File

@@ -0,0 +1,45 @@
package com.trueref.adapter.in.rest.dto;
import com.trueref.domain.model.TagPattern;
import com.trueref.domain.port.in.RegisterRepository;
import io.swagger.v3.oas.annotations.media.Schema;
import jakarta.validation.Valid;
import jakarta.validation.constraints.NotBlank;
import java.time.Duration;
import java.time.format.DateTimeParseException;
import java.util.List;
import org.jspecify.annotations.Nullable;
@Schema(description = "Request payload to register a new repository.")
public record RegisterRepositoryRequest(
@NotBlank @Schema(description = "Human-readable display name, e.g. spring-projects/spring-boot") String name,
@Schema(description = "Remote git URL (mutually exclusive with localPath)") @Nullable String remoteUrl,
@Schema(description = "Absolute local path to an already-cloned repo") @Nullable String localPath,
@Schema(description = "Per-repo ignore globs, ANDed with .gitignore") @Nullable List<String> ignoreGlobs,
@Schema(description = "Max file size in bytes; default 1MiB") @Nullable Long maxFileSizeBytes,
@Schema(description = "ISO-8601 duration (e.g. PT1H); 0 disables polling") @Nullable String pollInterval,
@Schema(description = "Max most-recent tags auto-indexed") @Nullable Integer tagCap,
@Schema(description = "Ordered tag-pattern rules for client version → tag mapping") @Valid @Nullable
List<TagPatternDto> versionMappingRules) {
public RegisterRepository.Command toCommand() {
Duration poll = parseDuration(pollInterval);
List<String> globs = ignoreGlobs == null ? List.of() : List.copyOf(ignoreGlobs);
List<TagPattern> rules = versionMappingRules == null
? List.of()
: versionMappingRules.stream().map(TagPatternDto::toModel).toList();
return new RegisterRepository.Command(
name, remoteUrl, localPath, globs, maxFileSizeBytes, poll, tagCap, rules);
}
private static @Nullable Duration parseDuration(@Nullable String iso) {
if (iso == null || iso.isBlank()) {
return null;
}
try {
return Duration.parse(iso);
} catch (DateTimeParseException e) {
throw new IllegalArgumentException("Invalid ISO-8601 duration: " + iso, e);
}
}
}

39
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/dto/RepositoryDto.java Normal file
View File

@@ -0,0 +1,39 @@
package com.trueref.adapter.in.rest.dto;
import com.trueref.domain.model.Repository;
import io.swagger.v3.oas.annotations.media.Schema;
import java.time.Instant;
import java.util.List;
import org.jspecify.annotations.Nullable;
@Schema(description = "A registered repository.")
public record RepositoryDto(
String id,
String name,
@Nullable String remoteUrl,
String localPath,
boolean managedClone,
List<String> ignoreGlobs,
long maxFileSizeBytes,
@Schema(description = "ISO-8601 duration, e.g. PT1H") String pollInterval,
int tagCap,
List<TagPatternDto> versionMappingRules,
Instant createdAt,
Instant updatedAt) {
public static RepositoryDto of(Repository repo) {
return new RepositoryDto(
repo.id().toString(),
repo.name(),
repo.remoteUrl(),
repo.localPath(),
repo.managedClone(),
List.copyOf(repo.ignoreGlobs()),
repo.maxFileSizeBytes(),
repo.pollInterval().toString(),
repo.tagCap(),
repo.versionMappingRules().stream().map(TagPatternDto::of).toList(),
repo.createdAt(),
repo.updatedAt());
}
}

28
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/dto/ResolveMatchDto.java Normal file
View File

@@ -0,0 +1,28 @@
package com.trueref.adapter.in.rest.dto;
import com.trueref.domain.port.in.ResolveLibraryId;
import io.swagger.v3.oas.annotations.media.Schema;
import java.util.List;
import org.jspecify.annotations.Nullable;
@Schema(description = "A single candidate library matching a resolve request.")
public record ResolveMatchDto(
String repoId,
String libraryId,
String name,
@Nullable String description,
int snippetCount,
List<ResolveVersionRefDto> availableVersions,
double score) {
public static ResolveMatchDto of(ResolveLibraryId.Match m) {
return new ResolveMatchDto(
m.repoId().toString(),
m.libraryId(),
m.name(),
m.description(),
m.snippetCount(),
m.availableVersions().stream().map(ResolveVersionRefDto::of).toList(),
m.score());
}
}

17
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/dto/ResolveRequest.java Normal file
View File

@@ -0,0 +1,17 @@
package com.trueref.adapter.in.rest.dto;
import com.trueref.domain.port.in.ResolveLibraryId;
import io.swagger.v3.oas.annotations.media.Schema;
import jakarta.validation.constraints.NotBlank;
import org.jspecify.annotations.Nullable;
@Schema(description = "Fuzzy library resolution request.")
public record ResolveRequest(
@NotBlank String libraryName,
@Schema(description = "Optional hint to rerank candidates by relevance") @Nullable String query,
@Nullable String version) {
public ResolveLibraryId.Query toQuery() {
return new ResolveLibraryId.Query(libraryName, query, version);
}
}

7
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/dto/ResolveResponse.java Normal file
View File

@@ -0,0 +1,7 @@
package com.trueref.adapter.in.rest.dto;
import io.swagger.v3.oas.annotations.media.Schema;
import java.util.List;
@Schema(description = "Ranked library matches for a resolve request.")
public record ResolveResponse(List<ResolveMatchDto> matches) {}

13
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/dto/ResolveVersionRefDto.java Normal file
View File

@@ -0,0 +1,13 @@
package com.trueref.adapter.in.rest.dto;
import com.trueref.domain.model.VersionStatus;
import com.trueref.domain.port.in.ResolveLibraryId;
import io.swagger.v3.oas.annotations.media.Schema;
@Schema(description = "One available version of a resolved library.")
public record ResolveVersionRefDto(String versionId, String tag, VersionStatus status) {
public static ResolveVersionRefDto of(ResolveLibraryId.VersionRef v) {
return new ResolveVersionRefDto(v.versionId().toString(), v.tag(), v.status());
}
}

37
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/dto/SearchHitDto.java Normal file
View File

@@ -0,0 +1,37 @@
package com.trueref.adapter.in.rest.dto;
import com.trueref.domain.model.SearchHit;
import io.swagger.v3.oas.annotations.media.Schema;
import org.jspecify.annotations.Nullable;
@Schema(description = "A single ranked snippet returned by search.")
public record SearchHitDto(
String chunkId,
String repoId,
String versionId,
String repoName,
String tag,
String filePath,
int startLine,
int endLine,
String language,
@Nullable String symbol,
String content,
double score) {
public static SearchHitDto of(SearchHit h) {
return new SearchHitDto(
h.chunkId().toString(),
h.repoId().toString(),
h.versionId().toString(),
h.repoName(),
h.tag(),
h.filePath(),
h.startLine(),
h.endLine(),
h.language(),
h.symbol(),
h.content(),
h.score());
}
}

41
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/dto/SearchRequest.java Normal file
View File

@@ -0,0 +1,41 @@
package com.trueref.adapter.in.rest.dto;
import com.trueref.domain.model.RepositoryId;
import com.trueref.domain.model.SearchScope;
import com.trueref.domain.model.VersionId;
import com.trueref.domain.port.in.SearchLibraryDocs;
import io.swagger.v3.oas.annotations.media.Schema;
import jakarta.validation.Valid;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotEmpty;
import jakarta.validation.constraints.Positive;
import java.util.List;
import org.jspecify.annotations.Nullable;
@Schema(description = "Hybrid search request scoped to one or more (repo, version) pairs.")
public record SearchRequest(
@NotBlank String text,
@Nullable String topic,
@NotEmpty @Valid List<ScopeRef> scope,
@Schema(description = "Token budget; clamped by the service to [500, 50000]") @Positive @Nullable
Integer tokensBudget,
@Positive @Nullable Integer maxHits) {
public static final int DEFAULT_TOKENS_BUDGET = 5000;
public static final int DEFAULT_MAX_HITS = 20;
public SearchLibraryDocs.Query toQuery() {
List<SearchScope.RepoVersionRef> refs = scope.stream()
.map(r -> new SearchScope.RepoVersionRef(RepositoryId.of(r.repoId()), VersionId.of(r.versionId())))
.toList();
return new SearchLibraryDocs.Query(
text,
topic,
new SearchScope(refs),
tokensBudget == null ? DEFAULT_TOKENS_BUDGET : tokensBudget,
maxHits == null ? DEFAULT_MAX_HITS : maxHits);
}
@Schema(description = "A (repo, version) pair to scope the search on.")
public record ScopeRef(@NotBlank String repoId, @NotBlank String versionId) {}
}

8
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/dto/SearchResponse.java Normal file
View File

@@ -0,0 +1,8 @@
package com.trueref.adapter.in.rest.dto;
import io.swagger.v3.oas.annotations.media.Schema;
import java.util.List;
import org.jspecify.annotations.Nullable;
@Schema(description = "Response body for search.")
public record SearchResponse(List<SearchHitDto> hits, int totalTokensReturned, @Nullable String topic) {}

41
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/dto/TagPatternDto.java Normal file
View File

@@ -0,0 +1,41 @@
package com.trueref.adapter.in.rest.dto;
import com.trueref.domain.model.TagPattern;
import io.swagger.v3.oas.annotations.media.Schema;
import org.jspecify.annotations.Nullable;
/** DTO for {@link TagPattern}. The sealed hierarchy is flattened into a {@code type} + optional {@code template}. */
@Schema(description = "Rule mapping a client-supplied version string to a git tag.")
public record TagPatternDto(
@Schema(
description = "Pattern kind",
allowableValues = {"EXACT", "V_PREFIX", "RELEASE_PREFIX", "SEMVER_FUZZY", "CUSTOM"})
String type,
@Schema(description = "Required when type=CUSTOM, e.g. release-{semver}") @Nullable String template) {
public static TagPatternDto of(TagPattern pattern) {
return switch (pattern) {
case TagPattern.Exact e -> new TagPatternDto("EXACT", null);
case TagPattern.VPrefix v -> new TagPatternDto("V_PREFIX", null);
case TagPattern.ReleasePrefix r -> new TagPatternDto("RELEASE_PREFIX", null);
case TagPattern.SemverFuzzy s -> new TagPatternDto("SEMVER_FUZZY", null);
case TagPattern.Custom c -> new TagPatternDto("CUSTOM", c.template());
};
}
public TagPattern toModel() {
return switch (type) {
case "EXACT" -> new TagPattern.Exact();
case "V_PREFIX" -> new TagPattern.VPrefix();
case "RELEASE_PREFIX" -> new TagPattern.ReleasePrefix();
case "SEMVER_FUZZY" -> new TagPattern.SemverFuzzy();
case "CUSTOM" -> {
if (template == null || template.isBlank()) {
throw new IllegalArgumentException("CUSTOM tag pattern requires a template");
}
yield new TagPattern.Custom(template);
}
default -> throw new IllegalArgumentException("Unknown tag pattern type: " + type);
};
}
}

31
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/dto/VersionDto.java Normal file
View File

@@ -0,0 +1,31 @@
package com.trueref.adapter.in.rest.dto;
import com.trueref.domain.model.Version;
import com.trueref.domain.model.VersionStatus;
import io.swagger.v3.oas.annotations.media.Schema;
import java.time.Instant;
import org.jspecify.annotations.Nullable;
@Schema(description = "A specific git tag (or branch) of a repository.")
public record VersionDto(
String id,
String repoId,
String tag,
String commitSha,
VersionStatus status,
@Nullable Instant indexedAt,
int chunkCount,
@Nullable String errorMessage) {
public static VersionDto of(Version v) {
return new VersionDto(
v.id().toString(),
v.repoId().toString(),
v.tag(),
v.commitSha(),
v.status(),
v.indexedAt(),
v.chunkCount(),
v.errorMessage());
}
}

3
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/dto/package-info.java Normal file
View File

@@ -0,0 +1,3 @@
/** Transport-layer DTO records used by {@code com.trueref.adapter.in.rest} controllers. */
@org.jspecify.annotations.NullMarked
package com.trueref.adapter.in.rest.dto;

6
trueref-adapters/src/main/java/com/trueref/adapter/in/rest/package-info.java Normal file
View File

@@ -0,0 +1,6 @@
/**
* REST driving adapter: controllers, DTOs, OpenAPI configuration, exception handling and SSE
* streaming for the trueref HTTP API.
*/
@org.jspecify.annotations.NullMarked
package com.trueref.adapter.in.rest;

67
trueref-adapters/src/main/resources/db/migration/V1__init_schema.sql Normal file
View File

@@ -0,0 +1,67 @@
-- trueref schema V1
-- All UUIDs stored as CHAR(36) for H2 portability.
CREATE TABLE repositories (
id CHAR(36) PRIMARY KEY,
name VARCHAR(512) NOT NULL UNIQUE,
remote_url VARCHAR(2048) NULL,
local_path VARCHAR(2048) NOT NULL,
managed_clone BOOLEAN NOT NULL,
ignore_globs CLOB NOT NULL, -- JSON array of strings
max_file_size_bytes BIGINT NOT NULL,
poll_interval_seconds BIGINT NOT NULL,
tag_cap INT NOT NULL,
version_mapping_rules CLOB NOT NULL, -- JSON array of TagPattern
created_at TIMESTAMP(9) WITH TIME ZONE NOT NULL,
updated_at TIMESTAMP(9) WITH TIME ZONE NOT NULL
);
CREATE TABLE versions (
id CHAR(36) PRIMARY KEY,
repo_id CHAR(36) NOT NULL REFERENCES repositories(id) ON DELETE CASCADE,
tag VARCHAR(512) NOT NULL,
commit_sha CHAR(40) NOT NULL,
status VARCHAR(32) NOT NULL,
indexed_at TIMESTAMP(9) WITH TIME ZONE NULL,
chunk_count INT NOT NULL DEFAULT 0,
error_message CLOB NULL,
UNIQUE (repo_id, tag)
);
CREATE INDEX idx_versions_repo_status ON versions(repo_id, status);
CREATE TABLE ingestion_jobs (
id CHAR(36) PRIMARY KEY,
repo_id CHAR(36) NOT NULL REFERENCES repositories(id) ON DELETE CASCADE,
version_id CHAR(36) NULL REFERENCES versions(id) ON DELETE CASCADE,
type VARCHAR(32) NOT NULL,
status VARCHAR(32) NOT NULL,
started_at TIMESTAMP(9) WITH TIME ZONE NULL,
finished_at TIMESTAMP(9) WITH TIME ZONE NULL,
created_at TIMESTAMP(9) WITH TIME ZONE NOT NULL
);
CREATE INDEX idx_jobs_repo_status ON ingestion_jobs(repo_id, status);
CREATE INDEX idx_jobs_status_created ON ingestion_jobs(status, created_at);
CREATE TABLE job_stages (
job_id CHAR(36) NOT NULL REFERENCES ingestion_jobs(id) ON DELETE CASCADE,
name VARCHAR(32) NOT NULL,
status VARCHAR(32) NOT NULL,
started_at TIMESTAMP(9) WITH TIME ZONE NULL,
finished_at TIMESTAMP(9) WITH TIME ZONE NULL,
items_processed BIGINT NOT NULL DEFAULT 0,
items_total BIGINT NOT NULL DEFAULT 0,
bytes_processed BIGINT NOT NULL DEFAULT 0,
error_message CLOB NULL,
PRIMARY KEY (job_id, name)
);
-- Persisted log buffer (last N per job kept by application logic; SSE streams from in-memory bus).
CREATE TABLE job_log_events (
id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
job_id CHAR(36) NOT NULL REFERENCES ingestion_jobs(id) ON DELETE CASCADE,
ts TIMESTAMP(9) WITH TIME ZONE NOT NULL,
level VARCHAR(8) NOT NULL,
stage VARCHAR(32),
message CLOB NOT NULL
);
CREATE INDEX idx_job_log_job_ts ON job_log_events(job_id, ts);

28
trueref-application/pom.xml Normal file
View File

@@ -0,0 +1,28 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.trueref</groupId>
<artifactId>trueref-parent</artifactId>
<version>0.1.0-SNAPSHOT</version>
</parent>
<artifactId>trueref-application</artifactId>
<name>trueref-application</name>
<description>Use-case implementations. Depends only on the domain.</description>
<dependencies>
<dependency>
<groupId>com.trueref</groupId>
<artifactId>trueref-domain</artifactId>
</dependency>
<!-- SLF4J only; orchestration may use virtual threads via JDK -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
</dependencies>
</project>

89
trueref-application/src/main/java/com/trueref/application/catalog/CatalogService.java Normal file
View File

@@ -0,0 +1,89 @@
package com.trueref.application.catalog;
import com.trueref.domain.error.RepositoryAlreadyRegistered;
import com.trueref.domain.error.RepositoryNotFound;
import com.trueref.domain.model.Repository;
import com.trueref.domain.model.RepositoryId;
import com.trueref.domain.model.TagPattern;
import com.trueref.domain.model.Version;
import com.trueref.domain.port.in.QueryCatalog;
import com.trueref.domain.port.in.RegisterRepository;
import com.trueref.domain.port.out.RepositoryStore;
import java.nio.file.Path;
import java.time.Duration;
import java.time.Instant;
import java.util.List;
import java.util.Optional;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/** Implements {@link RegisterRepository} and {@link QueryCatalog}. */
public final class CatalogService implements RegisterRepository, QueryCatalog {
private static final Logger log = LoggerFactory.getLogger(CatalogService.class);
private static final List<TagPattern> DEFAULT_RULES = List.of(
new TagPattern.Exact(),
new TagPattern.VPrefix(),
new TagPattern.ReleasePrefix(),
new TagPattern.SemverFuzzy());
private final RepositoryStore store;
private final Path trueRefHome;
public CatalogService(RepositoryStore store, Path trueRefHome) {
this.store = store;
this.trueRefHome = trueRefHome;
}
@Override
public Repository register(Command cmd) {
store.findByName(cmd.name()).ifPresent(r -> {
throw new RepositoryAlreadyRegistered(cmd.name());
});
boolean managed = cmd.remoteUrl() != null && cmd.localPath() == null;
String localPath = cmd.localPath() != null
? cmd.localPath()
: trueRefHome.resolve("repos").resolve(cmd.name().replace('/', '_')).toString();
Instant now = Instant.now();
Repository repo = new Repository(
RepositoryId.random(),
cmd.name(),
cmd.remoteUrl(),
localPath,
managed,
cmd.ignoreGlobs(),
cmd.maxFileSizeBytes() == null ? 1_048_576L : cmd.maxFileSizeBytes(),
cmd.pollInterval() == null ? Duration.ofHours(1) : cmd.pollInterval(),
cmd.tagCap() == null ? 100 : cmd.tagCap(),
cmd.versionMappingRules().isEmpty() ? DEFAULT_RULES : cmd.versionMappingRules(),
now,
now);
Repository saved = store.save(repo);
log.info("registered repository name={} id={} managed={} localPath={}",
saved.name(), saved.id(), managed, localPath);
return saved;
}
@Override
public void unregister(RepositoryId id) {
Repository existing = store.findById(id).orElseThrow(() -> new RepositoryNotFound(id.toString()));
store.delete(id);
log.info("unregistered repository name={} id={}", existing.name(), id);
}
@Override
public List<Repository> listRepositories() {
return store.findAll();
}
@Override
public Optional<Repository> findRepository(RepositoryId id) {
return store.findById(id);
}
@Override
public List<Version> listVersions(RepositoryId repoId) {
return store.findVersionsByRepo(repoId);
}
}

87
trueref-application/src/main/java/com/trueref/application/ingest/DiscoveryService.java Normal file
View File

@@ -0,0 +1,87 @@
package com.trueref.application.ingest;
import com.trueref.domain.error.RepositoryNotFound;
import com.trueref.domain.model.Repository;
import com.trueref.domain.model.Version;
import com.trueref.domain.model.VersionId;
import com.trueref.domain.model.VersionStatus;
import com.trueref.domain.port.in.DiscoverVersions;
import com.trueref.domain.port.out.GitClient;
import com.trueref.domain.port.out.RepositoryStore;
import com.trueref.domain.model.RepositoryId;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Comparator;
import java.util.List;
import java.util.Optional;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/** Fetches tags (git fetch + tag list) and persists new/updated {@link Version}s. */
public final class DiscoveryService implements DiscoverVersions {
private static final Logger log = LoggerFactory.getLogger(DiscoveryService.class);
private final RepositoryStore store;
private final GitClient git;
public DiscoveryService(RepositoryStore store, GitClient git) {
this.store = store;
this.git = git;
}
@Override
public List<Version> discover(RepositoryId repoId) {
Repository repo = store.findById(repoId).orElseThrow(() -> new RepositoryNotFound(repoId.toString()));
Path path = Path.of(repo.localPath());
// clone if managed and not present
if (repo.managedClone() && !Files.exists(path.resolve(".git"))) {
log.info("cloning for discovery: {}", repo.name());
git.cloneRepo(repo.remoteUrl(), path);
} else {
try { git.fetch(path); } catch (Exception e) {
log.warn("fetch failed for {}: {}", repo.name(), e.toString());
}
}
List<GitClient.TagInfo> tags = git.listTags(path);
// apply tag cap: keep top-N by epoch DESC (already sorted)
List<GitClient.TagInfo> capped = tags.stream().limit(Math.max(1, repo.tagCap())).toList();
for (GitClient.TagInfo t : capped) {
Optional<Version> existing = store.findVersionByTag(repoId, t.name());
if (existing.isPresent()) {
// refresh commit sha only if changed
if (!existing.get().commitSha().equalsIgnoreCase(t.commitSha())) {
Version updated = new Version(
existing.get().id(),
existing.get().repoId(),
t.name(),
t.commitSha(),
VersionStatus.DISCOVERED, // needs re-index
existing.get().indexedAt(),
existing.get().chunkCount(),
null);
store.saveVersion(updated);
log.info("tag {} changed commit; marked DISCOVERED", t.name());
}
} else {
Version v = new Version(
VersionId.random(),
repoId,
t.name(),
t.commitSha(),
VersionStatus.DISCOVERED,
null,
0,
null);
store.saveVersion(v);
log.info("discovered new tag {}", t.name());
}
}
return store.findVersionsByRepo(repoId).stream()
.sorted(Comparator.comparing(Version::tag))
.toList();
}
}

604
trueref-application/src/main/java/com/trueref/application/ingest/IngestionOrchestrator.java Normal file
View File

@@ -0,0 +1,604 @@
package com.trueref.application.ingest;
import com.trueref.domain.model.Chunk;
import com.trueref.domain.model.ChunkId;
import com.trueref.domain.model.ChunkVersion;
import com.trueref.domain.model.Embedding;
import com.trueref.domain.model.IngestionJob;
import com.trueref.domain.model.JobId;
import com.trueref.domain.model.JobLogEvent;
import com.trueref.domain.model.JobStage;
import com.trueref.domain.model.JobStatus;
import com.trueref.domain.model.JobType;
import com.trueref.domain.model.Repository;
import com.trueref.domain.model.RepositoryId;
import com.trueref.domain.model.Version;
import com.trueref.domain.model.VersionId;
import com.trueref.domain.model.VersionStatus;
import com.trueref.domain.port.in.IndexVersion;
import com.trueref.domain.port.out.ChunkStore;
import com.trueref.domain.port.out.CodeParser;
import com.trueref.domain.port.out.EmbeddingCache;
import com.trueref.domain.port.out.EmbeddingService;
import com.trueref.domain.port.out.GitClient;
import com.trueref.domain.port.out.JobEventBus;
import com.trueref.domain.port.out.JobStore;
import com.trueref.domain.port.out.RepositoryStore;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.FileSystems;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.PathMatcher;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
import java.security.MessageDigest;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;
/**
* Orchestrates the full ingestion pipeline for one (repo, version): clone/fetch → checkout →
* discover files → diff-vs-parent → parse → chunk → dedupe by hash → embed (with cache) → index
* into Lucene → commit.
*
* <p>The pipeline is split into two concurrent stages:
* <ol>
* <li><b>Parse phase</b> (virtual threads, up to {@code maxParseJobs} in parallel):
* FETCH/CLONE → CHECKOUT → DISCOVER_FILES → DIFF_FILES → PARSE.
* I/O-bound; no GPU use; worktree is removed immediately after parse.
* </li>
* <li><b>Embed phase</b> (single dedicated platform thread):
* EMBED → INDEX → COMMIT. GPU-bound; serialises ONNX inference to prevent CUDA
* context races. Runs on a platform thread for a stable OS thread identity.
* </li>
* </ol>
* Completed parse batches are handed off via a bounded {@link BlockingQueue}: if the embed
* worker is busy, parse workers block before queuing, naturally capping in-memory pressure.
* One orchestrator instance is shared across all jobs.
*/
public final class IngestionOrchestrator implements IndexVersion {
private static final Logger log = LoggerFactory.getLogger(IngestionOrchestrator.class);
// Built-in ignore globs (applied in addition to .gitignore + per-repo globs).
private static final List<String> BUILTIN_IGNORES = List.of(
"**/.git/**",
"**/node_modules/**",
"**/target/**",
"**/build/**",
"**/dist/**",
"**/out/**",
"**/.idea/**",
"**/.vscode/**",
"**/__pycache__/**",
"**/*.png", "**/*.jpg", "**/*.jpeg", "**/*.gif", "**/*.webp", "**/*.ico",
"**/*.pdf", "**/*.zip", "**/*.tar", "**/*.gz", "**/*.jar", "**/*.class",
"**/*.so", "**/*.dll", "**/*.dylib", "**/*.exe", "**/*.bin");
private final RepositoryStore repoStore;
private final JobStore jobStore;
private final ChunkStore chunkStore;
private final EmbeddingService embeddings;
private final EmbeddingCache embeddingCache;
private final GitClient git;
private final CodeParser parser;
private final JobEventBus bus;
private final ExecutorService parseExecutor;
private final Semaphore parseConcurrencyLimit;
private final BlockingQueue<ParsedBatch> embedQueue;
private final Thread embedWorker;
private volatile boolean shuttingDown = false;
private final Map<JobId, Boolean> running = new ConcurrentHashMap<>();
public IngestionOrchestrator(
RepositoryStore repoStore,
JobStore jobStore,
ChunkStore chunkStore,
EmbeddingService embeddings,
EmbeddingCache embeddingCache,
GitClient git,
CodeParser parser,
JobEventBus bus,
int maxParseJobs,
int embedQueueCapacity) {
this.repoStore = repoStore;
this.jobStore = jobStore;
this.chunkStore = chunkStore;
this.embeddings = embeddings;
this.embeddingCache = embeddingCache;
this.git = git;
this.parser = parser;
this.bus = bus;
this.parseExecutor = Executors.newVirtualThreadPerTaskExecutor();
// Fair semaphore caps parallel parse jobs (I/O + CPU heavy, no GPU).
this.parseConcurrencyLimit = new Semaphore(Math.max(1, maxParseJobs), true);
// Bounded queue between parse workers and the embed worker.
// Backpressure: parse workers block here when the embed worker is saturated,
// preventing unbounded in-memory accumulation of parsed chunks.
this.embedQueue = new LinkedBlockingQueue<>(Math.max(1, embedQueueCapacity));
// Single platform thread for GPU inference. Platform (not virtual) gives a
// stable OS thread identity for CUDA — the synchronized(session) in OnnxEmbeddingService
// already pins virtual threads, but a dedicated platform thread removes all doubt.
this.embedWorker = Thread.ofPlatform()
.name("embed-worker")
.daemon(false)
.start(this::drainEmbedQueue);
log.info("IngestionOrchestrator ready: maxParseJobs={} embedQueueCapacity={}",
Math.max(1, maxParseJobs), Math.max(1, embedQueueCapacity));
}
@Override
public JobId enqueue(RepositoryId repoId, VersionId versionId, boolean force) {
Repository repo = repoStore.findById(repoId).orElseThrow();
Version ver = repoStore.findVersion(versionId).orElseThrow();
if (!force && ver.status() == VersionStatus.INDEXED) {
log.info("version already indexed and not forcing; skipping repo={} tag={}", repo.name(), ver.tag());
JobId id = JobId.random();
IngestionJob skipped = new IngestionJob(
id,
repoId,
versionId,
JobType.INDEX_VERSION,
JobStatus.SUCCEEDED,
Instant.now(),
Instant.now(),
List.of());
jobStore.save(skipped);
bus.publishJob(skipped);
return id;
}
JobId jobId = JobId.random();
IngestionJob job = new IngestionJob(
jobId,
repoId,
versionId,
JobType.INDEX_VERSION,
JobStatus.QUEUED,
null,
null,
List.of());
jobStore.save(job);
bus.publishJob(job);
running.put(jobId, Boolean.TRUE);
parseExecutor.submit(() -> runParsePhase(jobId, repo, ver));
return jobId;
}
/**
* Carry struct that transfers parse-phase output to the embed worker.
* The git worktree has already been removed before this batch enters the queue;
* only in-memory chunk data travels across the thread boundary.
*/
private record ParsedBatch(
JobId jobId, Repository repo, Version ver, List<ParsedPiece> pieces) {}
/**
* Parse phase — runs on a virtual thread, up to {@code maxParseJobs} in parallel.
* Stages: FETCH/CLONE → CHECKOUT → DISCOVER_FILES → DIFF_FILES → PARSE.
* On completion, removes the worktree, releases the parse slot so the next job can
* start immediately, then blocks on {@link #embedQueue} until the embed worker has
* room (natural backpressure).
*/
private void runParsePhase(JobId jobId, Repository repo, Version ver) {
MDC.put("jobId", jobId.toString());
MDC.put("repo", repo.name());
MDC.put("tag", ver.tag());
try {
parseConcurrencyLimit.acquire();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
log.warn("job {} interrupted while waiting for parse slot — failing", jobId);
repoStore.updateVersionStatus(ver.id(), VersionStatus.FAILED, "interrupted");
transitionJob(jobId, JobStatus.FAILED, null, Instant.now());
running.remove(jobId);
MDC.clear();
return;
}
boolean slotReleased = false;
Path worktree = null;
Path repoPath = Path.of(repo.localPath());
try {
transitionJob(jobId, JobStatus.RUNNING, Instant.now(), null);
repoStore.updateVersionStatus(ver.id(), VersionStatus.INDEXING, null);
// STAGE: FETCH (or CLONE if managed and absent)
if (repo.managedClone() && !Files.exists(repoPath.resolve(".git"))) {
stage(jobId, JobStage.StageName.CLONE, () -> {
logEvent(jobId, JobLogEvent.Level.INFO, JobStage.StageName.CLONE,
"cloning " + repo.remoteUrl() + " → " + repoPath);
git.cloneRepo(repo.remoteUrl(), repoPath);
return 1L;
});
} else {
stage(jobId, JobStage.StageName.FETCH, () -> {
git.fetch(repoPath);
return 1L;
});
}
// STAGE: CHECKOUT
final Path wt = stageReturning(jobId, JobStage.StageName.CHECKOUT, () -> {
Path w = git.checkoutWorktree(repoPath, ver.tag());
logEvent(jobId, JobLogEvent.Level.INFO, JobStage.StageName.CHECKOUT,
"checked out at " + w);
return w;
});
worktree = wt;
// STAGE: DISCOVER_FILES
List<Path> files = stageReturning(jobId, JobStage.StageName.DISCOVER_FILES, () ->
discoverFiles(wt, repo));
logEvent(jobId, JobLogEvent.Level.INFO, JobStage.StageName.DISCOVER_FILES,
"found " + files.size() + " indexable files");
// STAGE: DIFF_FILES (select subset)
String baseRef = pickParentIndexedTag(repo, ver);
final List<Path> selectedFiles;
if (baseRef != null) {
Set<String> changedRel = stageReturning(jobId, JobStage.StageName.DIFF_FILES, () -> {
List<GitClient.DiffEntry> diff = git.diff(repoPath, baseRef, ver.tag());
Set<String> s = new HashSet<>();
for (GitClient.DiffEntry e : diff) {
if (e.change() != GitClient.DiffEntry.ChangeType.DELETED) s.add(e.path());
}
return s;
});
selectedFiles = files.stream()
.filter(f -> changedRel.contains(wt.relativize(f).toString().replace('\\', '/')))
.toList();
logEvent(jobId, JobLogEvent.Level.INFO, JobStage.StageName.DIFF_FILES,
"diff vs " + baseRef + " selects " + selectedFiles.size() + "/" + files.size());
} else {
selectedFiles = files;
}
// STAGE: PARSE + CHUNK + HASH (combined)
List<ParsedPiece> pieces = stageReturning(jobId, JobStage.StageName.PARSE, () ->
parseAll(selectedFiles, wt));
// Worktree no longer needed — free disk space before blocking on embed queue.
removeWorktreeQuietly(jobId, repoPath, wt);
worktree = null;
// Release parse slot before blocking so the next job can start parsing
// while this batch waits for the embed worker (maximises CPU/GPU overlap).
parseConcurrencyLimit.release();
slotReleased = true;
// Hand off to embed worker — blocks if the queue is at capacity.
embedQueue.put(new ParsedBatch(jobId, repo, ver, pieces));
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
log.warn("job {} interrupted during parse — failing", jobId);
repoStore.updateVersionStatus(ver.id(), VersionStatus.FAILED, "interrupted");
transitionJob(jobId, JobStatus.FAILED, null, Instant.now());
running.remove(jobId);
} catch (Exception e) {
log.error("parse phase failed for job {}", jobId, e);
logEvent(jobId, JobLogEvent.Level.ERROR, null, "parse phase failed: " + e.getMessage());
repoStore.updateVersionStatus(ver.id(), VersionStatus.FAILED, e.getMessage());
transitionJob(jobId, JobStatus.FAILED, null, Instant.now());
running.remove(jobId);
} finally {
if (worktree != null) removeWorktreeQuietly(jobId, repoPath, worktree);
if (!slotReleased) parseConcurrencyLimit.release();
MDC.clear();
}
}
/**
* Embed worker — runs on a single dedicated platform thread.
* Drains {@link #embedQueue} until {@link #shutdown()} signals stop.
* Stages per batch: EMBED → INDEX → COMMIT → mark version indexed → transition SUCCEEDED.
*/
private void drainEmbedQueue() {
log.info("embed worker started ({})", Thread.currentThread().getName());
while (!shuttingDown || !embedQueue.isEmpty()) {
ParsedBatch batch;
try {
batch = embedQueue.poll(500, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
break;
}
if (batch == null) continue;
runEmbedPhase(batch);
}
log.info("embed worker stopped");
}
private void runEmbedPhase(ParsedBatch batch) {
MDC.put("jobId", batch.jobId().toString());
MDC.put("repo", batch.repo().name());
MDC.put("tag", batch.ver().tag());
try {
// STAGE: EMBED
List<Chunk> chunks = stageReturning(batch.jobId(), JobStage.StageName.EMBED, () ->
embedAll(batch.jobId(), batch.pieces()));
// STAGE: INDEX
stage(batch.jobId(), JobStage.StageName.INDEX, () -> {
chunkStore.unlinkVersion(batch.ver().id());
List<ChunkVersion> links = buildLinks(batch.ver().id(), batch.pieces());
chunkStore.linkChunks(links);
return (long) links.size();
});
// STAGE: COMMIT
stage(batch.jobId(), JobStage.StageName.COMMIT, () -> {
chunkStore.commit();
return 1L;
});
repoStore.markVersionIndexed(batch.ver().id(), batch.pieces().size());
transitionJob(batch.jobId(), JobStatus.SUCCEEDED, null, Instant.now());
logEvent(batch.jobId(), JobLogEvent.Level.INFO, null,
"indexed " + chunks.size() + " chunks across " + batch.pieces().size() + " pieces");
} catch (Exception e) {
log.error("embed phase failed for job {}", batch.jobId(), e);
logEvent(batch.jobId(), JobLogEvent.Level.ERROR, null,
"embed phase failed: " + e.getMessage());
repoStore.updateVersionStatus(batch.ver().id(), VersionStatus.FAILED, e.getMessage());
transitionJob(batch.jobId(), JobStatus.FAILED, null, Instant.now());
} finally {
running.remove(batch.jobId());
MDC.clear();
}
}
private void removeWorktreeQuietly(JobId jobId, Path repoPath, Path worktree) {
try {
git.removeWorktree(repoPath, worktree);
} catch (Exception e) {
logEvent(jobId, JobLogEvent.Level.WARN, null, "worktree cleanup failed: " + e.getMessage());
}
}
/**
* Orderly shutdown: stops the parse executor, signals the embed worker to stop after
* finishing its current batch, then fails any batches still in the queue.
* Called by Spring via {@code @Bean(destroyMethod = "shutdown")} in ApplicationBeans.
*/
void shutdown() {
log.info("IngestionOrchestrator shutting down — stopping embed worker");
shuttingDown = true;
parseExecutor.shutdownNow();
embedWorker.interrupt();
try {
embedWorker.join(10_000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
// Fail any batches that parsed OK but never got to embed (restart mid-queue).
ParsedBatch orphan;
while ((orphan = embedQueue.poll()) != null) {
log.warn("failing orphaned embed batch for job {} (shutdown)", orphan.jobId());
repoStore.updateVersionStatus(orphan.ver().id(), VersionStatus.FAILED, "application shutdown");
transitionJob(orphan.jobId(), JobStatus.FAILED, null, Instant.now());
running.remove(orphan.jobId());
}
}
/* ------------------------------------------------------------------ */
/* Stage helpers */
/* ------------------------------------------------------------------ */
private interface StageBody {
long execute() throws Exception;
}
private interface StageBodyReturning<T> {
T execute() throws Exception;
}
private void stage(JobId id, JobStage.StageName name, StageBody body) {
stageReturning(id, name, () -> {
long n = body.execute();
return n;
});
}
private <T> T stageReturning(JobId id, JobStage.StageName name, StageBodyReturning<T> body) {
Instant start = Instant.now();
JobStage running = new JobStage(
id, name, JobStage.StageStatus.RUNNING, start, null, 0, 0, 0, null);
jobStore.upsertStage(running);
publishJob(id);
try {
T out = body.execute();
long items = (out instanceof Long l) ? l : (out instanceof List<?> l ? l.size() : 1);
JobStage done = new JobStage(
id, name, JobStage.StageStatus.SUCCEEDED, start, Instant.now(), items, items, 0, null);
jobStore.upsertStage(done);
publishJob(id);
return out;
} catch (Exception e) {
JobStage failed = new JobStage(
id, name, JobStage.StageStatus.FAILED, start, Instant.now(), 0, 0, 0, e.getMessage());
jobStore.upsertStage(failed);
publishJob(id);
throw new RuntimeException(e);
}
}
private void transitionJob(JobId id, JobStatus s, Instant startedAt, Instant finishedAt) {
jobStore.updateStatus(id, s, startedAt, finishedAt);
publishJob(id);
}
private void publishJob(JobId id) {
jobStore.findById(id).ifPresent(bus::publishJob);
}
private void logEvent(JobId id, JobLogEvent.Level level, JobStage.StageName stage, String msg) {
bus.publishLog(new JobLogEvent(id, Instant.now(), level, stage, msg));
}
/* ------------------------------------------------------------------ */
/* Pipeline steps */
/* ------------------------------------------------------------------ */
private List<Path> discoverFiles(Path root, Repository repo) throws IOException {
List<PathMatcher> matchers = new ArrayList<>();
for (String g : BUILTIN_IGNORES) matchers.add(FileSystems.getDefault().getPathMatcher("glob:" + g));
for (String g : repo.ignoreGlobs()) matchers.add(FileSystems.getDefault().getPathMatcher("glob:" + g));
long maxBytes = repo.maxFileSizeBytes();
List<Path> out = new ArrayList<>();
Files.walkFileTree(root, new SimpleFileVisitor<>() {
@Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) {
Path rel = root.relativize(file);
for (PathMatcher m : matchers) {
if (m.matches(rel) || m.matches(file.getFileName())) return FileVisitResult.CONTINUE;
}
if (attrs.size() > maxBytes) return FileVisitResult.CONTINUE;
out.add(file);
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) {
if (dir.equals(root)) return FileVisitResult.CONTINUE;
Path rel = root.relativize(dir);
for (PathMatcher m : matchers) {
if (m.matches(rel)) return FileVisitResult.SKIP_SUBTREE;
}
return FileVisitResult.CONTINUE;
}
});
return out;
}
private record ParsedPiece(
String contentHash,
String content,
String language,
String symbol,
int tokenCount,
String filePath,
int startLine,
int endLine) {}
private List<ParsedPiece> parseAll(List<Path> files, Path root) {
List<ParsedPiece> out = new ArrayList<>();
MessageDigest sha;
try {
sha = MessageDigest.getInstance("SHA-256");
} catch (Exception e) {
throw new IllegalStateException(e);
}
for (Path f : files) {
String rel = root.relativize(f).toString().replace('\\', '/');
try {
List<CodeParser.ParsedChunk> parsed = parser.parse(f, rel);
for (var pc : parsed) {
String hash = bytesToHex(sha.digest(pc.content().getBytes(StandardCharsets.UTF_8)));
sha.reset();
int tokens = Math.max(1, pc.content().length() / 4); // heuristic; refined if needed
out.add(new ParsedPiece(
hash, pc.content(), pc.language(), pc.symbol(), tokens, rel, pc.startLine(), pc.endLine()));
}
} catch (Exception e) {
log.warn("parse failed for {}: {}", rel, e.toString());
}
}
return out;
}
private List<Chunk> embedAll(JobId jobId, List<ParsedPiece> pieces) {
// Dedupe by hash across this batch AND against existing chunks in the store/cache.
Map<String, Chunk> resolved = new HashMap<>();
List<ParsedPiece> toEmbed = new ArrayList<>();
for (var p : pieces) {
if (resolved.containsKey(p.contentHash())) continue;
var existing = chunkStore.findByContentHash(p.contentHash());
if (existing.isPresent()) {
resolved.put(p.contentHash(), existing.get());
continue;
}
// cache?
var cached = embeddingCache.get(p.contentHash());
if (cached.isPresent()) {
Chunk c = upsert(p, cached.get());
resolved.put(p.contentHash(), c);
continue;
}
toEmbed.add(p);
}
if (!toEmbed.isEmpty()) {
List<String> texts = toEmbed.stream().map(ParsedPiece::content).toList();
List<float[]> vecs = embeddings.embed(texts);
for (int i = 0; i < toEmbed.size(); i++) {
var p = toEmbed.get(i);
float[] v = vecs.get(i);
embeddingCache.put(p.contentHash(), v);
Chunk c = upsert(p, v);
resolved.put(p.contentHash(), c);
}
logEvent(jobId, JobLogEvent.Level.INFO, JobStage.StageName.EMBED,
"embedded " + toEmbed.size() + " new chunks (cache/dedupe hits = " + (pieces.size() - toEmbed.size()) + ")");
}
return new ArrayList<>(resolved.values());
}
private Chunk upsert(ParsedPiece p, float[] vector) {
Chunk c = new Chunk(ChunkId.random(), p.contentHash(), p.content(), p.language(), p.symbol(), p.tokenCount());
return chunkStore.upsertChunk(c, new Embedding(c.id(), vector));
}
private List<ChunkVersion> buildLinks(VersionId versionId, List<ParsedPiece> pieces) {
// Piece → ChunkId requires knowing the chunk id assigned on upsert.
// We re-resolve via findByContentHash — cheap because it's a Term query.
List<ChunkVersion> links = new ArrayList<>(pieces.size());
Map<String, ChunkId> hashToId = new HashMap<>();
for (var p : pieces) {
ChunkId id = hashToId.computeIfAbsent(p.contentHash(), h ->
chunkStore.findByContentHash(h).orElseThrow().id());
links.add(new ChunkVersion(id, versionId, p.filePath(), p.startLine(), p.endLine()));
}
return links;
}
private String pickParentIndexedTag(Repository repo, Version ver) {
// Most recent previously-indexed version for this repo that isn't this one.
List<Version> indexed = repoStore.findVersionsByStatus(repo.id(), VersionStatus.INDEXED);
return indexed.stream()
.filter(v -> !v.id().equals(ver.id()))
.max((a, b) -> a.tag().compareTo(b.tag()))
.map(Version::tag)
.orElse(null);
}
private static String bytesToHex(byte[] bytes) {
char[] hex = "0123456789abcdef".toCharArray();
char[] out = new char[bytes.length * 2];
for (int i = 0; i < bytes.length; i++) {
int v = bytes[i] & 0xff;
out[i * 2] = hex[v >>> 4];
out[i * 2 + 1] = hex[v & 0x0f];
}
return new String(out);
}
}

71
trueref-application/src/main/java/com/trueref/application/observability/InMemoryJobEventBus.java Normal file
View File

@@ -0,0 +1,71 @@
package com.trueref.application.observability;
import com.trueref.domain.model.IngestionJob;
import com.trueref.domain.model.JobId;
import com.trueref.domain.model.JobLogEvent;
import com.trueref.domain.port.out.JobEventBus;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.function.Consumer;
/**
* In-process publish/subscribe implementation of {@link JobEventBus}. Listeners receive events on
* the publisher's thread; consumers should defer expensive work (e.g. SSE writes) to a virtual
* thread to keep the publisher fast.
*/
public final class InMemoryJobEventBus implements JobEventBus {
private final CopyOnWriteArrayList<Consumer<IngestionJob>> jobListeners = new CopyOnWriteArrayList<>();
private final Map<JobId, CopyOnWriteArrayList<Consumer<JobLogEvent>>> logListeners = new ConcurrentHashMap<>();
private final CopyOnWriteArrayList<Consumer<JobLogEvent>> globalLogListeners = new CopyOnWriteArrayList<>();
@Override
public void publishJob(IngestionJob job) {
for (Consumer<IngestionJob> l : jobListeners) {
try {
l.accept(job);
} catch (Exception ignored) {
// listener failures must not break publishing
}
}
}
@Override
public void publishLog(JobLogEvent event) {
var perJob = logListeners.get(event.jobId());
if (perJob != null) {
for (Consumer<JobLogEvent> l : perJob) {
try {
l.accept(event);
} catch (Exception ignored) {
}
}
}
for (Consumer<JobLogEvent> l : globalLogListeners) {
try {
l.accept(event);
} catch (Exception ignored) {
}
}
}
@Override
public AutoCloseable subscribeJobs(Consumer<IngestionJob> listener) {
jobListeners.add(listener);
return () -> jobListeners.remove(listener);
}
@Override
public AutoCloseable subscribeLogs(JobId jobId, Consumer<JobLogEvent> listener) {
var list = logListeners.computeIfAbsent(jobId, k -> new CopyOnWriteArrayList<>());
list.add(listener);
return () -> list.remove(listener);
}
/** Subscribe to ALL log events regardless of job (used by the dashboard). */
public AutoCloseable subscribeAllLogs(Consumer<JobLogEvent> listener) {
globalLogListeners.add(listener);
return () -> globalLogListeners.remove(listener);
}
}

47
trueref-application/src/main/java/com/trueref/application/observability/JobObservationService.java Normal file
View File

@@ -0,0 +1,47 @@
package com.trueref.application.observability;
import com.trueref.domain.model.IngestionJob;
import com.trueref.domain.model.JobId;
import com.trueref.domain.model.JobLogEvent;
import com.trueref.domain.model.JobStatus;
import com.trueref.domain.model.RepositoryId;
import com.trueref.domain.model.VersionId;
import com.trueref.domain.port.in.ObserveJobs;
import com.trueref.domain.port.out.JobEventBus;
import com.trueref.domain.port.out.JobStore;
import java.util.List;
import java.util.Optional;
import java.util.function.Consumer;
import org.jspecify.annotations.Nullable;
public final class JobObservationService implements ObserveJobs {
private final JobStore jobs;
private final JobEventBus bus;
public JobObservationService(JobStore jobs, JobEventBus bus) {
this.jobs = jobs;
this.bus = bus;
}
@Override
public Optional<IngestionJob> findJob(JobId id) {
return jobs.findById(id);
}
@Override
public List<IngestionJob> listJobs(
@Nullable RepositoryId repoId, @Nullable VersionId versionId, @Nullable JobStatus status, int limit) {
return jobs.find(repoId, versionId, status, limit);
}
@Override
public AutoCloseable subscribeJobs(Consumer<IngestionJob> listener) {
return bus.subscribeJobs(listener);
}
@Override
public AutoCloseable subscribeLogs(JobId jobId, Consumer<JobLogEvent> listener) {
return bus.subscribeLogs(jobId, listener);
}
}

3
trueref-application/src/main/java/com/trueref/application/observability/package-info.java Normal file
View File

@@ -0,0 +1,3 @@
/** In-process implementations of cross-cutting application services. */
@org.jspecify.annotations.NullMarked
package com.trueref.application.observability;

3
trueref-application/src/main/java/com/trueref/application/package-info.java Normal file
View File

@@ -0,0 +1,3 @@
/** Application services: use-case implementations. */
@org.jspecify.annotations.NullMarked
package com.trueref.application;

161
trueref-application/src/main/java/com/trueref/application/resolve/LibraryResolver.java Normal file
View File

@@ -0,0 +1,161 @@
package com.trueref.application.resolve;
import com.trueref.domain.model.Repository;
import com.trueref.domain.model.RepositoryId;
import com.trueref.domain.model.TagPattern;
import com.trueref.domain.model.Version;
import com.trueref.domain.model.VersionStatus;
import com.trueref.domain.port.in.IndexVersion;
import com.trueref.domain.port.in.ResolveLibraryId;
import com.trueref.domain.port.out.RepositoryStore;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Optional;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.jspecify.annotations.Nullable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Fuzzy library-name matching + version→tag mapping. Mirrors Context7's {@code resolve-library-id}
* semantics. When {@code version} is provided and maps to a known-but-not-yet-indexed tag, triggers
* an async index job (fire-and-forget).
*/
public final class LibraryResolver implements ResolveLibraryId {
private static final Logger log = LoggerFactory.getLogger(LibraryResolver.class);
private static final Pattern SEMVER = Pattern.compile("^v?(\\d+)(?:\\.(\\d+))?(?:\\.(\\d+))?.*$");
private final RepositoryStore store;
private final IndexVersion indexer;
public LibraryResolver(RepositoryStore store, IndexVersion indexer) {
this.store = store;
this.indexer = indexer;
}
@Override
public Result resolve(Query q) {
String needle = q.libraryName().toLowerCase();
List<Repository> all = store.findAll();
List<Match> matches = new ArrayList<>();
for (Repository r : all) {
double score = nameScore(r.name().toLowerCase(), needle);
if (score <= 0) continue;
List<Version> versions = store.findVersionsByRepo(r.id());
// If a version was requested, map it to a tag and ensure indexing.
if (q.version() != null && !q.version().isBlank()) {
Optional<Version> target = mapVersion(r, versions, q.version());
target.ifPresent(v -> ensureIndexed(r.id(), v));
}
int snippetCount = versions.stream()
.filter(v -> v.status() == VersionStatus.INDEXED)
.mapToInt(Version::chunkCount)
.sum();
List<VersionRef> refs = versions.stream()
.sorted(Comparator.comparing(Version::tag).reversed())
.map(v -> new VersionRef(v.id(), v.tag(), v.status()))
.toList();
String libraryId = "/" + r.name();
matches.add(new Match(r.id(), libraryId, r.name(), null, snippetCount, refs, score));
}
matches.sort(Comparator.comparingDouble(Match::score).reversed());
return new Result(matches);
}
/** Fuzzy name scoring: exact 1.0, prefix 0.9, contains 0.7, token overlap otherwise. */
private double nameScore(String haystack, String needle) {
if (haystack.equals(needle)) return 1.0;
if (haystack.endsWith("/" + needle) || haystack.startsWith(needle + "/")) return 0.95;
if (haystack.contains(needle)) return 0.8;
// token overlap
String[] hTok = haystack.split("[^a-z0-9]+");
String[] nTok = needle.split("[^a-z0-9]+");
int hit = 0;
for (String nt : nTok) {
if (nt.isBlank()) continue;
for (String ht : hTok) {
if (ht.equals(nt)) { hit++; break; }
}
}
if (hit == 0) return 0.0;
return 0.3 + 0.4 * ((double) hit / Math.max(1, nTok.length));
}
/**
* Maps a version string to the closest matching tag using the repo's configured mapping rules.
* Rules are tried in order.
*/
public Optional<Version> mapVersion(Repository repo, List<Version> versions, String requested) {
for (TagPattern rule : repo.versionMappingRules()) {
String candidate = switch (rule) {
case TagPattern.Exact e -> requested;
case TagPattern.VPrefix v -> "v" + stripV(requested);
case TagPattern.ReleasePrefix r -> "release-" + stripV(requested);
case TagPattern.Custom c -> c.template()
.replace("{version}", requested)
.replace("{semver}", stripV(requested));
case TagPattern.SemverFuzzy s -> null; // handled below
};
if (candidate == null) continue;
Optional<Version> exact = versions.stream()
.filter(v -> v.tag().equalsIgnoreCase(candidate))
.findFirst();
if (exact.isPresent()) return exact;
}
// Semver fuzzy: pick tag with closest semver distance
return semverClosest(versions, requested);
}
private Optional<Version> semverClosest(List<Version> versions, String requested) {
int[] r = parseSemver(requested);
if (r == null) return Optional.empty();
return versions.stream()
.map(v -> new Object[] {v, parseSemver(v.tag())})
.filter(t -> t[1] != null)
.min(Comparator.comparingLong(t -> semverDist((int[]) t[1], r)))
.map(t -> (Version) t[0]);
}
private static @Nullable int[] parseSemver(String s) {
Matcher m = SEMVER.matcher(s);
if (!m.matches()) return null;
return new int[] {
parseIntOrZero(m.group(1)),
parseIntOrZero(m.group(2)),
parseIntOrZero(m.group(3))
};
}
private static int parseIntOrZero(String s) {
if (s == null || s.isEmpty()) return 0;
try { return Integer.parseInt(s); } catch (NumberFormatException e) { return 0; }
}
private static long semverDist(int[] a, int[] b) {
long d = 0;
d += Math.abs(a[0] - b[0]) * 1_000_000L;
d += Math.abs(a[1] - b[1]) * 1_000L;
d += Math.abs(a[2] - b[2]);
return d;
}
private static String stripV(String s) {
return s.startsWith("v") || s.startsWith("V") ? s.substring(1) : s;
}
private void ensureIndexed(RepositoryId repoId, Version v) {
if (v.status() == VersionStatus.INDEXED || v.status() == VersionStatus.INDEXING) return;
try {
log.info("on-demand indexing: repo={} tag={}", repoId, v.tag());
indexer.enqueue(repoId, v.id(), false);
} catch (Exception e) {
log.warn("on-demand indexing enqueue failed: {}", e.toString());
}
}
}

238
trueref-application/src/main/java/com/trueref/application/search/HybridSearchService.java Normal file
View File

@@ -0,0 +1,238 @@
package com.trueref.application.search;
import com.trueref.domain.error.InvalidSearchRequest;
import com.trueref.domain.model.ChunkId;
import com.trueref.domain.model.Repository;
import com.trueref.domain.model.SearchHit;
import com.trueref.domain.model.SearchScope;
import com.trueref.domain.model.Version;
import com.trueref.domain.port.in.SearchLibraryDocs;
import com.trueref.domain.port.out.ChunkStore;
import com.trueref.domain.port.out.EmbeddingService;
import com.trueref.domain.port.out.RepositoryStore;
import com.trueref.domain.port.out.RerankerService;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Hybrid search: BM25 + dense kNN fused by Reciprocal Rank Fusion (RRF), then reranked by a
* cross-encoder, then packed to a token budget.
*/
public final class HybridSearchService implements SearchLibraryDocs {
private static final Logger log = LoggerFactory.getLogger(HybridSearchService.class);
/**
* Matches camelCase identifiers that are likely to be Phaser API method/class names (≥6 chars,
* must contain at least one uppercase letter after the first char, not all-caps).
* Examples: setCollideWorldBounds, createBitmapMask, addOverlap.
*/
private static final Pattern CAMEL_IDENT = Pattern.compile(
"\\b([a-z][a-zA-Z0-9]{5,})(?=\\b)");
private final ChunkStore chunks;
private final EmbeddingService embedder;
private final RerankerService reranker;
private final RepositoryStore repos;
private final int rrfK;
private final int rerankTopK;
private final int finalTopK;
public HybridSearchService(
ChunkStore chunks,
EmbeddingService embedder,
RerankerService reranker,
RepositoryStore repos,
int rrfK,
int rerankTopK,
int finalTopK) {
this.chunks = chunks;
this.embedder = embedder;
this.reranker = reranker;
this.repos = repos;
this.rrfK = rrfK;
this.rerankTopK = rerankTopK;
this.finalTopK = finalTopK;
}
@Override
public Result search(Query q) {
if (q.text() == null || q.text().isBlank()) {
throw new InvalidSearchRequest("query text must not be blank");
}
if (q.scope().refs().isEmpty()) {
throw new InvalidSearchRequest("search scope must not be empty");
}
String text = rewrite(q.text(), q.topic());
// Augment BM25 query with camelCase identifiers found in the text so that the exact
// method-name chunk scores higher in BM25 even when it competes with generic mentions.
String bm25Text = augmentWithCamelIdents(text);
List<SearchHit> bm25 = chunks.bm25Search(bm25Text, q.scope(), rerankTopK);
float[] vec = embedder.embed(List.of(text)).get(0);
List<SearchHit> dense = chunks.denseSearch(vec, q.scope(), rerankTopK);
List<SearchHit> fused = rrf(bm25, dense);
if (fused.size() > rerankTopK) fused = fused.subList(0, rerankTopK);
// Demote changelog / synthetic-skill / docs paths before the reranker sees them so that
// authoritative source-code chunks aren't squeezed out by historical migration notes.
List<SearchHit> biased = applyFilePathBias(fused);
// Enrich with repo name + tag (ChunkStore leaves these empty).
List<SearchHit> enriched = enrich(biased);
List<SearchHit> reranked = reranker.rerank(text, enriched);
List<SearchHit> packed = packByTokenBudget(reranked, q.tokensBudget(), q.maxHits() > 0 ? q.maxHits() : finalTopK);
int totalTokens = packed.stream().mapToInt(h -> estimateTokens(h.content())).sum();
return new Result(packed, totalTokens);
}
/* ------------------------------------------------------------------ */
private String rewrite(String text, String topic) {
String base = text.trim();
if (topic != null && !topic.isBlank()) {
return base + " " + topic.trim();
}
return base;
}
/**
* Returns a copy of {@code text} with each camelCase identifier repeated at the end (once).
* This lifts their BM25 term-frequency contribution without altering the semantic meaning
* used for the dense embedding query.
*
* <p>Example: "how to use setCollideWorldBounds" →
* "how to use setCollideWorldBounds setCollideWorldBounds"
*/
private static String augmentWithCamelIdents(String text) {
Matcher m = CAMEL_IDENT.matcher(text);
StringBuilder extra = new StringBuilder();
while (m.find()) {
String ident = m.group(1);
// Only repeat identifiers that contain at least one uppercase letter
// (filters out short common words like "should", "create").
if (!ident.equals(ident.toLowerCase())) {
extra.append(' ').append(ident);
}
}
return extra.isEmpty() ? text : text + extra;
}
/**
* Applies a path-based multiplier to RRF scores before handing candidates to the reranker.
* Changelogs and synthetic skill docs are semantically relevant but tend to outrank the
* authoritative source-code chunks when the query mentions API migration or breaking changes.
* Demoting them here keeps them retrievable while giving source files priority.
*
* <p>Multipliers (tuned against the phaser_rag_eval suite):
* <ul>
* <li>{@code changelog/} → ×0.50 — migration notes, not current API reference
* <li>{@code skills/} / {@code SKILL.md} → ×0.60 — synthetic summaries, not authoritative
* <li>{@code docs/} → ×0.75 — curated docs; useful but prefer source JSDoc
* <li>everything else (source, tests, configs) → ×1.0
* </ul>
*/
private static List<SearchHit> applyFilePathBias(List<SearchHit> hits) {
boolean anyChanged = false;
List<SearchHit> out = new ArrayList<>(hits.size());
for (SearchHit h : hits) {
double mult = filePathMultiplier(h.filePath());
if (mult == 1.0) {
out.add(h);
} else {
out.add(new SearchHit(
h.chunkId(), h.repoId(), h.versionId(), h.repoName(), h.tag(),
h.filePath(), h.startLine(), h.endLine(), h.language(), h.symbol(),
h.content(), h.score() * mult));
anyChanged = true;
}
}
if (!anyChanged) return hits;
out.sort(Comparator.comparingDouble(SearchHit::score).reversed());
return out;
}
private static double filePathMultiplier(String filePath) {
if (filePath == null || filePath.isEmpty()) return 1.0;
String lp = filePath.toLowerCase();
if (lp.startsWith("changelog/") || lp.contains("/changelog/")) return 0.50;
if (lp.contains("/skills/") || lp.endsWith("skill.md")) return 0.60;
if (lp.startsWith("docs/") || lp.contains("/docs/")) return 0.75;
return 1.0;
}
private List<SearchHit> rrf(List<SearchHit> a, List<SearchHit> b) {
Map<ChunkId, Double> scores = new HashMap<>();
Map<ChunkId, SearchHit> firstSeen = new HashMap<>();
addRankContribution(a, scores, firstSeen);
addRankContribution(b, scores, firstSeen);
return scores.entrySet().stream()
.sorted(Map.Entry.<ChunkId, Double>comparingByValue().reversed())
.map(e -> {
SearchHit h = firstSeen.get(e.getKey());
return new SearchHit(
h.chunkId(), h.repoId(), h.versionId(), h.repoName(), h.tag(),
h.filePath(), h.startLine(), h.endLine(), h.language(), h.symbol(),
h.content(), e.getValue());
})
.toList();
}
private void addRankContribution(List<SearchHit> hits, Map<ChunkId, Double> scores, Map<ChunkId, SearchHit> seen) {
for (int rank = 0; rank < hits.size(); rank++) {
SearchHit h = hits.get(rank);
scores.merge(h.chunkId(), 1.0 / (rrfK + rank + 1.0), Double::sum);
seen.putIfAbsent(h.chunkId(), h);
}
}
private List<SearchHit> enrich(List<SearchHit> hits) {
Map<String, String> repoNameByRepoId = new HashMap<>();
Map<String, String> tagByVersionId = new HashMap<>();
List<SearchHit> out = new ArrayList<>(hits.size());
for (SearchHit h : hits) {
String repoName = repoNameByRepoId.computeIfAbsent(
h.repoId().toString(),
k -> repos.findById(h.repoId()).map(Repository::name).orElse("?"));
String tag = tagByVersionId.computeIfAbsent(
h.versionId().toString(),
k -> repos.findVersion(h.versionId()).map(Version::tag).orElse("?"));
out.add(new SearchHit(
h.chunkId(), h.repoId(), h.versionId(),
repoName, tag,
h.filePath(), h.startLine(), h.endLine(), h.language(), h.symbol(),
h.content(), h.score()));
}
return out;
}
private List<SearchHit> packByTokenBudget(List<SearchHit> ranked, int tokenBudget, int maxHits) {
List<SearchHit> out = new ArrayList<>();
int used = 0;
for (SearchHit h : ranked) {
if (out.size() >= maxHits) break;
int t = estimateTokens(h.content());
if (used + t > tokenBudget && !out.isEmpty()) break;
out.add(h);
used += t;
}
return out;
}
/** 4 chars ≈ 1 token — same rule of thumb Context7 uses for packing. */
private static int estimateTokens(String s) {
return Math.max(1, s.length() / 4);
}
}

68
trueref-bootstrap/pom.xml Normal file
View File

@@ -0,0 +1,68 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.trueref</groupId>
<artifactId>trueref-parent</artifactId>
<version>0.1.0-SNAPSHOT</version>
</parent>
<artifactId>trueref-bootstrap</artifactId>
<name>trueref-bootstrap</name>
<description>Spring Boot entry point. Wires beans across modules. Produces the executable fat JAR.</description>
<dependencies>
<dependency>
<groupId>com.trueref</groupId>
<artifactId>trueref-domain</artifactId>
</dependency>
<dependency>
<groupId>com.trueref</groupId>
<artifactId>trueref-application</artifactId>
</dependency>
<dependency>
<groupId>com.trueref</groupId>
<artifactId>trueref-adapters</artifactId>
</dependency>
<dependency>
<groupId>com.trueref</groupId>
<artifactId>trueref-frontend</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<finalName>trueref</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<mainClass>com.trueref.bootstrap.TrueRefApplication</mainClass>
</configuration>
<executions>
<execution>
<goals><goal>repackage</goal></goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>

89
trueref-bootstrap/src/main/java/com/trueref/bootstrap/ApplicationBeans.java Normal file
View File

@@ -0,0 +1,89 @@
package com.trueref.bootstrap;
import com.trueref.application.catalog.CatalogService;
import com.trueref.application.ingest.DiscoveryService;
import com.trueref.application.ingest.IngestionOrchestrator;
import com.trueref.application.observability.InMemoryJobEventBus;
import com.trueref.application.observability.JobObservationService;
import com.trueref.application.resolve.LibraryResolver;
import com.trueref.application.search.HybridSearchService;
import com.trueref.domain.port.out.ChunkStore;
import com.trueref.domain.port.out.CodeParser;
import com.trueref.domain.port.out.EmbeddingCache;
import com.trueref.domain.port.out.EmbeddingService;
import com.trueref.domain.port.out.GitClient;
import com.trueref.domain.port.out.JobStore;
import com.trueref.domain.port.out.RepositoryStore;
import com.trueref.domain.port.out.RerankerService;
import java.nio.file.Path;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
/**
* Explicit bean wiring for the application layer (which stays Spring-annotation-free).
* We expose only concrete beans; Spring resolves interface dependencies against the single
* concrete implementation.
*/
@Configuration
public class ApplicationBeans {
@Bean
Path trueRefHome(@Value("${trueref.home:./data}") String home) {
return Path.of(home);
}
@Bean
InMemoryJobEventBus jobEventBus() {
return new InMemoryJobEventBus();
}
@Bean
CatalogService catalogService(RepositoryStore store, Path trueRefHome) {
return new CatalogService(store, trueRefHome);
}
@Bean
DiscoveryService discoveryService(RepositoryStore store, GitClient git) {
return new DiscoveryService(store, git);
}
@Bean(destroyMethod = "shutdown")
IngestionOrchestrator ingestionOrchestrator(
RepositoryStore repoStore,
JobStore jobStore,
ChunkStore chunkStore,
EmbeddingService embeddings,
EmbeddingCache embeddingCache,
GitClient git,
CodeParser parser,
InMemoryJobEventBus bus,
@Value("${trueref.ingestion.max-parse-jobs:4}") int maxParseJobs,
@Value("${trueref.ingestion.embed-queue-capacity:4}") int embedQueueCapacity) {
return new IngestionOrchestrator(
repoStore, jobStore, chunkStore, embeddings, embeddingCache, git, parser, bus,
maxParseJobs, embedQueueCapacity);
}
@Bean
LibraryResolver libraryResolver(RepositoryStore store, IngestionOrchestrator indexer) {
return new LibraryResolver(store, indexer);
}
@Bean
HybridSearchService hybridSearchService(
ChunkStore chunks,
EmbeddingService embedder,
RerankerService reranker,
RepositoryStore repos,
@Value("${trueref.search.rrf-k:60}") int rrfK,
@Value("${trueref.reranker.top-k:50}") int rerankTopK,
@Value("${trueref.search.final-top-k:20}") int finalTopK) {
return new HybridSearchService(chunks, embedder, reranker, repos, rrfK, rerankTopK, finalTopK);
}
@Bean
JobObservationService jobObservationService(JobStore jobs, InMemoryJobEventBus bus) {
return new JobObservationService(jobs, bus);
}
}

57
trueref-bootstrap/src/main/java/com/trueref/bootstrap/ScheduledPoller.java Normal file
View File

@@ -0,0 +1,57 @@
package com.trueref.bootstrap;
import com.trueref.application.ingest.DiscoveryService;
import com.trueref.domain.model.Repository;
import com.trueref.domain.model.Version;
import com.trueref.domain.model.VersionStatus;
import com.trueref.domain.port.in.IndexVersion;
import com.trueref.domain.port.out.RepositoryStore;
import java.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
/** Periodically fetches tags for registered repos and enqueues indexing for new ones. */
@Component
@EnableScheduling
@ConditionalOnProperty(name = "trueref.ingestion.poller-enabled", havingValue = "true", matchIfMissing = true)
public class ScheduledPoller {
private static final Logger log = LoggerFactory.getLogger(ScheduledPoller.class);
private final RepositoryStore repoStore;
private final DiscoveryService discovery;
private final IndexVersion indexer;
public ScheduledPoller(RepositoryStore repoStore, DiscoveryService discovery, IndexVersion indexer) {
this.repoStore = repoStore;
this.discovery = discovery;
this.indexer = indexer;
}
@Scheduled(fixedDelayString = "${trueref.ingestion.poll-interval-default:PT1H}")
public void pollAll() {
Instant start = Instant.now();
int scanned = 0;
int enqueued = 0;
for (Repository repo : repoStore.findAll()) {
try {
discovery.discover(repo.id());
for (Version v : repoStore.findVersionsByRepo(repo.id())) {
if (v.status() == VersionStatus.DISCOVERED) {
indexer.enqueue(repo.id(), v.id(), false);
enqueued++;
}
}
scanned++;
} catch (Exception e) {
log.warn("poll failed for repo={}: {}", repo.name(), e.toString());
}
}
log.info("poll completed in {}ms: repos scanned={} jobs enqueued={}",
java.time.Duration.between(start, Instant.now()).toMillis(), scanned, enqueued);
}
}

59
trueref-bootstrap/src/main/java/com/trueref/bootstrap/StaleJobCleanupStartup.java Normal file
View File

@@ -0,0 +1,59 @@
package com.trueref.bootstrap;
import com.trueref.domain.port.out.JobStore;
import com.trueref.domain.port.out.RepositoryStore;
import java.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;
/**
* On-startup cleanup for stale job state left by a previous crash or SIGKILL.
*
* <p>Any job that is RUNNING or QUEUED when the application starts must have been orphaned by a
* previous JVM exit (clean or unclean). We fail them all atomically before accepting traffic so
* the UI and API never show phantom RUNNING jobs. Matching INDEXING versions are also reset to
* FAILED so they can be re-queued immediately.
*
* <p>This fires <em>after</em> Flyway migrations and all beans are initialised, but before the
* application starts accepting HTTP requests (ApplicationReadyEvent fires before the embedded
* Tomcat connector starts accepting connections).
*/
@Component
class StaleJobCleanupStartup {
private static final Logger log = LoggerFactory.getLogger(StaleJobCleanupStartup.class);
private static final String RESTART_REASON = "interrupted by server restart";
private final JobStore jobStore;
private final RepositoryStore repositoryStore;
StaleJobCleanupStartup(JobStore jobStore, RepositoryStore repositoryStore) {
this.jobStore = jobStore;
this.repositoryStore = repositoryStore;
}
@EventListener(ApplicationReadyEvent.class)
public void cleanupStaleJobs() {
Instant now = Instant.now();
int failedJobs = jobStore.failStaleJobs(now);
if (failedJobs > 0) {
log.warn(
"Startup cleanup: marked {} orphaned job(s) as FAILED (were RUNNING or QUEUED at shutdown).",
failedJobs);
} else {
log.info("Startup cleanup: no stale jobs found.");
}
int failedVersions = repositoryStore.failStaleIndexingVersions(RESTART_REASON);
if (failedVersions > 0) {
log.warn(
"Startup cleanup: reset {} INDEXING version(s) to FAILED (their jobs did not complete).",
failedVersions);
}
}
}

17
trueref-bootstrap/src/main/java/com/trueref/bootstrap/TrueRefApplication.java Normal file
View File

@@ -0,0 +1,17 @@
package com.trueref.bootstrap;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
/**
* Trueref entry point. The only place where Spring component scanning is allowed across the
* {@code com.trueref} package tree. Adapters and application modules expose explicit
* {@code @Configuration} classes that this class imports via component scanning.
*/
@SpringBootApplication(scanBasePackages = "com.trueref")
public class TrueRefApplication {
public static void main(String[] args) {
SpringApplication.run(TrueRefApplication.class, args);
}
}

114
trueref-bootstrap/src/main/resources/application.yml Normal file
View File

@@ -0,0 +1,114 @@
spring:
application:
name: trueref
threads:
virtual:
enabled: true
datasource:
url: jdbc:h2:file:${trueref.home:./data}/h2/trueref;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE;MV_STORE=TRUE
username: sa
password: ""
driver-class-name: org.h2.Driver
hikari:
# Embedded H2 serialises writes internally; 8 connections is ample for virtual-thread
# workloads. 32 is wasteful and causes unnecessary H2 lock contention.
maximum-pool-size: 8
minimum-idle: 2
flyway:
enabled: true
locations: classpath:db/migration
mvc:
async:
request-timeout: 0 # SSE streams must not time out
# Spring AI MCP server. In Spring AI 1.0.0 the WebMVC transport is SSE-based
# (WebMvcSseServerTransportProvider) — the closest available transport to the 2025-03-26
# "Streamable HTTP" spec; there is no separate "protocol: streamable" property in this
# starter. JSON-RPC POSTs land on `sse-message-endpoint` (/mcp); server-initiated
# notifications stream over `sse-endpoint` (/sse). See com.trueref.adapter.in.mcp.
ai:
mcp:
server:
enabled: true
name: trueref
version: 0.1.0
type: SYNC
sse-message-endpoint: /mcp
sse-endpoint: /sse
server:
port: 8080
shutdown: graceful
management:
endpoints:
web:
exposure:
include: health,info,metrics,prometheus
endpoint:
health:
show-details: always
springdoc:
api-docs:
path: /v3/api-docs
swagger-ui:
path: /swagger-ui.html
trueref:
home: ${TRUEREF_HOME:./data}
ingestion:
poll-interval-default: PT1H
tag-cap-default: 100
max-file-size-bytes-default: 1048576
watched-folder: ${trueref.home}/watched
# Max parallel parse jobs (FETCH/CLONE → CHECKOUT → DISCOVER → DIFF → PARSE).
# Parse is I/O + CPU only — no GPU. 4 is safe on this machine (Ryzen 9 3900X, 62 GB RAM);
# increase for repos with small files, decrease if git I/O saturates disk.
max-parse-jobs: 4
# Max parsed batches buffered between parse workers and the embed worker.
# When the embed worker is busy, parse workers block here — natural backpressure.
# Total peak in-memory batches = max-parse-jobs + embed-queue-capacity.
embed-queue-capacity: 4
embedding:
model: bge-base-en-v1.5
onnx-providers: cuda,directml,cpu
session-count: 1
batch-size: 32
max-seq-len: 512
# Which CUDA device to bind ONNX sessions to. Passed directly to ORT's CUDA EP
# as the physical device index — ORT uses the CUDA driver/NVML API which can bypass
# CUDA_VISIBLE_DEVICES remapping. The ./trueref script sets this to $TRUEREF_GPU (default: 1 = RTX 3060).
gpu-device-id: 0
# Per-session GPU memory cap in bytes. 0 = unbounded. With session-count=1 there
# is no pool contention, so leave this unbounded — capping it risks exhausting the
# BFC arena during model-weight loading before inference starts. The ./trueref script
# defaults to 0 and can be overridden with TRUEREF_MEM_LIMIT.
gpu-mem-limit-bytes: 0
# Override download URLs per (model, file). The built-in defaults (in ModelDownloader)
# cover bge-base-en-v1.5, ms-marco-MiniLM-L6-v2, bge-m3, and bge-reranker-v2-m3.
# Set HF_TOKEN in the environment for higher rate limits or gated models.
# model-sources:
# bge-base-en-v1.5:
# model.onnx:
# - https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/onnx/model.onnx
reranker:
model: ms-marco-MiniLM-L6-v2
top-k: 100
embedding-cache:
# Must match the embedding model's output dimension. Changing this automatically
# wipes the stale .f32 files in the cache directory on next startup.
dimension: 768
search:
rrf-k: 60
final-top-k: 20
mcp:
tokens-default: 5000
tokens-min: 500
tokens-max: 50000
logging:
level:
root: INFO
com.trueref: INFO
org.eclipse.jgit: WARN
org.apache.lucene: WARN

62
trueref-bootstrap/src/main/scripts/trueref Executable file
View File

@@ -0,0 +1,62 @@
#!/usr/bin/env bash
# trueref launcher — wraps the fat JAR with the JVM flags required to silence
# the FFM (foreign linker) restricted-method warning emitted by JNA-based
# tokenizer libraries and to make Lucene's Vector API path readable.
#
# --enable-native-access=ALL-UNNAMED
# Lucene 10 + DJL HuggingFace Tokenizers use the new java.lang.foreign
# Linker API; on Java 21 this requires explicit native-access opt-in.
# --add-modules jdk.incubator.vector
# Lucene 10 ships an incubator-vector codepath that is significantly
# faster for cosine/dot-product math but only loads if the module is
# made readable from the unnamed module.
#
# Usage:
# bin/trueref # default settings
# bin/trueref --server.port=18080 # forward Spring properties
# TRUEREF_JAR=/path/to/trueref.jar bin/trueref
#
# Environment overrides:
# TRUEREF_JAR Path to the fat JAR (default: <script-dir>/../trueref.jar)
# JAVA Path to the java binary (default: ${JAVA_HOME:-}/bin/java or `java` on PATH)
# JAVA_OPTS Extra JVM flags (e.g. -Xmx16g, -XX:+UseZGC)
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
JAR_DEFAULT="${SCRIPT_DIR}/../trueref.jar"
JAR="${TRUEREF_JAR:-$JAR_DEFAULT}"
if [[ ! -f "$JAR" ]]; then
echo "trueref: jar not found at $JAR" >&2
echo "trueref: set TRUEREF_JAR or place trueref.jar next to this script" >&2
exit 1
fi
if [[ -n "${JAVA:-}" ]]; then
:
elif [[ -n "${JAVA_HOME:-}" && -x "${JAVA_HOME}/bin/java" ]]; then
JAVA="${JAVA_HOME}/bin/java"
else
JAVA="$(command -v java || true)"
fi
if [[ -z "${JAVA:-}" || ! -x "${JAVA}" ]]; then
echo "trueref: java not found; set JAVA_HOME or install JDK 21+" >&2
exit 1
fi
# ONNX Runtime CUDA EP needs cuDNN 9 on LD_LIBRARY_PATH. Many distros only ship
# cuDNN via the system package manager or via a Python wheel (nvidia-cudnn-cu12).
# If the user sets TRUEREF_CUDNN_LIB we trust it; otherwise we leave LD_LIBRARY_PATH
# alone and let CUDA fall back to CPU with a logged warning.
if [[ -n "${TRUEREF_CUDNN_LIB:-}" ]]; then
export LD_LIBRARY_PATH="${TRUEREF_CUDNN_LIB}${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
fi
exec "$JAVA" \
--enable-native-access=ALL-UNNAMED \
--add-modules=jdk.incubator.vector \
${JAVA_OPTS:-} \
-jar "$JAR" \
"$@"

21
trueref-domain/pom.xml Normal file
View File

@@ -0,0 +1,21 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.trueref</groupId>
<artifactId>trueref-parent</artifactId>
<version>0.1.0-SNAPSHOT</version>
</parent>
<artifactId>trueref-domain</artifactId>
<name>trueref-domain</name>
<description>Pure domain model + ports. No Spring, no I/O, no third-party libs beyond JSpecify.</description>
<!-- Hexagonal contract: domain has ZERO runtime dependencies beyond JSpecify (annotations only). -->
<dependencies>
<!-- inherits jspecify + test deps from parent -->
</dependencies>
</project>

9
trueref-domain/src/main/java/com/trueref/domain/error/IngestionFailed.java Normal file
View File

@@ -0,0 +1,9 @@
package com.trueref.domain.error;
import org.jspecify.annotations.Nullable;
public final class IngestionFailed extends TrueRefException {
public IngestionFailed(String message, @Nullable Throwable cause) {
super("ingestion_failed", message, cause);
}
}

7
trueref-domain/src/main/java/com/trueref/domain/error/InvalidSearchRequest.java Normal file
View File

@@ -0,0 +1,7 @@
package com.trueref.domain.error;
public final class InvalidSearchRequest extends TrueRefException {
public InvalidSearchRequest(String message) {
super("invalid_search_request", message, null);
}
}

7
trueref-domain/src/main/java/com/trueref/domain/error/RepositoryAlreadyRegistered.java Normal file
View File

@@ -0,0 +1,7 @@
package com.trueref.domain.error;
public final class RepositoryAlreadyRegistered extends TrueRefException {
public RepositoryAlreadyRegistered(String name) {
super("repository_already_registered", "Repository already registered: " + name, null);
}
}

7
trueref-domain/src/main/java/com/trueref/domain/error/RepositoryNotFound.java Normal file
View File

@@ -0,0 +1,7 @@
package com.trueref.domain.error;
public final class RepositoryNotFound extends TrueRefException {
public RepositoryNotFound(String idOrName) {
super("repository_not_found", "Repository not found: " + idOrName, null);
}
}

7
trueref-domain/src/main/java/com/trueref/domain/error/TagNotFound.java Normal file
View File

@@ -0,0 +1,7 @@
package com.trueref.domain.error;
public final class TagNotFound extends TrueRefException {
public TagNotFound(String repo, String tag) {
super("tag_not_found", "Tag not found in repository: " + repo + "@" + tag, null);
}
}

25
trueref-domain/src/main/java/com/trueref/domain/error/TrueRefException.java Normal file
View File

@@ -0,0 +1,25 @@
package com.trueref.domain.error;
import org.jspecify.annotations.Nullable;
/** Root of all domain errors. Carries a stable string {@link #code()} for client localization. */
public abstract sealed class TrueRefException extends RuntimeException
permits RepositoryAlreadyRegistered,
RepositoryNotFound,
VersionNotFound,
VersionNotIndexed,
TagNotFound,
IngestionFailed,
InvalidSearchRequest {
private final String code;
protected TrueRefException(String code, String message, @Nullable Throwable cause) {
super(message, cause);
this.code = code;
}
public String code() {
return code;
}
}

7
trueref-domain/src/main/java/com/trueref/domain/error/VersionNotFound.java Normal file
View File

@@ -0,0 +1,7 @@
package com.trueref.domain.error;
public final class VersionNotFound extends TrueRefException {
public VersionNotFound(String repo, String version) {
super("version_not_found", "Version not found: " + repo + "@" + version, null);
}
}

8
trueref-domain/src/main/java/com/trueref/domain/error/VersionNotIndexed.java Normal file
View File

@@ -0,0 +1,8 @@
package com.trueref.domain.error;
/** Thrown when a search request targets a known version that has not been indexed yet. */
public final class VersionNotIndexed extends TrueRefException {
public VersionNotIndexed(String repo, String version) {
super("version_not_indexed", "Version not yet indexed: " + repo + "@" + version, null);
}
}

5
trueref-domain/src/main/java/com/trueref/domain/error/package-info.java Normal file
View File

@@ -0,0 +1,5 @@
/**
* Sealed exception hierarchy for the domain. Adapters translate these to HTTP / JSON-RPC responses.
*/
@org.jspecify.annotations.NullMarked
package com.trueref.domain.error;

18
trueref-domain/src/main/java/com/trueref/domain/model/Chunk.java Normal file
View File

@@ -0,0 +1,18 @@
package com.trueref.domain.model;
import org.jspecify.annotations.Nullable;
/**
* A globally-deduplicated piece of content (function, class, markdown section, sliding-window
* fallback). Identified by {@link #contentHash()}: two chunks with the same hash are the same
* chunk, regardless of which repo/tag/file they originated from.
*
* @param symbol AST symbol name when applicable (e.g. function or class), null for prose chunks
*/
public record Chunk(
ChunkId id,
String contentHash,
String content,
String language,
@Nullable String symbol,
int tokenCount) {}

16
trueref-domain/src/main/java/com/trueref/domain/model/ChunkId.java Normal file
View File

@@ -0,0 +1,16 @@
package com.trueref.domain.model;
public record ChunkId(java.util.UUID value) {
public static ChunkId random() {
return new ChunkId(java.util.UUID.randomUUID());
}
public static ChunkId of(String s) {
return new ChunkId(java.util.UUID.fromString(s));
}
@Override
public String toString() {
return value.toString();
}
}

8
trueref-domain/src/main/java/com/trueref/domain/model/ChunkVersion.java Normal file
View File

@@ -0,0 +1,8 @@
package com.trueref.domain.model;
/**
* Many-to-many edge between a {@link Chunk} and a {@link Version}. Carries the location of the
* chunk inside the version's source tree.
*/
public record ChunkVersion(
ChunkId chunkId, VersionId versionId, String filePath, int startLine, int endLine) {}

19
trueref-domain/src/main/java/com/trueref/domain/model/Embedding.java Normal file
View File

@@ -0,0 +1,19 @@
package com.trueref.domain.model;
/** Vector representation of a {@link Chunk}. Dense float vector; sparse channel deferred. */
public record Embedding(ChunkId chunkId, float[] vector) {
public Embedding {
// Defensive copy to make the record effectively immutable.
vector = vector.clone();
}
@Override
public float[] vector() {
return vector.clone();
}
public int dimension() {
return vector.length;
}
}

25
trueref-domain/src/main/java/com/trueref/domain/model/IngestionJob.java Normal file
View File

@@ -0,0 +1,25 @@
package com.trueref.domain.model;
import java.time.Instant;
import java.util.List;
import org.jspecify.annotations.Nullable;
/**
* A unit of orchestrated work. One job has many {@link JobStage stages} executed in sequence.
*
* @param versionId null for repo-level jobs (e.g. {@link JobType#DISCOVER_TAGS})
*/
public record IngestionJob(
JobId id,
RepositoryId repoId,
@Nullable VersionId versionId,
JobType type,
JobStatus status,
@Nullable Instant startedAt,
@Nullable Instant finishedAt,
List<JobStage> stages) {
public IngestionJob {
stages = List.copyOf(stages);
}
}

16
trueref-domain/src/main/java/com/trueref/domain/model/JobId.java Normal file
View File

@@ -0,0 +1,16 @@
package com.trueref.domain.model;
public record JobId(java.util.UUID value) {
public static JobId random() {
return new JobId(java.util.UUID.randomUUID());
}
public static JobId of(String s) {
return new JobId(java.util.UUID.fromString(s));
}
@Override
public String toString() {
return value.toString();
}
}

20
trueref-domain/src/main/java/com/trueref/domain/model/JobLogEvent.java Normal file
View File

@@ -0,0 +1,20 @@
package com.trueref.domain.model;
import java.time.Instant;
import org.jspecify.annotations.Nullable;
/** A single emitted observability event for an ingestion job. Streamed via SSE to the UI. */
public record JobLogEvent(
JobId jobId,
Instant ts,
Level level,
JobStage.@Nullable StageName stage,
String message) {
public enum Level {
DEBUG,
INFO,
WARN,
ERROR
}
}

37
trueref-domain/src/main/java/com/trueref/domain/model/JobStage.java Normal file
View File

@@ -0,0 +1,37 @@
package com.trueref.domain.model;
import java.time.Instant;
import org.jspecify.annotations.Nullable;
public record JobStage(
JobId jobId,
StageName name,
StageStatus status,
@Nullable Instant startedAt,
@Nullable Instant finishedAt,
long itemsProcessed,
long itemsTotal,
long bytesProcessed,
@Nullable String errorMessage) {
public enum StageName {
CLONE,
FETCH,
CHECKOUT,
DISCOVER_FILES,
DIFF_FILES,
PARSE,
CHUNK,
EMBED,
INDEX,
COMMIT
}
public enum StageStatus {
PENDING,
RUNNING,
SUCCEEDED,
FAILED,
SKIPPED
}
}

9
trueref-domain/src/main/java/com/trueref/domain/model/JobStatus.java Normal file
View File

@@ -0,0 +1,9 @@
package com.trueref.domain.model;
public enum JobStatus {
QUEUED,
RUNNING,
SUCCEEDED,
FAILED,
CANCELLED
}

8
trueref-domain/src/main/java/com/trueref/domain/model/JobType.java Normal file
View File

@@ -0,0 +1,8 @@
package com.trueref.domain.model;
public enum JobType {
DISCOVER_TAGS,
INDEX_VERSION,
REFRESH,
COMPACT
}

37
trueref-domain/src/main/java/com/trueref/domain/model/Repository.java Normal file
View File

@@ -0,0 +1,37 @@
package com.trueref.domain.model;
import java.time.Duration;
import java.time.Instant;
import java.util.List;
import org.jspecify.annotations.Nullable;
/**
* A registered git repository (local or remote-cloned). The {@code localPath} is always present;
* for remote repositories it points to our managed clone directory and {@code managedClone} is true.
*
* @param remoteUrl git URL when {@code managedClone} is true; null otherwise
* @param ignoreGlobs per-repo globs ANDed with .gitignore + built-in defaults
* @param maxFileSizeBytes files larger than this are skipped during ingestion
* @param pollInterval scheduled fetch interval; {@link Duration#ZERO} disables polling
* @param tagCap max most-recent tags to auto-index; UI/MCP can index more on demand
* @param versionMappingRules ordered patterns mapping a client version (e.g. {@code "1.2.3"}) to a tag
*/
public record Repository(
RepositoryId id,
String name,
@Nullable String remoteUrl,
String localPath,
boolean managedClone,
List<String> ignoreGlobs,
long maxFileSizeBytes,
Duration pollInterval,
int tagCap,
List<TagPattern> versionMappingRules,
Instant createdAt,
Instant updatedAt) {
public Repository {
ignoreGlobs = List.copyOf(ignoreGlobs);
versionMappingRules = List.copyOf(versionMappingRules);
}
}

17
trueref-domain/src/main/java/com/trueref/domain/model/RepositoryId.java Normal file
View File

@@ -0,0 +1,17 @@
package com.trueref.domain.model;
/** Type-safe identifier for a registered repository. */
public record RepositoryId(java.util.UUID value) {
public static RepositoryId random() {
return new RepositoryId(java.util.UUID.randomUUID());
}
public static RepositoryId of(String s) {
return new RepositoryId(java.util.UUID.fromString(s));
}
@Override
public String toString() {
return value.toString();
}
}

18
trueref-domain/src/main/java/com/trueref/domain/model/SearchHit.java Normal file
View File

@@ -0,0 +1,18 @@
package com.trueref.domain.model;
import org.jspecify.annotations.Nullable;
/** A single ranked snippet returned from a search. */
public record SearchHit(
ChunkId chunkId,
RepositoryId repoId,
VersionId versionId,
String repoName,
String tag,
String filePath,
int startLine,
int endLine,
String language,
@Nullable String symbol,
String content,
double score) {}

16
trueref-domain/src/main/java/com/trueref/domain/model/SearchScope.java Normal file
View File

@@ -0,0 +1,16 @@
package com.trueref.domain.model;
import java.util.List;
/**
* Defines the (repo, version) scope of a search request. Multiple scopes can be ORed together so a
* single query may span "spring-boot v3.5.4" and "spring-boot v3.4.0", for example.
*/
public record SearchScope(List<RepoVersionRef> refs) {
public SearchScope {
refs = List.copyOf(refs);
}
public record RepoVersionRef(RepositoryId repoId, VersionId versionId) {}
}

24
trueref-domain/src/main/java/com/trueref/domain/model/TagPattern.java Normal file
View File

@@ -0,0 +1,24 @@
package com.trueref.domain.model;
/**
* Strategy for mapping a client-supplied version string to a git tag in a repository. Patterns are
* tried in order; the first match wins. Built-in patterns: EXACT, V_PREFIX, RELEASE_PREFIX,
* SEMVER_FUZZY. CUSTOM allows a user-supplied template like {@code "release-{semver}"}.
*/
public sealed interface TagPattern {
/** {@code "1.2.3"} → tag {@code "1.2.3"}. */
record Exact() implements TagPattern {}
/** {@code "1.2.3"} → tag {@code "v1.2.3"}. */
record VPrefix() implements TagPattern {}
/** {@code "1.2.3"} → tag {@code "release-1.2.3"}. */
record ReleasePrefix() implements TagPattern {}
/** Any tag whose semver is closest to the requested version. */
record SemverFuzzy() implements TagPattern {}
/** Custom template containing {@code {version}} or {@code {semver}} placeholders. */
record Custom(String template) implements TagPattern {}
}

15
trueref-domain/src/main/java/com/trueref/domain/model/Version.java Normal file
View File

@@ -0,0 +1,15 @@
package com.trueref.domain.model;
import java.time.Instant;
import org.jspecify.annotations.Nullable;
/** A specific git tag (or branch) of a {@link Repository} that may be indexed independently. */
public record Version(
VersionId id,
RepositoryId repoId,
String tag,
String commitSha,
VersionStatus status,
@Nullable Instant indexedAt,
int chunkCount,
@Nullable String errorMessage) {}

16
trueref-domain/src/main/java/com/trueref/domain/model/VersionId.java Normal file
View File

@@ -0,0 +1,16 @@
package com.trueref.domain.model;
public record VersionId(java.util.UUID value) {
public static VersionId random() {
return new VersionId(java.util.UUID.randomUUID());
}
public static VersionId of(String s) {
return new VersionId(java.util.UUID.fromString(s));
}
@Override
public String toString() {
return value.toString();
}
}

14
trueref-domain/src/main/java/com/trueref/domain/model/VersionStatus.java Normal file
View File

@@ -0,0 +1,14 @@
package com.trueref.domain.model;
public enum VersionStatus {
/** Tag known but not yet indexed. */
DISCOVERED,
/** Indexing job currently running. */
INDEXING,
/** Successfully indexed and queryable. */
INDEXED,
/** Last indexing attempt failed; see {@link Version#errorMessage()}. */
FAILED,
/** Tag no longer exists upstream; chunks reclaimable by compaction. */
INACTIVE
}

7
trueref-domain/src/main/java/com/trueref/domain/model/package-info.java Normal file
View File

@@ -0,0 +1,7 @@
/**
* Pure domain model for trueref. Contains records and enums describing repositories, versions,
* chunks, ingestion jobs, and search results. <strong>Must remain free of any I/O, Spring,
* Jackson, or other framework concerns.</strong> JSpecify nullability annotations are allowed.
*/
@org.jspecify.annotations.NullMarked
package com.trueref.domain.model;

12
trueref-domain/src/main/java/com/trueref/domain/port/in/DiscoverVersions.java Normal file
View File

@@ -0,0 +1,12 @@
package com.trueref.domain.port.in;
import com.trueref.domain.model.RepositoryId;
import com.trueref.domain.model.Version;
import java.util.List;
/** Use case: discover/refresh git tags of a repository. */
public interface DiscoverVersions {
/** Performs git fetch (if managed) + tag enumeration. Returns the now-known versions. */
List<Version> discover(RepositoryId repoId);
}

12
trueref-domain/src/main/java/com/trueref/domain/port/in/IndexVersion.java Normal file
View File

@@ -0,0 +1,12 @@
package com.trueref.domain.port.in;
import com.trueref.domain.model.JobId;
import com.trueref.domain.model.RepositoryId;
import com.trueref.domain.model.VersionId;
/** Use case: schedule indexing of a specific (repo, tag/version). */
public interface IndexVersion {
/** Enqueues an INDEX_VERSION job. Returns immediately with the job id. */
JobId enqueue(RepositoryId repoId, VersionId versionId, boolean force);
}

27
trueref-domain/src/main/java/com/trueref/domain/port/in/ObserveJobs.java Normal file
View File

@@ -0,0 +1,27 @@
package com.trueref.domain.port.in;
import com.trueref.domain.model.IngestionJob;
import com.trueref.domain.model.JobId;
import com.trueref.domain.model.JobLogEvent;
import com.trueref.domain.model.JobStatus;
import com.trueref.domain.model.RepositoryId;
import com.trueref.domain.model.VersionId;
import java.util.List;
import java.util.Optional;
import java.util.function.Consumer;
import org.jspecify.annotations.Nullable;
/** Use case: read jobs and subscribe to job/log streams (for SSE in the UI). */
public interface ObserveJobs {
Optional<IngestionJob> findJob(JobId id);
List<IngestionJob> listJobs(
@Nullable RepositoryId repoId, @Nullable VersionId versionId, @Nullable JobStatus status, int limit);
/** Subscribes to live status updates of all jobs. Returns an unsubscribe handle. */
AutoCloseable subscribeJobs(Consumer<IngestionJob> listener);
/** Subscribes to log events of a single job. Returns an unsubscribe handle. */
AutoCloseable subscribeLogs(JobId jobId, Consumer<JobLogEvent> listener);
}

17
trueref-domain/src/main/java/com/trueref/domain/port/in/QueryCatalog.java Normal file
View File

@@ -0,0 +1,17 @@
package com.trueref.domain.port.in;
import com.trueref.domain.model.Repository;
import com.trueref.domain.model.RepositoryId;
import com.trueref.domain.model.Version;
import java.util.List;
import java.util.Optional;
/** Use case: read-only access to repositories and their versions. */
public interface QueryCatalog {
List<Repository> listRepositories();
Optional<Repository> findRepository(RepositoryId id);
List<Version> listVersions(RepositoryId repoId);
}

32
trueref-domain/src/main/java/com/trueref/domain/port/in/RegisterRepository.java Normal file
View File

@@ -0,0 +1,32 @@
package com.trueref.domain.port.in;
import com.trueref.domain.model.Repository;
import com.trueref.domain.model.RepositoryId;
import com.trueref.domain.model.TagPattern;
import java.time.Duration;
import java.util.List;
import org.jspecify.annotations.Nullable;
/** Use case: register a new repository (local path or remote URL). */
public interface RegisterRepository {
Repository register(Command cmd);
record Command(
String name,
@Nullable String remoteUrl,
@Nullable String localPath,
List<String> ignoreGlobs,
@Nullable Long maxFileSizeBytes,
@Nullable Duration pollInterval,
@Nullable Integer tagCap,
List<TagPattern> versionMappingRules) {
public Command {
ignoreGlobs = List.copyOf(ignoreGlobs);
versionMappingRules = List.copyOf(versionMappingRules);
}
}
void unregister(RepositoryId id);
}

40
trueref-domain/src/main/java/com/trueref/domain/port/in/ResolveLibraryId.java Normal file
View File

@@ -0,0 +1,40 @@
package com.trueref.domain.port.in;
import com.trueref.domain.model.RepositoryId;
import com.trueref.domain.model.VersionId;
import com.trueref.domain.model.VersionStatus;
import java.util.List;
import org.jspecify.annotations.Nullable;
/**
* Use case: turn a fuzzy library name (and optional version) into one or more concrete (repo,
* version) handles, ranked by relevance. Mirrors Context7's {@code resolve-library-id}.
*/
public interface ResolveLibraryId {
Result resolve(Query query);
record Query(String libraryName, @Nullable String query, @Nullable String version) {}
record Result(List<Match> matches) {
public Result {
matches = List.copyOf(matches);
}
}
record Match(
RepositoryId repoId,
String libraryId, // "/owner/repo[/version]"
String name,
@Nullable String description,
int snippetCount,
List<VersionRef> availableVersions,
double score) {
public Match {
availableVersions = List.copyOf(availableVersions);
}
}
record VersionRef(VersionId versionId, String tag, VersionStatus status) {}
}

30
trueref-domain/src/main/java/com/trueref/domain/port/in/SearchLibraryDocs.java Normal file
View File

@@ -0,0 +1,30 @@
package com.trueref.domain.port.in;
import com.trueref.domain.model.SearchHit;
import com.trueref.domain.model.SearchScope;
import java.util.List;
import org.jspecify.annotations.Nullable;
/** Use case: hybrid (BM25 + dense) search with rerank, scoped to specific (repo, version) pairs. */
public interface SearchLibraryDocs {
Result search(Query query);
record Query(
String text,
@Nullable String topic,
SearchScope scope,
int tokensBudget,
int maxHits) {}
/**
* @param hits ranked snippets, packed to fit within {@link Query#tokensBudget()}
* @param totalTokensReturned cumulative token count of returned snippets
*/
record Result(List<SearchHit> hits, int totalTokensReturned) {
public Result {
hits = List.copyOf(hits);
}
}
}

6
trueref-domain/src/main/java/com/trueref/domain/port/in/package-info.java Normal file
View File

@@ -0,0 +1,6 @@
/**
* Driving ports — interfaces implemented by the application layer and called by adapters
* (REST controllers, MCP tool handlers, scheduled tasks, etc.).
*/
@org.jspecify.annotations.NullMarked
package com.trueref.domain.port.in;

63
trueref-frontend/pom.xml Normal file
View File

@@ -0,0 +1,63 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.trueref</groupId>
<artifactId>trueref-parent</artifactId>
<version>0.1.0-SNAPSHOT</version>
</parent>
<artifactId>trueref-frontend</artifactId>
<name>trueref-frontend</name>
<description>SvelteKit static UI built with frontend-maven-plugin and packaged as a resource jar.</description>
<packaging>jar</packaging>
<build>
<resources>
<!-- Point directly at the SvelteKit build output so resources:resources
(bound to process-resources) finds the files that npm-build already
created in generate-resources. The intermediate copy-frontend-build
step used target/frontend-dist as the resource directory, but that
directory is only populated later in process-resources, causing an
empty JAR on clean builds. -->
<resource>
<directory>web/build</directory>
<targetPath>static</targetPath>
</resource>
</resources>
<plugins>
<plugin>
<groupId>com.github.eirslett</groupId>
<artifactId>frontend-maven-plugin</artifactId>
<configuration>
<workingDirectory>web</workingDirectory>
<installDirectory>${project.build.directory}</installDirectory>
<nodeVersion>${node.version}</nodeVersion>
<npmVersion>${npm.version}</npmVersion>
</configuration>
<executions>
<execution>
<id>install-node-and-npm</id>
<goals><goal>install-node-and-npm</goal></goals>
<phase>generate-resources</phase>
</execution>
<execution>
<id>npm-install</id>
<goals><goal>npm</goal></goals>
<phase>generate-resources</phase>
<configuration><arguments>install</arguments></configuration>
</execution>
<execution>
<id>npm-build</id>
<goals><goal>npm</goal></goals>
<phase>generate-resources</phase>
<configuration><arguments>run build</arguments></configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>

9
trueref-frontend/web/.gitignore vendored Normal file
View File

@@ -0,0 +1,9 @@
node_modules
/build
/.svelte-kit
/package
.env
.env.*
!.env.example
.DS_Store
*.log

1
trueref-frontend/web/.npmrc Normal file
View File

@@ -0,0 +1 @@
engine-strict=false

9
trueref-frontend/web/.prettierrc Normal file
View File

@@ -0,0 +1,9 @@
{
"useTabs": false,
"tabWidth": 2,
"singleQuote": true,
"trailingComma": "none",
"printWidth": 100,
"plugins": ["prettier-plugin-svelte"],
"overrides": [{ "files": "*.svelte", "options": { "parser": "svelte" } }]
}

Some files were not shown because too many files have changed in this diff Show More