feat: Pass B — consolidation & AI anti-pattern analysis (report-only)
Adds the cross-file analysis pass on top of v1's per-node simplification: - F1 consolidation: duplicate enums/labels (ast-grep+LLM cluster), near-duplicate functions (cpd exact + ast-grep skeleton for renamed), untyped repeated DTOs. - F2 reinventing-the-wheel: ast-grep signature index + judge. - F3 lint enrichment: bundled Biome (JS/TS) + ruff (Python), read-only, mapped to ledger (defer to linters, don't reimplement). - F4 metrics: scc + ast-grep-derived; flags god/long functions. Engines bundled as standalone single binaries (manifest + checksums + LFS): ast-grep, cpd, scc, biome, ruff. Qlty REJECTED at the non-intrusiveness gate (requires `qlty init` writing .qlty/ into the target) → contingency cpd+scc. Pass B is report-only with a mandatory model judge + AHA guardrails (validated on software-house: 5 enum clusters incl. a latent bug, 7 near-dup fn clusters, biome 148 diags, scc flags dispatch.ts cx=698). Docs: references/CONSOLIDATION.md, references/ANTIPATTERNS.md, docs/PLAN-v2-antipatterns.md; SKILL.md Pass B section; ledger schema consolidation kind. linux-x64 binaries bundled; other platforms via manifest fetch+verify. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Giancarmine Salucci
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
c14d8bf411
commit
4b80117933
+10
-2
@@ -1,4 +1,12 @@
|
||||
# Engine binaries are large (~50 MB each) and tracked with Git LFS so the repo stays
|
||||
# usable while fresh clones are fully self-contained (offline, no first-run fetch).
|
||||
# Engine binaries are large and tracked with Git LFS so the repo stays usable while
|
||||
# clones stay self-contained (offline, no first-run fetch). See bin/manifest.json.
|
||||
bin/**/ast-grep filter=lfs diff=lfs merge=lfs -text
|
||||
bin/**/ast-grep.exe filter=lfs diff=lfs merge=lfs -text
|
||||
bin/**/cpd filter=lfs diff=lfs merge=lfs -text
|
||||
bin/**/cpd.exe filter=lfs diff=lfs merge=lfs -text
|
||||
bin/**/scc filter=lfs diff=lfs merge=lfs -text
|
||||
bin/**/scc.exe filter=lfs diff=lfs merge=lfs -text
|
||||
bin/**/biome filter=lfs diff=lfs merge=lfs -text
|
||||
bin/**/biome.exe filter=lfs diff=lfs merge=lfs -text
|
||||
bin/**/ruff filter=lfs diff=lfs merge=lfs -text
|
||||
bin/**/ruff.exe filter=lfs diff=lfs merge=lfs -text
|
||||
|
||||
@@ -9,6 +9,21 @@ functions **while preserving behavior** — detecting opportunities mechanically
|
||||
bundled [`ast-grep`](https://ast-grep.github.io/) engine, ranking the biggest wins,
|
||||
and applying fixes one at a time with build/test verification.
|
||||
|
||||
## Two passes
|
||||
|
||||
- **Pass A — structural simplification** (per-node): nesting, magic numbers, booleans,
|
||||
params, dead branches — apply or report, verified per change.
|
||||
- **Pass B — consolidation & anti-pattern analysis** (cross-file, **report-only**):
|
||||
duplicate enums/types, near-duplicate functions, untyped repeated DTOs, reinvented
|
||||
utilities, plus lint/metrics enrichment. Targets the measured AI failure mode
|
||||
(duplication up, reuse down). Every candidate is model-judged with anti-over-abstraction
|
||||
(AHA) guardrails; nothing is auto-merged. See `references/CONSOLIDATION.md` and
|
||||
`references/ANTIPATTERNS.md`.
|
||||
|
||||
All engines are bundled single static binaries (ast-grep, cpd, scc, biome, ruff —
|
||||
`bin/manifest.json`); nothing is assumed about the target environment, and Pass B never
|
||||
writes into the project.
|
||||
|
||||
## Why it stays fast and cheap
|
||||
|
||||
- **Codebase out of context.** `ast-grep` finds + ranks candidates mechanically; the
|
||||
|
||||
@@ -36,8 +36,17 @@ directory, or a whole repository.
|
||||
|
||||
## How it works (overview)
|
||||
|
||||
Two complementary passes:
|
||||
- **Pass A — structural simplification** (Steps 0–7 below): per-node fixes via ast-grep
|
||||
(nesting, magic numbers, booleans, params…), with apply/report modes.
|
||||
- **Pass B — consolidation & anti-pattern analysis** (see "Pass B" section): cross-file,
|
||||
AI-typical problems (duplication, failure to reuse, reinvented utilities) — **report-only**.
|
||||
Trigger when the user mentions duplication / consolidate / reuse / DRY / "duplicated
|
||||
enums or types" / anti-patterns, or asks for a deeper review. Full method:
|
||||
`references/CONSOLIDATION.md`; taxonomy: `references/ANTIPATTERNS.md`.
|
||||
|
||||
```
|
||||
detect (ast-grep, 0 tokens) → assemble ledger (cheap) → infer build/test/lint (once)
|
||||
Pass A: detect (ast-grep, 0 tokens) → assemble ledger (cheap) → infer build/test/lint (once)
|
||||
→ baseline gate → apply top findings (Track A: ast-grep autofix | Track B: span edit)
|
||||
→ verify each change → report
|
||||
```
|
||||
@@ -80,6 +89,12 @@ Detect platform and pick the binary; fall back gracefully.
|
||||
|
||||
Refer to the binary below as `AST_GREP`.
|
||||
|
||||
**Aux engines for Pass B** (optional; same bundle/fetch/verify mechanism): `cpd`
|
||||
(duplication), `scc` (metrics), `biome` (JS/TS lint), `ruff` (Python lint) — see
|
||||
`bin/manifest.json` for per-platform assets + `bin/checksums.txt`. Each is selected the same
|
||||
way (`bin/<os>-<arch>/<tool>`), and absence of any aux tool never blocks a run (the relevant
|
||||
detector just degrades or is skipped with a note).
|
||||
|
||||
## Step 1 — Scope the candidate files
|
||||
|
||||
Never touch files git ignores.
|
||||
@@ -231,6 +246,43 @@ In **report mode** also emit proposed changes as `simplify/patches/*.patch` (e.g
|
||||
|
||||
---
|
||||
|
||||
## Pass B — consolidation & anti-pattern analysis (report-only)
|
||||
|
||||
Run when asked to de-duplicate / consolidate / find reuse / DRY / review anti-patterns, or
|
||||
as a deeper second pass. **Pass B never edits files** — it produces a report a human (or a
|
||||
later explicit apply) acts on. Full method + the mandatory judge contract + AHA guardrails:
|
||||
`references/CONSOLIDATION.md`. Taxonomy of what each anti-pattern maps to:
|
||||
`references/ANTIPATTERNS.md`.
|
||||
|
||||
Scope = the same git-visible, test-excluded set as Step 1. Every detector follows
|
||||
**extract (0 tokens) → cluster → JUDGE (mandatory, cheap model + AHA guardrails) → report**;
|
||||
only report `verdict != keep-separate` with `confidence ≥ 0.6`.
|
||||
|
||||
- **F1 Consolidation core**
|
||||
- *Duplicate enums/labels/unions*: `AST_GREP` extracts enum/union/`as const` defs →
|
||||
member-set cluster → judge (merge vs derive intentional subset vs keep-separate).
|
||||
- *Near-duplicate functions*: `cpd` for exact copy-paste (`cpd <paths> --min-tokens 50
|
||||
--reporters json --output simplify/cpd --silent`) **and** `AST_GREP` skeleton extraction
|
||||
for renamed/structural near-dups → judge (unify / partial / keep-separate).
|
||||
- *Untyped repeated DTOs*: `AST_GREP` extracts object literals → key-set cluster → report
|
||||
only shapes repeated ≥3× with **no** matching interface → judge.
|
||||
- **F2 Reinventing-the-wheel**: `AST_GREP` builds a function-signature index → name/signature
|
||||
similarity → judge whether new code re-implements an existing util (recommend reuse).
|
||||
- **F3 Lint enrichment (defer, don't reimplement)**: run bundled `biome` (JS/TS) and `ruff`
|
||||
(Python) **read-only** (no config needed, nothing written): e.g.
|
||||
`biome lint --reporter=json <paths>`, `ruff check --output-format=json <paths>`. Map each
|
||||
diagnostic into the ledger (`detector: "biome:<rule>"`); a tool-fixable rule → mechanical.
|
||||
Deeper/type-aware/other-language lint and cross-file dead code: optional project toolchain
|
||||
only-if-present (tsc/eslint, knip, vulture, staticcheck, clippy) — never assume it.
|
||||
- **F4 Complexity/metrics**: `scc` (file-level complexity/LOC) + `AST_GREP`-derived
|
||||
per-function nesting/length/params → flag god/long functions; feeds ranking. Report.
|
||||
|
||||
Output: `simplify/consolidation-report.md` (clusters + verdict/why/risk/recommendation; a
|
||||
"kept separate (and why)" section; a "latent drift/bugs" callout) + ledger findings with
|
||||
`kind: "consolidation"`.
|
||||
|
||||
---
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Never modify git-ignored files. Never change public/observable behavior unless the
|
||||
|
||||
@@ -17,6 +17,7 @@
|
||||
"detected_from": { "type": "string", "description": "e.g. package.json | pyproject.toml | go.mod | Cargo.toml | pom.xml | Makefile" },
|
||||
"runtime": { "type": "string", "description": "node | python | go | rust | java | ..." },
|
||||
"include_tests": { "type": "boolean", "description": "false by default — test files are excluded from scope unless the user opted in" },
|
||||
"passes": { "type": "array", "items": { "enum": ["A", "B"] }, "description": "which passes ran: A=structural, B=consolidation/anti-pattern" },
|
||||
"inferred_at": { "type": "string", "description": "commit hash or marker used to detect manifest changes" }
|
||||
}
|
||||
},
|
||||
@@ -36,6 +37,10 @@
|
||||
"required": ["id", "file", "start_line", "end_line", "lang", "detector", "apply_track", "difficulty", "severity", "status"],
|
||||
"properties": {
|
||||
"id": { "type": "string", "description": "stable id, e.g. f0001" },
|
||||
"kind": { "enum": ["structural", "lint", "metric", "consolidation"], "description": "structural=Pass A rule; lint=Biome/ruff; metric=scc/derived; consolidation=Pass B (report-only)" },
|
||||
"verdict": { "enum": ["consolidate", "partial", "keep-separate"], "description": "Pass B only: the judge's verdict" },
|
||||
"members": { "type": "array", "items": { "type": "object", "properties": { "file": {"type": "string"}, "line": {"type": "integer"} } }, "description": "Pass B only: the cluster's member locations" },
|
||||
"recommendation": { "type": "string", "description": "Pass B only: the concrete suggested change (never auto-applied)" },
|
||||
"file": { "type": "string" },
|
||||
"start_line": { "type": "integer", "minimum": 1 },
|
||||
"end_line": { "type": "integer", "minimum": 1 },
|
||||
|
||||
@@ -26,6 +26,10 @@ For each platform, download the release archive, extract the `ast-grep` binary i
|
||||
the matching `bin/<os>-<arch>/` directory, and record its SHA-256 in `checksums.txt`.
|
||||
Pin a single upstream version across all platforms (see `VERSION`).
|
||||
|
||||
> Engines (per platform): ast-grep ~51 MB, biome ~60 MB, ruff ~27 MB, scc ~4 MB, cpd ~4 MB
|
||||
> → **~146 MB/platform** (~875 MB across all 6 platform slots). Versions/URLs in
|
||||
> `manifest.json`; SKILL.md Step 0 selects/fetches per platform and verifies `checksums.txt`.
|
||||
>
|
||||
> Size note: the `ast-grep` binary is ~50 MB per platform (~250 MB total). The binaries
|
||||
> are **committed via Git LFS** (`.gitattributes`), so a clone is fully self-contained and
|
||||
> offline. After cloning, ensure `git lfs` is installed and run `git lfs pull` if the files
|
||||
|
||||
@@ -3,3 +3,7 @@ e08eea52023724e309c7104ca32c41fb9b205d976d3b3e6e681753b2394a3069 linux-x64/ast-
|
||||
bdc4ad8bd72ae05a9c816296ccace501dafb3fb4a98df0a87fe49699b016057d darwin-x64/ast-grep
|
||||
428f35359b297f7562cfc536bdf3104fdc6bc6c50ab38df70994ec15e992df0b darwin-arm64/ast-grep
|
||||
584c59eaf3b50bf436ad43cf36193af1d2fe5e29ddb21b7485c39f131691390f win32-x64/ast-grep.exe
|
||||
f2e2d03a17da85afe6706b84ebe4839f09886abbda6c90ff707ecb949397aece linux-x64/cpd
|
||||
1fde0c8fcd2b31f65d18caa442331eb015171fdb71ba9177df95af0d42e44126 linux-x64/scc
|
||||
e7df298f0551dd90bea4425779369aa3130d9817f4acc4f663ef63c327206a19 linux-x64/biome
|
||||
67ccd1209756fa588e35db5e46969cabed37f0df7e9ecfa4985895490733d3a1 linux-x64/ruff
|
||||
|
||||
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
@@ -0,0 +1,65 @@
|
||||
{
|
||||
"_comment": "Bundled engines for simplify-code. All MIT/Apache, all standalone single binaries, no runtime deps. SKILL.md selects bin/<os-arch>/<binary> per platform; if missing it fetches the asset below and verifies against checksums.txt before exec. Qlty was evaluated and REJECTED: it requires `qlty init` writing .qlty/ into the target repo (intrusive) — replaced by cpd+scc.",
|
||||
"platforms": ["linux-x64", "linux-arm64", "darwin-x64", "darwin-arm64", "win32-x64"],
|
||||
"tools": {
|
||||
"ast-grep": {
|
||||
"version": "0.44.0", "role": "structural detection + definition/function extraction",
|
||||
"repo": "ast-grep/ast-grep", "required": true,
|
||||
"assets": {
|
||||
"linux-x64": {"url": "https://github.com/ast-grep/ast-grep/releases/download/0.44.0/app-x86_64-unknown-linux-gnu.zip", "bin": "ast-grep"},
|
||||
"linux-arm64": {"url": "https://github.com/ast-grep/ast-grep/releases/download/0.44.0/app-aarch64-unknown-linux-gnu.zip", "bin": "ast-grep"},
|
||||
"darwin-x64": {"url": "https://github.com/ast-grep/ast-grep/releases/download/0.44.0/app-x86_64-apple-darwin.zip", "bin": "ast-grep"},
|
||||
"darwin-arm64":{"url": "https://github.com/ast-grep/ast-grep/releases/download/0.44.0/app-aarch64-apple-darwin.zip", "bin": "ast-grep"},
|
||||
"win32-x64": {"url": "https://github.com/ast-grep/ast-grep/releases/download/0.44.0/app-x86_64-pc-windows-msvc.zip", "bin": "ast-grep.exe"}
|
||||
}
|
||||
},
|
||||
"cpd": {
|
||||
"version": "5.0.11", "role": "duplication / copy-paste clones (token-based, multi-language)",
|
||||
"repo": "kucherenko/jscpd", "required": false,
|
||||
"assets": {
|
||||
"linux-x64": {"url": "https://github.com/kucherenko/jscpd/releases/download/v5.0.11/cpd-linux-x64-gnu.tar.gz", "bin": "cpd"},
|
||||
"linux-arm64": {"url": "https://github.com/kucherenko/jscpd/releases/download/v5.0.11/cpd-linux-arm64-gnu.tar.gz", "bin": "cpd"},
|
||||
"darwin-x64": {"url": "https://github.com/kucherenko/jscpd/releases/download/v5.0.11/cpd-darwin-x64.tar.gz", "bin": "cpd"},
|
||||
"darwin-arm64":{"url": "https://github.com/kucherenko/jscpd/releases/download/v5.0.11/cpd-darwin-arm64.tar.gz", "bin": "cpd"},
|
||||
"win32-x64": {"url": "https://github.com/kucherenko/jscpd/releases/download/v5.0.11/cpd-windows-x64-msvc.tar.gz", "bin": "cpd.exe"}
|
||||
}
|
||||
},
|
||||
"scc": {
|
||||
"version": "3.7.0", "role": "metrics: LOC + file-level complexity (multi-language)",
|
||||
"repo": "boyter/scc", "required": false,
|
||||
"assets": {
|
||||
"linux-x64": {"url": "https://github.com/boyter/scc/releases/download/v3.7.0/scc_Linux_x86_64.tar.gz", "bin": "scc"},
|
||||
"linux-arm64": {"url": "https://github.com/boyter/scc/releases/download/v3.7.0/scc_Linux_arm64.tar.gz", "bin": "scc"},
|
||||
"darwin-x64": {"url": "https://github.com/boyter/scc/releases/download/v3.7.0/scc_Darwin_x86_64.tar.gz", "bin": "scc"},
|
||||
"darwin-arm64":{"url": "https://github.com/boyter/scc/releases/download/v3.7.0/scc_Darwin_arm64.tar.gz", "bin": "scc"},
|
||||
"win32-x64": {"url": "https://github.com/boyter/scc/releases/download/v3.7.0/scc_Windows_x86_64.zip", "bin": "scc.exe"}
|
||||
}
|
||||
},
|
||||
"biome": {
|
||||
"version": "2.5.0", "role": "JS/TS lint smells + unused (dead) code (no tsconfig needed)",
|
||||
"repo": "biomejs/biome", "required": false,
|
||||
"assets": {
|
||||
"linux-x64": {"url": "https://github.com/biomejs/biome/releases/download/@biomejs/biome@2.5.0/biome-linux-x64", "bin": "biome"},
|
||||
"linux-arm64": {"url": "https://github.com/biomejs/biome/releases/download/@biomejs/biome@2.5.0/biome-linux-arm64", "bin": "biome"},
|
||||
"darwin-x64": {"url": "https://github.com/biomejs/biome/releases/download/@biomejs/biome@2.5.0/biome-darwin-x64", "bin": "biome"},
|
||||
"darwin-arm64":{"url": "https://github.com/biomejs/biome/releases/download/@biomejs/biome@2.5.0/biome-darwin-arm64", "bin": "biome"},
|
||||
"win32-x64": {"url": "https://github.com/biomejs/biome/releases/download/@biomejs/biome@2.5.0/biome-win32-x64.exe", "bin": "biome.exe"}
|
||||
}
|
||||
},
|
||||
"ruff": {
|
||||
"version": "0.15.18", "role": "Python lint smells + unused (dead) code",
|
||||
"repo": "astral-sh/ruff", "required": false,
|
||||
"assets": {
|
||||
"linux-x64": {"url": "https://github.com/astral-sh/ruff/releases/download/0.15.18/ruff-x86_64-unknown-linux-gnu.tar.gz", "bin": "ruff"},
|
||||
"linux-arm64": {"url": "https://github.com/astral-sh/ruff/releases/download/0.15.18/ruff-aarch64-unknown-linux-gnu.tar.gz", "bin": "ruff"},
|
||||
"darwin-x64": {"url": "https://github.com/astral-sh/ruff/releases/download/0.15.18/ruff-x86_64-apple-darwin.tar.gz", "bin": "ruff"},
|
||||
"darwin-arm64":{"url": "https://github.com/astral-sh/ruff/releases/download/0.15.18/ruff-aarch64-apple-darwin.tar.gz", "bin": "ruff"},
|
||||
"win32-x64": {"url": "https://github.com/astral-sh/ruff/releases/download/0.15.18/ruff-x86_64-pc-windows-msvc.zip", "bin": "ruff.exe"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"notes": {
|
||||
"win-arm64": "ast-grep & cpd have no win-arm64 build; on win-arm64 use the win32-x64 binary under emulation.",
|
||||
"optional-enrichment": "Dead-code beyond JS/TS-Python and type-aware lint require the project's own toolchain (tsc/eslint, knip, vulture, staticcheck, clippy) — used only if already on PATH, never bundled, never assumed."
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,177 @@
|
||||
# Plan v2 — AI anti-pattern detection & consolidation pass
|
||||
|
||||
Status: **IN IMPLEMENTATION**. Extends the shipped v1 skill (per-node structural
|
||||
simplification) with cross-file, AI-anti-pattern analysis.
|
||||
|
||||
> **Milestone-1 gate outcome:** Qlty was **rejected** — `qlty metrics`/`qlty smells`
|
||||
> require `qlty init` writing `.qlty/` into the target repo (no `--config`/no-config flag),
|
||||
> violating the non-intrusive rule. Per the pre-approved contingency the engine is now
|
||||
> **`cpd` (duplication, 4 MB) + `scc` (metrics, 7 MB)**, with renamed near-dup functions via
|
||||
> **ast-grep skeleton + LLM** (proven in measurement). Net win: ~78 MB lighter/platform than
|
||||
> Qlty, and fully non-intrusive. Bundled engines: ast-grep, cpd, scc, biome, ruff.
|
||||
|
||||
## Context
|
||||
|
||||
v1 simplifies *in-the-small* (per-node: nesting, magic numbers, booleans, params) via
|
||||
bundled ast-grep. Two measurement runs on the `software-house` monorepo (321 files) plus
|
||||
a literature sweep showed the highest-value *unaddressed* problems are *in-the-large* and
|
||||
specific to AI-written code: **duplication / failure to reuse** (GitClear: 4× clone growth,
|
||||
refactoring collapsed 25%→<10% since AI). Measured on software-house: 5 duplicate-label
|
||||
clusters (incl. a latent `ThrottleStatus.warming` bug), 7 near-duplicate-function clusters
|
||||
(~75% real), and untyped repeated object shapes. The per-node lint smells from the
|
||||
literature (`any`, `@ts-ignore`, useless-catch) were largely **absent** there — and are
|
||||
**owned by linters** anyway. So v2's unique value is the **consolidation family**, with
|
||||
linters *deferred to*, not reimplemented.
|
||||
|
||||
Validated design principle (twice): mechanical clustering has false positives (a skeleton
|
||||
match ≠ same logic; an inline shape matching an interface is normal usage), so a
|
||||
**cheap-model judge with AHA guardrails is mandatory**, and these findings are
|
||||
**report-only** (consolidation risks coupling; never autofix).
|
||||
|
||||
## Goals / Non-goals
|
||||
|
||||
**Goals**
|
||||
- Detect: (F1) duplicate enums/labels, near-duplicate functions, untyped repeated DTOs;
|
||||
(F2) reinventing-the-wheel; (F3) portable lint smells; (F4) complexity/metrics.
|
||||
- Stay **self-contained & cross-platform**: every engine is a bundled single static binary
|
||||
(like ast-grep). **No assumptions about the target environment** (no required node/eslint/
|
||||
python/jvm).
|
||||
- New semantic findings are **report-only, Track B**, with AHA guardrails.
|
||||
|
||||
**Non-goals**
|
||||
- No reimplementing what linters own (defer). No autofix for consolidation findings. No
|
||||
mutating the target project (read-only analysis; never write config into it). No bundling
|
||||
GPL tools (universal-ctags out). No dead-code / type-aware analysis that needs the
|
||||
project's toolchain — those are optional-if-present only.
|
||||
|
||||
## Hard constraints
|
||||
1. **Self-contained**: engines bundled per-platform via git-lfs (user decision).
|
||||
2. **Cross-platform**: identical CLI on Win/macOS/Linux; per-platform binary selection.
|
||||
3. **Non-intrusive**: analysis must run read-only on a path; never run `qlty init` or write
|
||||
`.qlty/` (or anything) into the target repo; never require network at analysis time.
|
||||
4. **Defer, don't reimplement**: linters via bundled Biome/ruff; deeper checks optional.
|
||||
5. **AHA**: rule-of-three, preserve intentional subsets / layer & bounded-context decoupling.
|
||||
|
||||
## Bundled tool stack (verified, permissive-licensed, standalone binaries)
|
||||
|
||||
| Tool | Role | Platforms (verified assets) | License |
|
||||
|---|---|---|---|
|
||||
| **ast-grep** 0.44 (have) | structural rules + definition/function extraction | linux x64/arm64, darwin x64/arm64, win x64 | MIT |
|
||||
| **cpd** (jscpd) 5.0.11 | duplication / copy-paste (token Type-1/2) | linux x64/arm64, darwin x64/arm64, win x64 | MIT |
|
||||
| **scc** 3.7.0 | metrics: LOC + file-level complexity | all 5 + win-arm64 | MIT |
|
||||
| ~~Qlty 0.630~~ | ~~dup+metrics~~ — **rejected: requires `qlty init` (intrusive)** | — | — |
|
||||
| **Biome** 2.5 | JS/TS lint smells + unused-code (~75% type-aware w/o tsc) | all 5 + win-arm64 | MIT/Apache |
|
||||
| **ruff** 0.15 | Python lint smells | all 5 + win-arm64 | MIT |
|
||||
|
||||
**Optional enrichment, only if already on PATH (never bundled, never assumed):** project
|
||||
`tsc`/eslint (type-aware: no-floating-promises, no-unsafe-*), `knip`/`vulture`/`staticcheck`/
|
||||
`cargo clippy` (dead code), `cargo`/`go` toolchains. Absence never blocks a run.
|
||||
|
||||
> **win-arm64 gap:** ast-grep & cpd have no win-arm64 build → on win-arm64 fall back to the
|
||||
> x64 binary (runs under emulation) or skip that engine with a logged note.
|
||||
|
||||
## Architecture — two pass types
|
||||
|
||||
```
|
||||
PASS A · per-node structural (v1, unchanged)
|
||||
ast-grep scan -c sgconfig.yml → Track A autofix | Track B span edit (apply or report)
|
||||
|
||||
PASS B · analysis passes (v2, NEW — report-only, Track B)
|
||||
extract (ast-grep / Qlty, 0 tokens)
|
||||
→ cluster mechanically (Qlty for clones; LLM for small-N enum/DTO/symbol sets)
|
||||
→ cheap-model JUDGE each cluster (AHA guardrails) ← mandatory
|
||||
→ consolidation report (+ patches/suggestions, never auto-applied)
|
||||
```
|
||||
|
||||
Pass B never edits; it emits a `consolidation-report.md` + ledger entries a human (or a
|
||||
follow-up explicit apply) acts on.
|
||||
|
||||
## Detectors
|
||||
|
||||
### F1 — Consolidation core
|
||||
- **Duplicate enums / labels / unions** — ast-grep extracts `enum`/union-type/`as const`
|
||||
definitions → LLM clusters by member-set overlap (small N) → judge: same concept? merge
|
||||
vs intentional subset/layer. (sh: 5 clusters, 1 latent bug.)
|
||||
- **Near-duplicate functions** — **Qlty duplication** (AST Type-2/3) finds clone clusters →
|
||||
judge: unify / partial (extract shared core) / keep-separate (essential difference). (sh:
|
||||
7 clusters, ~75% real; 1 skeleton-coincidence correctly rejected by the judge.)
|
||||
- **Untyped repeated DTOs** — ast-grep extracts object literals → key-set cluster → flag
|
||||
**only** shapes repeated ≥3× with **no** matching interface (matching an existing
|
||||
interface is normal usage, not a smell) → judge: name a DTO? (sh: ~1 real candidate.)
|
||||
|
||||
### F2 — Reinventing-the-wheel
|
||||
ast-grep extracts every function signature (name + params + return) into an index → for a
|
||||
target span (or whole repo), find existing functions with high name/signature/shape
|
||||
similarity → judge: is the new code re-implementing an existing util? Recommend reuse.
|
||||
No new tool (universal-ctags is GPL → excluded). Hard tier; report-only.
|
||||
|
||||
### F3 — Lint enrichment (defer, don't reimplement)
|
||||
Run bundled **Biome** (JS/TS) and **ruff** (Python) read-only → map their findings into the
|
||||
ledger (`detector: biome:<rule>`), classified mechanical (if tool has a safe fix) or
|
||||
semantic. Other languages + type-aware rules: optional-if-toolchain-present. The skill never
|
||||
ships its own `any`/`non-null`/`console` rules — those are linter-owned.
|
||||
|
||||
### F4 — Complexity / metrics
|
||||
**Qlty metrics** (cyclomatic, cohesion, LOC, per-language) + ast-grep-derived
|
||||
nesting/length/params (v1) → feed ranking and flag god-functions/long-functions
|
||||
(e.g. the 3,111-line `dispatch.ts`). Report + ranking input.
|
||||
|
||||
## Shared judge layer
|
||||
- One judge contract used by F1/F2 (and F4 borderline): input = a cluster/candidate +
|
||||
real code spans; output = `{same_concept, verdict: consolidate|partial|keep-separate,
|
||||
why, risk, recommendation}`.
|
||||
- **AHA guardrails in the prompt**: rule-of-three; do NOT merge across layers/bounded
|
||||
contexts (DB vs API vs UI, adapter vs adapter); prefer keeping intentional subsets;
|
||||
"wrong abstraction > duplication" when callers would diverge.
|
||||
- **Model tiering** (capability-detected, from v1 OPTIMIZATION.md): cheap model clusters &
|
||||
judges; escalate only on low confidence. **Parallel fan-out** where the runtime supports
|
||||
sub-agents (one judge per cluster), else sequential.
|
||||
|
||||
## Ledger / report changes
|
||||
- New finding `kind`: `consolidation` (sub-types: dup-enum, dup-function, dto, reinvented).
|
||||
- New `config.passes` (which passes ran) + `config.include_tests` (v1).
|
||||
- Pass B writes `simplify/consolidation-report.md` (clusters, verdicts, recommendations,
|
||||
and for KEEP-SEPARATE the *reason*, so re-runs don't re-flag).
|
||||
- `ledger.schema.json` extended; Pass B is report-only regardless of apply/report mode.
|
||||
|
||||
## Skill structure changes
|
||||
```
|
||||
bin/<os-arch>/{ast-grep, qlty, biome, ruff(.exe)} # all via LFS; bin/manifest.json (tool→version)
|
||||
bin/checksums.txt # all tools, all platforms
|
||||
rules/... # unchanged (no new lint rules — deferred)
|
||||
references/ANTIPATTERNS.md # the taxonomy + which tool/treatment each
|
||||
references/CONSOLIDATION.md # F1/F2 pipeline, judge contract, AHA guardrails
|
||||
SKILL.md # new "Pass B (analysis)" section + tool detection
|
||||
```
|
||||
|
||||
## Cross-platform mechanics
|
||||
- Platform detect (v1 Step 0) extended to select each tool's binary; per-tool fallback chain
|
||||
(bundled → PATH → skip-with-note). win-arm64 → x64 fallback for ast-grep/Qlty.
|
||||
- `bin/manifest.json` pins versions; `bin/checksums.txt` covers every tool×platform; first
|
||||
use verifies checksum before exec.
|
||||
- LFS already configured (`.gitattributes`); add the new binaries.
|
||||
|
||||
## Build milestones
|
||||
1. **Bundle + verify tools**: fetch Qlty/Biome/ruff per-platform, checksums, manifest, LFS;
|
||||
platform-select + fallback in SKILL.md. **Gate: confirm `qlty metrics`/`qlty smells` run
|
||||
read-only on an arbitrary path with NO `qlty init`, NO `.qlty/` write, NO network.** If
|
||||
not → fall back to jscpd(`cpd`)+scc for F1-functions/F4 (documented contingency).
|
||||
2. **F3 lint enrichment**: Biome/ruff → ledger mapping (cheapest, immediately useful).
|
||||
3. **F4 metrics**: Qlty metrics → ranking + god/long-function flags.
|
||||
4. **F1 consolidation core**: enums (LLM cluster) + near-dup functions (Qlty) + DTOs;
|
||||
shared judge layer + CONSOLIDATION.md; report-only output.
|
||||
5. **F2 reinventing-the-wheel**: signature index + similarity + judge.
|
||||
6. **Docs + validation**: ANTIPATTERNS.md; re-run full suite on software-house.
|
||||
|
||||
## Validation
|
||||
- Re-run on software-house; expect: F1 reproduces the 5 enum + 7 function clusters with the
|
||||
same verdicts (incl. the `ThrottleStatus` bug and the dead-letter/run-monitor KEEP-SEPARATE);
|
||||
F3 surfaces the `as`/non-null counts via Biome; F4 flags `dispatch.ts`. Report-only — no
|
||||
files changed. Confirm on Linux + Windows. `skills-ref validate` still passes.
|
||||
|
||||
## Risks / open verifications
|
||||
- **Qlty non-intrusiveness/network** (milestone-1 gate; contingency = jscpd+scc). Load-bearing.
|
||||
- **Bundle size**: ast-grep≈50 + Qlty≈15 + Biome≈20 + ruff≈30 ≈ **115 MB/platform × ~6 ≈ 600–700 MB** LFS. Accepted (user chose bundle-all); mitigate with musl/slim variants where available.
|
||||
- **Judge false-positives** → keep report-only; require ≥0.6 confidence; show the reason.
|
||||
- **win-arm64 gap** for ast-grep/Qlty → x64 fallback.
|
||||
- **Tool version drift** → pin in `bin/manifest.json`; checksum-gate.
|
||||
@@ -0,0 +1,42 @@
|
||||
# AI-generated code anti-patterns — taxonomy & treatment
|
||||
|
||||
Consolidated from empirical studies (GitClear: ~4× clone growth & refactoring collapse
|
||||
since AI; arXiv code-smell studies; Sonar) + practitioner taxonomies, mapped to how this
|
||||
skill treats each. **Principle: the skill does NOT reimplement what linters own — it defers
|
||||
to bundled Biome/ruff (and optional project toolchains) for per-node lint smells, and spends
|
||||
its unique effort on the cross-file consolidation family that linters can't see.**
|
||||
|
||||
Treatments: **HAVE** (Pass A rule, v1) · **BUILD** (Pass B, this plan) · **LINT** (bundled
|
||||
Biome/ruff) · **ENRICH** (optional project tool, only-if-present) · **REPORT** (surfaced,
|
||||
not auto-fixed) · **SKIP** (out of scope).
|
||||
|
||||
| Anti-pattern | Detect | Treatment |
|
||||
|---|---|---|
|
||||
| Redundant booleans, `indexOf`, `==null` | AST | HAVE (Pass A) |
|
||||
| Deep nesting, long params, magic numbers | AST/metric | HAVE (Pass A) |
|
||||
| **Duplicate enums/labels/constants for one concept** | extract+cluster+judge | **BUILD** F1 |
|
||||
| **Near-duplicate functions** (Type-2/3, parametrizable) | cpd + ast-grep skeleton + judge | **BUILD** F1 |
|
||||
| **Untyped repeated object shapes** | extract+cluster+judge | **BUILD** F1 (conservative) |
|
||||
| **Reinventing existing utilities** (no reuse) | signature index + judge | **BUILD** F2 |
|
||||
| God/long functions, high cyclomatic | scc + ast-grep metrics | **BUILD** F4 (REPORT) |
|
||||
| Explicit `any`, unsafe casts | linter | LINT (Biome) / ENRICH (tsc) |
|
||||
| Non-null `!`, useless catch, `no-console` | linter | LINT (Biome) |
|
||||
| Redundant `async`/`await`, extra boolean cast | linter | LINT (Biome/ruff) |
|
||||
| Unused vars/imports/exports (dead code) | linter | LINT (Biome/ruff); ENRICH (knip/vulture/staticcheck) for cross-file |
|
||||
| Type-aware (floating promises, no-unsafe-*) | needs types | ENRICH (project tsc/eslint only) |
|
||||
| Over-commenting / comment restating code | metric (ratio) | REPORT (optional) |
|
||||
| Over-defensive null checks, single-use wrappers | semantic | REPORT (judge) |
|
||||
| Speculative/premature abstraction | semantic | REPORT (rare; AHA-sensitive) |
|
||||
| Architectural drift, circular deps | graph | SKIP v1 (future: madge-style) |
|
||||
| Security vulns, missing edge cases, hallucinated APIs | semantic/security | SKIP (use a security/SAST tool; out of scope for a simplifier) |
|
||||
|
||||
## Why "defer, don't reimplement"
|
||||
On a disciplined real codebase (software-house) the per-node lint smells were largely
|
||||
absent (0 `any` via strict TS, 0 `@ts-ignore`, 0 useless-catch) — and where present, Biome
|
||||
finds them in milliseconds with no config. Re-encoding those as ast-grep rules would
|
||||
duplicate mature linters and drift. The durable, unique value is the **consolidation
|
||||
family** (the rows marked BUILD), which no linter does and which directly counters the
|
||||
measured AI failure mode (duplication up, reuse down).
|
||||
|
||||
See `CONSOLIDATION.md` for the Pass B pipeline and judge contract, and `OPTIMIZATION.md` for
|
||||
model tiering / fan-out. Bundled engines and versions: `../bin/manifest.json`.
|
||||
@@ -0,0 +1,99 @@
|
||||
# Pass B — consolidation & anti-pattern analysis
|
||||
|
||||
Pass B finds *cross-file, AI-typical* problems that per-node rules (Pass A) can't:
|
||||
duplication, failure to reuse, reinvented utilities. It is **report-only** (these changes
|
||||
risk coupling — never autofix) and every candidate is **judged by a model with AHA
|
||||
guardrails** before it's reported, because mechanical similarity has false positives.
|
||||
|
||||
Pipeline (each detector): **extract (0 tokens) → cluster (tool or cheap model) → JUDGE
|
||||
(mandatory) → consolidation report**. Tools are the bundled, non-intrusive binaries
|
||||
(`bin/manifest.json`); nothing is written into the target project.
|
||||
|
||||
---
|
||||
|
||||
## The judge contract (shared by all Pass B detectors)
|
||||
|
||||
For each candidate cluster, give the model the real code spans and ask for exactly:
|
||||
|
||||
```json
|
||||
{
|
||||
"same_concept": true,
|
||||
"verdict": "consolidate | partial | keep-separate",
|
||||
"confidence": 0.0,
|
||||
"why": "cite concrete evidence from the code",
|
||||
"risk": "what coupling/regression consolidating could cause",
|
||||
"recommendation": "the concrete change, or why to leave it"
|
||||
}
|
||||
```
|
||||
|
||||
Only report findings with `verdict != keep-separate` AND `confidence >= 0.6`. For
|
||||
`keep-separate`, record the reason in the report so re-runs don't re-flag it.
|
||||
|
||||
### AHA guardrails (put these in the judge prompt — they prevent harmful merges)
|
||||
- **Rule of three**: 2 occurrences usually isn't worth abstracting; prefer ≥3.
|
||||
- **"The wrong abstraction is worse than duplication"** (Sandi Metz). If unifying would
|
||||
need extra params/conditionals to satisfy diverging callers → `keep-separate`.
|
||||
- **Respect boundaries**: do NOT merge across layers / bounded contexts — e.g. a DB enum vs
|
||||
an API/wire enum vs a UI label set; one adapter vs another adapter; per-service copies in
|
||||
a microservice split. These look identical but are decoupled on purpose.
|
||||
- **Preserve intentional subsets**: a smaller set that's a *named category* (e.g.
|
||||
`COORDINATOR_ROLES` ⊂ `AgentRole`) is not duplication — recommend deriving it from the
|
||||
canonical source, not merging it away.
|
||||
- **Drift is a signal, not always a bug**: when near-identical definitions have *diverged*
|
||||
(one has an extra member), flag the divergence — it may be a latent bug (a value produced
|
||||
but unrepresentable elsewhere).
|
||||
|
||||
Model tiering (see OPTIMIZATION.md): cluster + judge on a cheap model; escalate to a
|
||||
stronger model only on low confidence. Fan out one judge per cluster where the runtime has
|
||||
sub-agents; else judge sequentially.
|
||||
|
||||
---
|
||||
|
||||
## F1 — Consolidation core
|
||||
|
||||
### Duplicate enums / labels / unions
|
||||
1. Extract with ast-grep: `enum_declaration`, `type_alias_declaration` (keep string-literal
|
||||
unions), `as const` arrays/objects. Record each definition's **member set** + location.
|
||||
2. Cluster (cheap model, small N): group by member-set overlap — exact, subset/superset,
|
||||
high Jaccard.
|
||||
3. Judge each cluster. Typical outcomes: identical union+const in the same package →
|
||||
consolidate (idiomatic TS: one `as const` array as source of truth, derive the union via
|
||||
`typeof arr[number]`); same-named types in different packages that drifted → consolidate +
|
||||
reconcile; intentional subset → derive, don't merge.
|
||||
|
||||
### Near-duplicate functions
|
||||
Two complementary signals:
|
||||
- **Exact / copy-paste** → `cpd` (bundled): `cpd <paths> --min-tokens 50 --reporters json
|
||||
--output <tmp> --silent`. Token-based (Type-1/2). 0 model tokens.
|
||||
- **Renamed / structural ("almost the same")** → ast-grep extracts function nodes; normalize
|
||||
each to a **skeleton** (strip comments; replace identifiers→`V`, strings/numbers→`L`;
|
||||
drop whitespace); group by identical skeleton (ignore trivial: skeleton < ~120 chars).
|
||||
This catches Type-2/3 that `cpd` misses (proven on software-house: 7 clusters).
|
||||
Then judge each cluster: `unify` (extract shared helper / parametrize the literal that
|
||||
differs) | `partial` (share a stateless core, keep thin wrappers) | `keep-separate` (the
|
||||
"not quite" is essential — different data source, retry strategy, adapter, etc.).
|
||||
|
||||
### Untyped repeated object shapes (→ DTO)
|
||||
1. ast-grep extracts object literals; record each one's **top-level key set**.
|
||||
2. Keep only shapes with ≥3 keys repeated **≥3×** that do **NOT** match an existing
|
||||
`interface`/`type` member set. (An inline object matching an existing interface is normal
|
||||
usage — TS structurally type-checks it — **not** a smell. Do not report those.)
|
||||
3. Judge: is this a real DTO worth naming, or incidental? Recommend an interface + reuse.
|
||||
|
||||
## F2 — Reinventing-the-wheel
|
||||
1. ast-grep builds a lightweight signature index across the repo: `{name, params, returns,
|
||||
file, line}` for every function/method (no extra tool; universal-ctags is GPL → excluded).
|
||||
2. For a target (a changed span, or scanning the repo): find existing functions with high
|
||||
name/parameter/return similarity to the candidate.
|
||||
3. Judge: does the new code re-implement an existing utility? If yes, recommend importing the
|
||||
existing one. Guardrail: confirm the existing function's contract actually fits (don't
|
||||
force reuse across incompatible interfaces/perf constraints).
|
||||
|
||||
---
|
||||
|
||||
## Output
|
||||
Write `simplify/consolidation-report.md`: clusters grouped by detector, each with the
|
||||
verdict, why, risk, and recommendation; a separate "kept separate (and why)" section; and a
|
||||
"latent drift/bugs" callout. Add a ledger finding per reported cluster with
|
||||
`kind: "consolidation"` (see `assets/ledger.schema.json`). Pass B never edits files,
|
||||
regardless of the run mode.
|
||||
Reference in New Issue
Block a user