# KB Evaluation Plan ## Goal Evaluate whether building and maintaining a `kb` knowledge base is materially useful for real development work, and whether a split workflow works better: - Agent A builds the product/codebase. - Agent B maintains and refreshes the knowledge base. The core hypothesis is that this produces better systems faster, with lower token cost, better requirement capture, and better recall of project knowledge than relying on the primary coding agent's transient context alone. ## Primary Question After building a KB from scratch for the evaluation target repository, can `kb` answer important questions about the project accurately and usefully enough to justify the extra maintenance work? ## Secondary Questions 1. Does `kb init` produce a usable knowledge base from the current repo without manual surgery? 2. Does the resulting KB support both retrieval-style questions (`kb query`) and synthesis-style questions (`kb chat`) across multiple topic areas? 3. Is the resulting graph store populated enough to plausibly improve retrieval and follow-up questioning? 4. What is the cost of producing this KB in time, tokens, and operator effort? 5. In a later comparison run, does a dedicated KB-maintenance agent improve outcomes versus a single-agent baseline? ## Evaluation Target **Canonical external benchmark:** the [raylib C library](https://github.com/raysan5/raylib) — use suite `raylib` (its `repo_url` is defined in `eval/suites/raylib.yaml`; override with `--repo` only when needed). Reasons: mature, well-documented, not kb itself (avoids evaluator familiarity bias), rich graph structure, stable upstream. **Kb self-check:** suite `kb` (its `repo_url` is defined in `eval/suites/kb.yaml`; override with `--repo` only when needed). That is a product smoke test, not the primary raylib benchmark. For day-to-day kb architecture work on your checkout, use `--base dogfood` (separate from disposable eval bases). ### Base naming convention | Base | Purpose | Lifetime | |------|---------|----------| | `-YYYY-MM-DD-HHmm` | Default disposable base from `eval-run.mjs`: **same string** as `~/.kb/evaluations//` (override with `--base`) | Ephemeral | | `raylib` | Persistent agent comparison base — accumulates across tasks, never wiped | Permanent | | `dogfood` | kb's own architectural knowledge | Permanent | The `raylib` base is the KB that a KB-backed agent would actually use during real development on a long-lived raylib tree. Do not reuse ephemeral eval run names for it — the compounding hypothesis requires the same base to persist across multiple task sessions. ### Published docs location Eval runs do **not** publish Jekyll output. We only capture init/query evidence artifacts. ## Automated harvest (`scripts/eval-run.mjs`) One runner drives all disposable-base harvests. **No local target directory:** repo URL resolves from suite YAML `repo_url`, with optional `--repo` override. Each run does a **fresh snapshot clone** under `~/.kb/evaluations//repo/`; scratch JSON and the default artifact live under `~/.kb/evaluations//`. **Default `--base`** for `init` / `all` equals **``** (e.g. `raylib-2026-04-27-1303` = repo leaf + date + `HHmm`; same-minute collision adds `-2`, `-3`, …). From the kb repo root (after `pnpm run build`): | npm script | Maps to | |------------|---------| | `npm run eval:init -- --suite raylib …` | Clone suite `repo_url`, init, queries | | `npm run eval:init -- --suite kb …` | Clone suite `repo_url`, init, queries | | `npm run eval:init -- --suite generic --repo …` | Generic suite requires explicit repo override, then clone/init/queries | | `npm run eval:all …` | Same as `eval:init` (alias) | | `npm run eval:query -- --suite raylib --base …` | No init: docs + graph + logs + 8× `kb query` (still clones repo for cwd) | | `npm run eval:gen-doc` | `kb docs generate` smoke (introduction + howto) on `--base` (default `dogfood`); artifact under `~/.kb/evaluations//` | **Modes** - `init` (or legacy `all`) — Fresh clone → `kb init --non-interactive`, then metrics + eight queries. - `query` — Fresh clone → same capture minus init; requires `--base` for an already-populated KB session. **Suites (`--suite`)** - `raylib` — Eight **raylib-specific** questions (this document). - `kb` — Eight **kb-repo / product** questions (contributor dogfood). - `generic` — Eight **repo-neutral** questions. Use with `--repo` for arbitrary upstreams. Override questions with `--questions-file path.json` (JSON array of exactly eight strings) to lock a custom suite without forking the script. **Example** ```bash npm run eval:init -- --suite generic \ --repo https://github.com/raysan5/raylib.git \ --label raylib-upstream-smoke \ --auto-score ``` Options: `--clone-branch main`, `--clone-depth 1` (default shallow; use `0` for full history). The artifact records `run.clone_url`, `run.target_cwd`, `run.run_dir`, `run.run_name`. **Artifacts** - Default path: **`~/.kb/evaluations//artifact.json`**. Override with `--out`. - Rebuild artifact from existing scratch: `--skip-init --run-dir ~/.kb/evaluations//` (expects matching clone at `~/.kb/evaluations//repo/`). - Automated artifacts may include extra `run` fields for traceability. Tools should treat unknown keys as forward-compatible metadata. **Docs generate smoke (`scripts/eval-gen-doc.mjs`)** - Same run root: **`~/.kb/evaluations//`** with `artifact.json` and `gen-doc.log`. - Flow matches CLI: each scenario runs **`docs generate --finalize`** (draft + `awaiting_review`), optional **`--reject-once ""`** (one LLM revision; writes **`diff-introduction.txt`** / **`diff-howto.txt`** when a patch is produced), then **`docs generate --accept`** to commit the SQLite document. - Each finalized doc is also written as **`export-introduction.md`** and **`export-howto.md`** (SQLite body from `docs view --output json`). Open **`README-exports.md`** in that folder for absolute paths and a one-line `open` / `xdg-open` hint. - Default **`--base dogfood`**. Optional **`--skip-purge`** to skip deleting prior eval-titled docs (ids derived from fixed `documentTitle` strings). - Exit `1` only on hard failure; artifact `status` is `complete` when automated checks pass for both scenarios. - Interactive parity: **`kb chat`** supports **`/docs generate "" …`** (questionnaire + review loop) and **`/facts`** (same surface as **`kb facts`**). ## Evaluation Design This evaluation should be run at least twice against the same codebase snapshot or equivalent branch state: 1. Baseline run: - Build the KB from scratch with the normal workflow. - No special second-agent KB-maintenance strategy beyond answering `kb init` questions accurately. 2. Comparison run: - Repeat on a fresh disposable base after using the intended two-agent workflow. - Keep the question set, scoring rubric, and artifact schema identical. ## Standard Procedure ### Phase 1: Initialize a Fresh KB From the target repo root (or a clone): 1. Run: `kb init --base raylib-2026-04-27-1303 --non-interactive` (pick a fresh disposable name; `eval-run.mjs` generates this pattern automatically). 2. Or interactively: start `kb`, then `/init --base ` 3. Let `kb init` complete all passes through `pass-graph`. 4. Save the resulting run metadata. Use a disposable base name that matches your eval run folder when using `eval-run.mjs`, or any unique name for manual runs. ### Phase 2: Capture Build Metrics Collect: - Base name - Git branch and commit of `~/raylib/` - Start/end timestamps - `kb init` run ID from `kb logs` - Total init duration - Total init input tokens - Total init output tokens - Estimated init cost - Number of documents created - Graph entity count - Graph relationship count ### Phase 3: Evaluate Answer Quality Run a fixed question set: - `kb query "" --base ci-raylib- --output json` - `kb chat` against the same base (optional, mark `not_captured` if skipped) Questions adapted for raylib (see Canonical Question Set below). ### Phase 4: Score the Results Each question gets a rubric score on four axes. Use `--auto-score` with the eval runner when available. ### Phase 5: Publish ```bash kb publish jekyll --base ci-raylib- --dir ~/raylib-kb-docs/ --apply ``` ## Canonical Question Set Use these raylib-adapted questions for all runs. If revised, copy the old suite forward and record the change in the artifact. 1. What is raylib for, and what are its main capabilities? 2. How does raylib's architecture work, including modules and platform support? 3. How do I install and build raylib, including dependencies and build systems? 4. What configuration options and compile flags does raylib support? 5. How does raylib handle graphics backends and platform-specific rendering? 6. What are the coding conventions and style guidelines for contributing to raylib? 7. What are the main gotchas, constraints, and known limitations of raylib? 8. What does the raylib roadmap say about future plans, and what is the recent version history? ## Scoring Rubric Score each answer on four axes from `0` to `4`. ### Correctness - `4`: Factually correct and grounded in the repo/KB. - `3`: Mostly correct with minor omissions. - `2`: Mixed; contains meaningful inaccuracies or unsupported inference. - `1`: Mostly wrong or misleading. - `0`: No useful answer. ### Usefulness - `4`: Directly helps a developer act or understand the system. - `3`: Helpful but incomplete. - `2`: Some signal, but requires substantial follow-up. - `1`: Barely helpful. - `0`: Not helpful. ### Specificity - `4`: Uses concrete repo-specific details, commands, or mechanisms. - `3`: Some concrete detail, but still generic in places. - `2`: Partly generic. - `1`: Mostly generic. - `0`: Purely generic or evasive. ### Evidence Handling - `4`: Clearly constrained to evidence, acknowledges uncertainty appropriately. - `3`: Reasonably evidence-grounded. - `2`: Some speculation or weak grounding. - `1`: Strong speculation or unsupported claims. - `0`: No evidence discipline. ## Aggregate Metrics For each run, compute: - Mean score per axis for `query` - Mean score per axis for `chat` - Combined mean score - Pass rate where `correctness >= 3` and `usefulness >= 3` - Coverage notes by topic area ## Success Thresholds Treat a run as promising if all are true: 1. `kb init` completes successfully on a fresh disposable base. 2. The graph store is populated with non-zero entities and relationships. 3. Combined pass rate is at least `70%`. 4. At least `6/8` questions score `correctness >= 3`. 5. At least `6/8` questions score `usefulness >= 3`. Treat the two-agent theory as supported only if the comparison run beats the baseline on at least one of: - Better combined answer quality - Lower total token cost - Lower elapsed time - Better requirement/process capture in qualitative notes without causing a meaningful regression in the other categories. ## Artifact Storage Every run — even weak or partial ones — should still **emit** a JSON artifact so comparisons stay reproducible. Default layout: `evaluation/runs/YYYY-MM-DD-