Repository Audit — 2026-07-02¶
Scope. Full /audit drift check plus a trust-path bug hunt and a clean-room
run verification, performed 2026-07-02 at main = 7be24ac after ~10
autonomously-executed milestones. Findings are recorded here and acted on via
Act 2, Milestone 16 (ROADMAP2.md); per the /audit
contract, nothing was fixed in the auditing session itself.
Headline verdict: the project works exactly as documented. A skeptical
clean-room run: gate fully green (443/443 tests, strict mypy, ruff — 8.5s warm),
live eval numbers match eval/badge.json, eval/history.jsonl, and the
README's inline example table digit-for-digit (including the two deliberately
sub-1.0 offline numbers), all seven demo doors behave as advertised, the MCP
server boots and lists its 7 tools, and the whole verification left zero
tracked-file changes. The drift is real but shallow: documentation lag around
the in-flight Milestone 15, phantom spec numbers, and eight code findings —
none in the deterministic core engine paths, all in the newest (M15/agent)
layers or in documented verifier blind spots.
1. Milestone 15 state (in flight)¶
Plan: specs/0103 — 5 units. Units 1–3
(plan; best-effort idempotency + ADR 0026; recorder/scrubber/runbook + the 0106
refactor) are merged (PRs #114–#117). Units 4–5 remain: the single
maintainer-authorized real send, and the milestone close (docs, empty-diff
audit, tag).
A real send has never happened. Evidence: data/execution/ contains only
its README; eval/history.jsonl ends 2026-06-28 (M10 close); README still
truthfully states nothing has been sent. To execute (runbook:
DEPLOYMENT.md): create a sandbox repo, mint a fine-grained
PAT (Issues: Read and write on that one repo — Read is needed by the
idempotency pre-check), set TESSERA_EXEC_OWNER / TESSERA_EXEC_REPO /
TESSERA_GITHUB_TOKEN in the gitignored .env, rehearse without
TESSERA_EXEC_APPROVE, then run once with TESSERA_EXEC_APPROVE=true and
commit the scrubbed receipt. Do this only after fixing findings B1–B4 below
— two of them directly affect the one-shot's artifact and safety.
2. Drift findings (docs vs. reality)¶
| # | Sev | Finding |
|---|---|---|
| D1 | HIGH | STATUS.md ends at Milestone 14 while four M15 PRs (#114–#117) are merged — the "resumable from written state" rule was violated for those sessions. CHANGELOG.md [Unreleased] says "(nothing yet)", which is false. |
| D2 | HIGH | WRITEUP.md limitation "a real create-issue is not idempotent … recorded as a caller responsibility … rather than engineering a dedup key" is now false — ADR 0026 + agent/execution.py engineered exactly that key. |
| D3 | HIGH | Seven spec numbers were consumed but never committed as files: 0050, 0069, 0071, 0076 (milestone-close units), and — in the most trust-bearing recent work — 0104, 0105, 0106. STATUS/commits cite them as if they exist (verified via git log --all --diff-filter=A). The spec-before-code discipline regressed at M15. |
| D4 | MED | DEPLOYMENT.md self-contradicts: the component table still says embeddings are "Deliberately not built" (Phase-4 text) while the same file documents the M6/M7 HANA runs. |
| D5 | MED | README: "nine post-roadmap milestones" — it narrates ten (M5–M14). |
| D6 | LOW | CAPABILITIES.md describes unbuilt capabilities (PDF ingestion, incremental re-ingest, conversational continuity) without status markers — readable as current behavior. |
| D7 | LOW | ~60 stale merged remote branches (origin/feat/*, origin/m*-*); duplicate spec basenames 0075/0079. |
Tree hygiene is otherwise clean: site/ ignored, .env gitignored (holds only
HANA_* variables), gitleaks in pre-commit, no strays, no scope creep found.
3. Code findings (trust-bearing paths)¶
Spot-verified locations; each fix is a unit in M16 (ROADMAP2.md) with its own spec + regression test, and the trust-bearing ones carry the mandated adversarial review.
| # | Sev | Confidence | Finding |
|---|---|---|---|
| B1 | HIGH (for the M15 one-shot) | Verified | scripts/record_real_execution.py writes data/execution/receipt.json + MANIFEST.json unconditionally on any approved attempt — a re-run (e.g. the documented "re-run returns exists" demonstration, or an error outcome) overwrites the historic created receipt. Fix: never clobber (timestamped filenames or refuse-if-exists). |
| B2 | MED | Likely | GithubActuator._existing pre-checks issues via the idem- label filter and only scans bodies within that list (execution.py:~450–584). If the label is dropped/deleted/uncreatable, the filter returns empty → duplicate create despite the body marker. (Correction, review 0109: ADR 0026's original Consequences did name and accept the attach-failure variant as a known limit; the deletion and permission-based silent-drop variants were unnamed.) Fix: make the scan label-independent (unfiltered bounded body-marker scan); ADR 0026 addendum. |
| B3 | MED (cheap) | Verified | GithubActuator.token is a plain frozen-dataclass field (execution.py:402) — the PAT appears in repr()/tracebacks/logs. Fix: field(repr=False). |
| B4 | MED | Verified | payloads._section wraps log/diff values in fences with **no escaping** ([payloads.py:223–229](https://github.com/robert-vetter/tessera/blob/main/src/tessera/agent/payloads.py)) — a log line containing breaks out of the fence and injects markdown into a real created issue. (Speculative extension: a log embedding a crafted tessera-idempotency-key marker could force a false exists → denial-of-create.) Fix: neutralize fence runs (longer dynamic fences) + document the marker-spoof residual. |
| B5 | LOW | Speculative | {pr} path-segment validation rejects only empty///whitespace (payloads.py:272–291); ?, #, .. would pass into the URL path. Unexploitable with current synthetic ids; add an allowlist (^[A-Za-z0-9._-]+$). |
| B6 | LOW (documented posture) | Verified behavior | Verifier blind spots inherent to ADR 0005 but not all named: (a) over-citation is unpenalized — is_supported accepts a claim if any cited record supports it (metrics.py:88); (b) normalize() strips spaces/punctuation, so containment can match across word boundaries. Action: ADR 0005 addendum naming both; optionally report (not gate) an over-citation metric. |
| B7 | LOW | Verified behavior | Refuse-kind eval cases skip faithfulness accounting entirely (harness.py:142–145) — claims emitted on a wrongly-answered refuse case are never faithfulness-checked. Action: score them. |
| B8 | LOW | Speculative/edge | Genericness fixpoint counts empty stems as distinct firms (resolution.py:258–278); downstream gates contain the blast radius; no current-data impact. Action: note in ADR 0018; regression-pin if touched. |
Positives worth recording: no silent exception swallowing on trust paths
(transport failures become explicit error/inconclusive outcomes); no
hardcoded demo entities in engine logic; ordering systematically pinned
(sorted traversals, canonical JSON); credentials provably absent from receipts
by construction, with redact_receipt + gitleaks as depth.
4. Eval health (what a skeptical senior reviewer would say)¶
The mechanics are sound (one shared scorer, per-vertical batteries, floor
enforced via exit code in scripts/gate.sh and CI). The honest critique:
- Lookup-path faithfulness is partly tautological — claims are the cited record's text; the metric has teeth only on composed/aggregate/conclusion/ shared-fragment shapes (where the adversarial tests do prove it can fail).
- 25 gold cases (11/9/5) authored by the system's own author over self-generated synthetic data; the synthetic batteries derive expectations from the same graph they score. Real mitigations exist (the recorded miss→close trails 0.929→1.0 / 0.917→1.0 / 0.000→1.0 on the real connector; two sub-1.0 numbers deliberately kept in CI; the committed ADR 0005 counter-specimen) — but external validity is thin. The strongest possible answer is evaluation on data we didn't author (Act 2: BYO-repo battery, SALT real data).
- The M11–M15 agent arc is verified by pytest properties, not by eval
batteries — "faithfulness 1.0 across four boundaries" is a test-suite
claim;
eval/history.jsonlends at M10. Fine, but the docs should say so precisely. - B6/B7 above are metric blind spots that belong named in ADR 0005.
5. External dependencies / accounts inventory¶
| Capability | Requires | Note |
|---|---|---|
| Engine, both verticals, github_actions battery, eval + gate, MCP server (simulated), chat | Nothing — offline, pure stdlib | The default and CI posture; clone-and-run in ~2–5 min |
| Semantic retrieval / embedding-ER / cloud eval (M6–M7 closes) | SAP HANA Cloud instance (NLP feature), HANA_* env, cloud extra |
Recorded closes are timestamped history; re-demonstrating live needs a running instance — if it was a trial it may have expired; verify before any live demo |
| M15 real one-shot | GitHub sandbox repo + fine-grained PAT (Issues RW) + TESSERA_EXEC_* env |
~15 min of maintainer work; never in CI |
| Narration (optional, ADR 0013) | GenAI Hub key or ANTHROPIC_API_KEY |
Contract-tested against fakes; never exercised live |
| Docs site | GitHub Pages | Live |
6. Demo readiness¶
- For an SAP engineer: strong today.
tessera-chat(routing,:show Nprovenance walks,:trust, live verifier verdict), the ER refusal stories, the gated eval, the MCP arc, 26 ADRs, the write-up. 2–5 minutes from clone. - For a non-technical audience: weak. Everything is a terminal; no web UI, no hosted instance, synthetic data. Answers surface raw markdown chunks (an evidence dump, honest but unimpressive to a "summarize" request).
- Missing for a "product" impression: web UI, hosted demo, bring-your-own data, persistence, and the completed "it actually acted, once, on the record" story (one maintainer command away). These are precisely Act 2's M17/M18.