Skip to content

The Faithfulness Floor

What does an agent without an evidence gate do on the same corpus, the same questions, judged by the same verifier? This benchmark answers that with a deterministic, reproducible measurement — no LLM anywhere in the measurement, no LLM judge, no accounts — comparing Tessera's evidence-gated answer engine against its own retrieval layer run ungated. Anyone can reproduce every number below from a clone:

uv sync
uv run tessera-benchmark             # the summary
uv run tessera-benchmark --markdown  # exactly the tables below
uv run tessera-benchmark --cases     # every case (gold + synthetic), both sides

A CI test regenerates the tables below on every build and fails on any byte difference, so the tables on this page cannot silently drift from the measured truth (a second test pins the headline numbers this page and the README quote in prose). Spec: specs/0122.

What is being compared

Both answerers run over the same corpora (the three measured batteries: business, devex, and github_actions — the last built from real CI logs of this repository), the same cases (curated gold + generated synthetic), and are judged by the same deterministic verifier with the same claim grammars (ADR 0005, ADR 0011). The scoring function is the eval harness's own — imported, not reimplemented — so benchmark semantics cannot drift from eval semantics.

Tessera (gated) — the full engine: question routing, knowledge-graph entity resolution, cross-source composition with recomputed (fully cited) aggregates, multi-hop RCA, conflict surfacing, principled refusal on ambiguity / missing evidence / incomparability, and the verifier standing behind every claim.

Ungated retrieval (retrieve-and-recite) — Tessera's own retrieval layer with everything above it removed: BM25 top-5 over the identical records, each hit recited verbatim as a claim with its provenance, refusing only when the question shares no content token with any record. This is deliberately not an authored strawman: it is the exact retrieval the gated engine itself uses, so the measured gap is attributable to precisely the layers being claimed — the gate, not the retriever. Where recitation suffices (plain lookups, zero-overlap refusals), the baseline scores full marks, and the per-case table below shows it.

What the numbers mean

All scoring semantics are the eval's own (ADR 0005, tessera/eval/harness.py):

  • Faithfulness — fraction of emitted claims deterministically supported by their cited evidence (the verifier). Tessera gates its build on this being 1.000; it is provably able to fail and has failed before (WRITEUP).
  • Coverage — fraction of the expected supporting evidence an answer actually cites.
  • Quality — fraction of cases handled correctly: answerable cases must contain the expected facts; refusal cases must refuse.
  • Trustworthy outcomes (the composite, defined here) — fraction of cases where nothing went wrong: an answer-kind case counts only if the answer is grounded, every claim passes the verifier, all expected facts appear, and all expected evidence is cited; a refuse-kind case counts only if the system refuses — and any claim it still emits alongside the refusal must pass the verifier too (claims on refuse-kind cases are scored, not exempt). Each component is computed by the harness's own scoring function run per-case, so the composite introduces no new semantics. (Weighting differs by design: faithfulness/coverage/quality are pooled over claims / expected ids / cases exactly as the eval pools them; trustworthy outcomes are per-case, unweighted.)

The numbers

Mode: offline, deterministic (lexical BM25, no semantic index, no LLM); baseline depth k=5; 110 cases across 3 batteries.

Battery · cases Answerer Faithfulness Coverage Quality Trustworthy outcomes
business · gold (11) Tessera (gated) 1.000 1.000 1.000 1.000
business · gold (11) ungated retrieval 1.000 0.375 0.182 0.182
business · synthetic (53) Tessera (gated) 1.000 1.000 1.000 1.000
business · synthetic (53) ungated retrieval 1.000 0.194 0.038 0.038
devex · gold (9) Tessera (gated) 1.000 0.950 0.889 0.889
devex · gold (9) ungated retrieval 1.000 0.300 0.222 0.222
devex · synthetic (24) Tessera (gated) 1.000 1.000 1.000 1.000
devex · synthetic (24) ungated retrieval 1.000 0.414 0.583 0.125
github_actions · gold (5) Tessera (gated) 1.000 0.833 0.800 0.800
github_actions · gold (5) ungated retrieval 1.000 0.667 0.000 0.000
github_actions · synthetic (8) Tessera (gated) 1.000 1.000 1.000 1.000
github_actions · synthetic (8) ungated retrieval 1.000 0.500 0.625 0.250

Trustworthy-outcome gap (gated − ungated):

  • business · gold: 1.000 vs 0.182 (+0.818)
  • business · synthetic: 1.000 vs 0.038 (+0.962)
  • devex · gold: 0.889 vs 0.222 (+0.667)
  • devex · synthetic: 1.000 vs 0.125 (+0.875)
  • github_actions · gold: 0.800 vs 0.000 (+0.800)
  • github_actions · synthetic: 1.000 vs 0.250 (+0.750)

Structural notes (computed — properties of the cases and corpus, independent of answerer). reachable Q: answer-kind cases whose expected facts all occur verbatim in some record, i.e. where recitation could in principle pass the quality check — on the rest, the expected phrasing is the gated engine's own composed output, so the quality gap there measures whether composition happened at all, not retrieval quality. >k support: cases expecting more evidence ids than the baseline's retrieval depth (k=5) — full coverage is structurally out of reach there. vacuous: answer-kind cases declaring no expected support / no expected facts (those components auto-pass, per the harness's own convention).

  • business · gold: 6 answer / 5 refuse · reachable Q 1/6 · >k support 1 · vacuous 1 support, 0 facts
  • business · synthetic: 45 answer / 8 refuse · reachable Q 0/45 · >k support 18 · vacuous 0 support, 0 facts
  • devex · gold: 6 answer / 3 refuse · reachable Q 3/6 · >k support 1 · vacuous 0 support, 0 facts
  • devex · synthetic: 11 answer / 13 refuse · reachable Q 10/10 · >k support 0 · vacuous 0 support, 1 facts
  • github_actions · gold: 3 answer / 2 refuse · reachable Q 2/3 · >k support 0 · vacuous 0 support, 0 facts
  • github_actions · synthetic: 3 answer / 5 refuse · reachable Q 0/0 · >k support 0 · vacuous 0 support, 3 facts

Per-case outcomes on the curated gold sets (✓ trustworthy; ✗ with the failed checks — F: a claim failed the verifier, C: expected evidence not cited, Q: wrong disposition or expected facts missing):

Case Kind Engine Tessera (gated) Ungated retrieval
business/01_mueller_cross_source answer compose ✓ ✗ C+Q
business/02_mueller_lookup_retrieve answer retrieve ✓ ✓
business/03_lumiere_billing_compose answer compose ✓ ✗ Q
business/04_atlas_mixed_currency answer compose ✓ ✗ C+Q
business/05_ambiguous_refusal refuse compose ✓ ✗ Q
business/06_out_of_scope_refusal refuse compose ✓ ✓
business/07_mueller_renewal_conflict answer compose ✓ ✗ C+Q
business/08_superlative_synonym answer route ✓ ✗ Q
business/09_currency_scope_refusal refuse route ✓ ✗ Q
business/10_same_name_address_refusal refuse compose ✓ ✗ Q
business/11_same_name_same_address_refusal refuse compose ✓ ✗ Q
devex/01_r1042_rca_recurrence answer rca ✓ ✗ C+Q
devex/02_pr201_change_summary answer summary ✓ ✗ C+Q
devex/03_payments_oncall_lookup answer route ✓ ✓
devex/04_notifications_oncall answer route ✓ ✗ C+Q
devex/05_passed_run_refusal refuse route ✓ ✗ Q
devex/06_unknown_run_refusal refuse route ✓ ✗ Q
devex/07_out_of_scope_refusal refuse route ✓ ✓
devex/08_r1042_fix_chain answer rca ✓ ✗ C+Q
devex/09_checkout_oncall_semantic answer route ✗ C+Q ✗ C+Q
github_actions/01_ruff_format_failure answer rca ✓ ✗ C+Q
github_actions/02_pages_deploy_recurrence answer rca ✓ ✗ Q
github_actions/03_passed_run_refusal refuse rca ✓ ✗ Q
github_actions/04_unknown_run_refusal refuse rca ✓ ✗ Q
github_actions/05_pages_synonymy_lookup answer lookup ✗ C+Q ✗ C+Q

How to read this honestly

The baseline's faithfulness is 1.000 — and that is the finding, not an error. A system that only recites verbatim snippets passes a containment verifier by construction: recitation is trivially "faithful". It still fails the task — it answers questions it should refuse (ambiguous entities, conflicting evidence, cross-currency sums, runs that never failed), and it cannot state what the evidence adds up to (a recomputed total, a recurrence across runs, a fix chain), so the facts a correct answer must contain never appear. The floor is cheap if you never assert anything beyond quotes. Tessera holds the same floor while actually asserting things — sourced aggregates, entity-resolved cross-source links, multi-hop conclusions — and that combination, not faithfulness alone, is what "trustworthy outcomes" measures. On these corpora, the gap column is what removing the gate costs.

Part of the quality gap is definitional — and the artifact computes how much. The expected facts on compose/RCA cases are exact strings in the gated engine's own phrasing ("Recurring failure", "Total net order value across …"), because the gold sets double as that engine's regression suite. An answerer that does not compose — or composes in different words — cannot satisfy those facts regardless of retrieval quality. The worked example is in the per-case table: on github_actions/02_pages_deploy_recurrence the baseline cited every expected record and fails only the engine-phrased recurrence fact. The structural notes inside the generated block count exactly where this line falls ("reachable Q"): on business synthetic, the baseline's published 0.038 is its structural ceiling — those rows measure the value of composing at all, under our phrasing, not retrieval ranking. Where the expected phrasing is plain record text (devex synthetic: 10/10 reachable), the baseline is credited — quality 0.583 — which is how you can tell the harness credits whatever recitation can legitimately earn. The refusal and coverage columns are phrasing-independent throughout (disposition and evidence ids, not wording).

Tessera's own misses are in the table. The gated side is not decorated to 1.000: offline, devex gold shows 0.889 trustworthy (a named semantic near-miss the deterministic matcher does not bridge: devex/09_checkout_oncall_semantic) and github_actions gold shows 0.800 (a real error-class synonymy in live CI logs: github_actions/05_pages_synonymy_lookup; its close via in-database embeddings on SAP HANA is recorded separately in the WRITEUP — it needs an account, so it is not part of this offline artifact). Both sides fail those two cases for the same lexical reason. A benchmark where our own side shows its misses is the kind you can trust.

What the extractive baseline does — and does not — proxy. The deterministic baseline never paraphrases, never blends sources, never invents a number. For faithfulness that makes it a best case by construction: recitation maxes a containment verifier, and paraphrase, blending, and invention can only score worse there. On task outcomes the direction is not guaranteed: a retrieve-then-generate agent might recover some of this baseline's quality/coverage failures (state the total, cite the right rows) — and anything it asserts would still have to survive the same verifier. Measuring that honestly would require an LLM inside the benchmark, which would end its determinism and reproducibility (and put a model in a trust path, which this project refuses). The seam at the bottom of this page takes any answerer — measure yours.

How this can fail

A benchmark that cannot fail is decorative. This one can, in four ways:

  1. The doc pin. CI regenerates the tables above and fails on any difference — if the corpus, engine, or scoring changes, this page must be regenerated in the same PR or the build is red.
  2. The direction pin. A CI test asserts the gated engine's trustworthy-outcome rate is strictly above the ungated baseline's on every battery and case set. If a change makes that false anywhere, the build says so; the claim is not allowed to outlive its truth.
  3. The equality pin. A CI test asserts the gated side reproduces the offline eval numbers exactly — the benchmark cannot quietly diverge from the numbers the project publishes elsewhere.
  4. The floor itself. Faithfulness 1.000 on the gated side is enforced by the build gate independently of this benchmark, and the verifier is adversarially tested (an injected unfaithful claim is caught).

Limitations

  • The corpora are ours. business and devex are authored synthetic corpora (with deliberate ambiguity, conflicts, and variant entities); github_actions is real CI data from this repository. Nothing here is a third-party benchmark suite; the artifact's claim is about the measured gap between gated and ungated answering over identical evidence, not about generalization to your data. (For that, tessera smoke runs a trust-floor battery on your own connected repo — see PILOT.md.)
  • The cases are ours, written against the gated engine. The gold sets double as its regression suite (two cases it fails are kept in), and the refuse-kind share (40% of gold) shapes the headline gap — which is why the kind mix and every per-case outcome are printed above rather than left to aggregate impressions.
  • n is small and stated: 110 cases (25 gold, 85 synthetic) across three batteries. Synthetic expected values are recomputed from the data at eval time (ADR 0007); on business compose cases the expected phrasing is the engine's template (the definitional boundary above; recorded in the ADR 0007 addendum), while devex/github_actions synthetic facts are plain record text.
  • The baseline is extractive. By construction this understates the faithfulness risk of generation (see above); on task quality a generating agent could recover some of the failures — the answerer seam below exists to test exactly that.
  • Offline mode only. The recorded online close of the synonymy case (embeddings on SAP HANA) is documented in the WRITEUP but excluded here to keep the artifact account-free and byte-reproducible.

If you want to attack this: swap in your own baseline answerer (tessera.eval.benchmark.ungated_variant shows the seam — any (case, graph, kb, index) -> Answer callable), or add cases to the gold sets and re-run. The coverage and refusal columns are open challenges for any answerer; the quality column on compose/RCA cases keys on the gated engine's phrasing (the structural notes say exactly where), so the honest way to attack it is with record-phrased cases of your own. Issues and PRs that break the number are the point.