Measuring AI output in a GTM stack, instead of trusting it

The problem

AI-generated GTM output gets trusted or distrusted on vibes. A brief reads fluent, so it ships. Then someone finds a fabricated fact in an email that went to a real prospect, and the whole category loses credibility.

The gap isn't model quality. It's that almost nobody attaches a number to "how often is this wrong, and wrong in what way." Doing that is unglamorous, and it is the actual skill: turning an unpredictable system into a governable one.

What it does

• Reports rates with intervals, never bare. 60.5% from 43 samples and 60.5% from 4,300 samples are different claims. Every proportion ships with a Wilson score interval, so sample size is visible inside the claim rather than buried in a methods note.
• Refuses to return flattering defaults. Precision over zero predictions is None, not 0.0. A system that flagged nothing has undefined precision, and saying so is the honest answer.
• Breaks results down by group and flags concentration. If most of the sample came from one source, the pooled number is largely a statement about that source — the report says so, unprompted.
• Chance-corrects agreement. Raw agreement between an LLM judge and human labels is inflated by base rate — two raters who both mostly say "no" agree ~90% of the time while sharing no signal. Cohen's kappa is reported alongside.

Result one: re-analysing a published number, including my own

groundtruth scanned 30 public repositories for a specific defect pattern, hand-checked all 43 candidates against the live GitHub API, and published 26 true positives / 17 false positives = 60.5% precision, false positives included rather than hidden. That's already more honest than most published numbers. This harness asks the next two questions of it.

Precision: 60.5% (26/43) · 95% Wilson interval [45.6%, 73.6%] — 28.1 percentage points wide. At n=43 the sample cannot distinguish "about half" from "about three quarters." The point estimate was never wrong; alone, it implied a precision the sample doesn't support.

65% of the sample (28 of 43) is a single repository. The detector scores 78.6% on mem0ai/mem0 and 0-for-5 on Flask, 0-for-2 on Express, 0-for-2 on Streamlit. The pooled 60.5% is close to a weighted average of "works well on mem0" and "no signal elsewhere" — a materially different claim from "60% precision on open-source repos." Neither finding needed new data; both were sitting in the committed JSON.

python scripts/analyze_groundtruth.py [path/to/groundtruth]

No network access, no API key. Every number recomputes from output/verified_findings.json in a groundtruth checkout.

Result two: can a reader verify the numbers in a README?

A write-up citing figures a reader cannot find in the repository is asking to be taken on trust. scripts/check_readme_grounding.py extracts every numeric claim from a README and checks it against that repo's committed data files (via git ls-files — the question is what a reader of the published repo can actually verify). Run across this whole portfolio:

The standout is salesforce-gtm-org at 25%, and it's structural, not dishonest. Its committed data files were package.json and sfdx-project.json only — no deploy log, no raw SOQL output — so the org results couldn't be checked by anyone reading the repository. A manual audit had reached the same conclusion independently; this reproduced it in seconds and caught the next instance without anyone remembering to look. Rates are not comparable across repos — a repo committing rich output scores high regardless of whether its claims are true; the number is a prompt to look, not a verdict.

python scripts/check_readme_grounding.py ../some-repo [../another-repo ...]

Run it

python3 -m venv .venv
./.venv/bin/python -m pip install -e ".[dev]"
./.venv/bin/python -m pytest tests/ -q     # 87 tests

Tested on Python 3.9.6 — the system Python that ships with macOS. The metrics core imports nothing outside the standard library, so it runs anywhere Python does.

What this doesn't do

• It doesn't call a model. Numeric claims are labelled deterministically, which is why this runs at zero cost with no API key.
• It only checks numeric claims. Whether a sentence fairly characterises a source needs a reader or a judge model — LLMClaimChecker defines that slot and raises NotImplementedError rather than pretending to fill it.
• "Supported" means the value appears in a source, not that it's used correctly. A brief that swaps two real figures between quarters passes this check — pinned in place by test_documented_limitation_swapped_numbers_pass so it stays known rather than becoming a surprise.
• The groundtruth re-analysis is not independent validation of groundtruth. Same author, same underlying run — a sharper reading of a published result, not a second opinion on it.
• Wilson intervals assume independent samples. The 28 candidates from mem0ai/mem0 share one codebase's conventions and are plausibly correlated, meaning the true interval is likely wider than reported, not narrower.

A real bug this caught in itself

The concentration warning used >= against its 0.5 threshold, which fired on a perfectly balanced two-group sample — the one case that should never warn. Fixed to >; the test that caught it is test_no_warning_on_balanced_sample. The three decisions worth arguing about — undefined is not zero, point estimates never ship alone, agreement is measured against chance — are documented at the top of src/gtm_eval/metrics.py.

Files

src/gtm_eval/metrics.py            Wilson intervals, precision/recall/F1 (None-safe), Cohen's kappa
src/gtm_eval/grounding.py          numeric-claim extraction + normalisation (currency, %, lakh/crore)
src/gtm_eval/report.py             stratified reporting, concentration warnings
scripts/analyze_groundtruth.py     Result one, above
scripts/check_readme_grounding.py  Result two, above
tests/                             87 tests