Groundtruth · real-data verification run

A PR can claim to fix a bug and only add a test. We scanned 30 real repos to see how often.

Groundtruth checks developer activity against deterministic, checkable rules — no LLM guessing. This is Rule #001 (auto-close-without-fix), run against 30 real, active public repositories, with every candidate match hand-verified against the actual GitHub issue and PR content before being called a finding. It is a candidate-generation heuristic with mandatory human verification, not an auto-blocking enforcement gate — see "What this is / is not" below.

30real repos scanned
43raw candidate matches
26verified true positives
23distinct issues (3 pairs share an issue, different authors)
17verified false positives
0true positives that merged
60.5%precision, PR-level (26/43)
69.7%precision, issue-level, deduped (23/33)

The rule

Auto-close-without-fix: a PR references an issue with a GitHub closing keyword (Fixes #N, Closes #N, Resolves #N) while its diff touches only test files. If that PR merges, GitHub silently closes the issue — without the underlying code ever changing.

Every finding below is checkable, not probabilistic: the rule doesn't guess whether a PR "seems" thin, it reads the actual linked issue's labels and text, and the actual list of changed files, and applies one fact-based test.

What this is / is not

Is: a candidate-generation heuristic. It flags PRs matching a deterministic pattern, and every flag on this page was then manually checked against the real issue and PR before being called a finding — that manual step is why zero of the true positives below actually caused harm (see "The number that matters most").

Is not: an auto-blocking enforcement gate. It does not merge, close, or fail a check on its own judgment. Recent tooling (CodeRabbit's Agentic Change Management, launched the same week as their $1.5B valuation; GitHub Copilot's coding-agent auto-validation) automates the verification step itself — this rule automates only the flagging, and treats verification as a human's call, on purpose.

Nearest published research: arXiv:2601.04886 scanned 23,247 AI-authored PRs and named "Phantom Changes" — descriptions claiming a change that was never implemented — as 45.4% of message-code inconsistencies found, releasing a 974-PR annotated dataset because no detector existed at the time. Same failure mode this rule targets, at PR-description granularity rather than repo-artifact granularity. Caveat in the same paper: high-inconsistency PRs are only 1.7% of their full corpus — this is a real but narrow-incidence problem, not a majority-case one.

The number that matters most

Zero of the 26 verified true positives actually merged. Every one was caught and rejected by a human maintainer, by hand, before it could auto-close a real bug. Groundtruth didn't catch anything a maintainer missed — it demonstrates that the same judgment call, made once per PR by a person, could be made instantly and consistently by a rule instead.

Two bugs the verification pass found — in Groundtruth's own rule

Verifying against real data didn't just validate the rule. It broke it, twice, in ways synthetic test fixtures never would have.

1. PR-template boilerplate matched as a real reference

GitHub's default PR template includes example text like <!-- e.g. "fixes #123" -->. The original regex matched that literal example just like a genuine reference — and #123 is a real (usually unrelated) issue in nearly every repo old enough to have one. This alone produced 10 false positives across pydantic/pydantic and vitejs/vite in the first scan pass. Fixed by stripping HTML comments before matching; confirmed by rescan — all 10 disappeared.

2. A filename can look like a test and not be one

Streamlit ships a real product feature — its AppTest testing framework — implemented in a file literally named app_test.py. The filename heuristic can't distinguish that from an actual test file. Left as a documented, tested limitation rather than a repo-specific patch, which would just trade one blind spot for another.

Verification method

Nothing here is published on the heuristic's word alone.

What "verified" means

For every one of the 43 candidates, the real GitHub issue (title, body, labels) and the real PR (body, merge state) were fetched from the live API and read. A true positive requires the linked issue to be a genuine, usually maintainer-labeled (bug, often accepted) product defect — not a request to add test coverage or fix the test suite itself.

What "false positive" means here

In every false positive in this dataset, the linked issue was itself about tests, CI, or a naming collision with real product code — meaning a test-only PR was the correct fix, not a shortcut. That's the exact class the rule's own design doc names as its known limitation.

Every finding, by repository

Ranked by true-positive count. Every PR and issue number links to the live page.

mem0ai/mem0

22 true positives · 6 false positives
17 of these 22 true positives are the same contributor (jaythehardcoder) submitting the same test-only-PR pattern repeatedly — this repo's count is one prolific pattern caught every time, not 22 independent incidents.
PR #6727 → issue #6719 false positive open, not merged
Linked issue is itself about test/CI infrastructure (a broken test assertion, a duplicate test file, a CI matrix), not product behavior — a test-only PR correctly closes it.
tests/vector_stores/test_faiss.py
PR #6687 → issue #6686 false positive merged
Linked issue is itself about test/CI infrastructure (a broken test assertion, a duplicate test file, a CI matrix), not product behavior — a test-only PR correctly closes it.
tests/vector_stores/test_e2e_threshold.pytests/vector_stores/test_score_normalization.py
PR #6943 → issue #6095 false positive closed, not merged
Linked issue is itself about test/CI infrastructure (a broken test assertion, a duplicate test file, a CI matrix), not product behavior — a test-only PR correctly closes it.
tests/embeddings/test_gemini_embeddings.pytests/embeddings/test_langchain_embeddings.py
PR #6234 → issue #6095 false positive closed, not merged
Linked issue is itself about test/CI infrastructure (a broken test assertion, a duplicate test file, a CI matrix), not product behavior — a test-only PR correctly closes it.
tests/embeddings/test_gemini_embeddings.pytests/embeddings/test_langchain_embeddings.py
PR #6332 → issue #6310 false positive closed, not merged
Linked issue is itself about test/CI infrastructure (a broken test assertion, a duplicate test file, a CI matrix), not product behavior — a test-only PR correctly closes it.
mem0-ts/src/oss/tests/helpers/vectorStoreResultContract.tsmem0-ts/src/oss/tests/vector-store-result-contract.test.ts
PR #6832 → issue #6830 false positive open, not merged
Linked issue is itself about test/CI infrastructure (a broken test assertion, a duplicate test file, a CI matrix), not product behavior — a test-only PR correctly closes it.
tests/test_oss_to_platform_migrate.py
PR #6048 → issue #5646 true positive open, not merged
Real TS-side entity-dedup logic bug (unanchored substring check), a regression of an already-fixed Python-side bug (#5630). PR only adds a test.
mem0-ts/src/oss/tests/entity-extraction.test.ts
PR #6644 → issue #6346 true positive closed, not merged
Issue labeled bug/P1-high/accepted: mem0's agent quick-init path exits 0 without MCP-ready state (JSON envelope missing, MEM0_API_KEY unset). PR only adds a test.
tests/test_issue_6346.py
PR #6643 → issue #6362 true positive closed, not merged
Issue labeled bug/P1-high/accepted: reranker failure fallback mutates the caller's input documents in place, writing rerank_score=0.0 onto them (cohere/huggingface/sentence_transformer/zero_entropy). PR only adds a test.
tests/test_issue_6362.py
PR #6642 → issue #6368 true positive closed, not merged
Issue labeled bug/sdk-python/P1-high/accepted: AWS Bedrock Anthropic Converse parsing crashes when a reasoningContent block precedes the text block — indexes content[0] unconditionally. PR only adds a test.
tests/test_issue_6368.py
PR #6641 → issue #6448 true positive closed, not merged
Issue labeled bug/sdk-python/P1-high/accepted: search(rerank=True) reranks only the already-truncated top-k, so reranking can never surface a result ranked just below the limit. PR only adds a test.
tests/test_issue_6448.py
PR #6625 → issue #6562 true positive closed, not merged
Issue labeled bug/sdk-python/P1-high/vector-store/accepted: Turbopuffer vector store silently drops every filter operator except gte/lte. PR only adds a test.
tests/test_issue_6562.py
PR #6618 → issue #6526 true positive closed, not merged
Issue labeled bug/P2-medium/sdk-python: LMSTUDIO_BASE_URL env var is ignored because the default URL is baked into config before the env fallback runs. PR only adds a test.
tests/test_issue_6526.py
PR #6617 → issue #6563 true positive closed, not merged
Issue labeled bug/sdk-python/P1-high/accepted: AWS Bedrock tool calls send malformed Converse requests for amazon and cohere models — system prompt and message-block formatting wrong for non-Anthropic providers. PR only adds a test.
tests/test_issue_6563.py
PR #6616 → issue #6546 true positive closed, not merged
Issue labeled bug/sdk-python/P1-high/vector-store/accepted: S3 Vectors search score is hardcoded for cosine distance, so euclidean-metric scores clamp to 0.0 for any distance > 1. PR only adds a test.
tests/test_issue_6546.py
PR #6615 → issue #6550 true positive closed, not merged
Issue labeled bug/sdk-python/P1-high/vector-store/accepted: Redis vector store only handles scalar equality in filters, so operator dicts (gte/lte) and $or are silently ignored. PR only adds a test.
tests/test_issue_6550.py
PR #6614 → issue #6560 true positive closed, not merged
Issue labeled bug/P1-high/vector-store/accepted: UpstashVector.list() ignores its top_k argument — hardcodes top_k=100 internally and never trims results to match the requested count. PR only adds a test.
tests/test_issue_6560.py
PR #6440 → issue #6346 true positive closed, not merged
Issue labeled bug/P1-high/accepted: mem0's agent quick-init path exits 0 without MCP-ready state (JSON envelope missing, MEM0_API_KEY unset) — duplicate PR against the same issue as #6644. PR only adds a test.
tests/test_issue_6346.py
PR #6439 → issue #6368 true positive closed, not merged
Issue labeled bug/sdk-python/P1-high/accepted: AWS Bedrock Anthropic Converse parsing crashes when a reasoningContent block precedes the text block — duplicate PR against the same issue as #6642. PR only adds a test.
tests/test_issue_6368.py
PR #6408 → issue #6362 true positive closed, not merged
Issue labeled bug/P1-high/accepted: reranker failure fallback mutates the caller's input documents in place — duplicate PR against the same issue as #6643. PR only adds a test.
tests/test_issue_6362.py
PR #6376 → issue #6181 true positive closed, not merged
Real production regression: Codex lifecycle hooks fail on every Windows event, flooding user sessions with errors. PR only adds a test.
tests/test_issue_6181.py
PR #6360 → issue #6301 true positive closed, not merged
Issue labeled bug: HuggingFace embedding client drops api_key on the base_url (TEI/HF Inference Endpoint) path — auth is silently never sent. PR only adds a test.
tests/test_issue_6301.py
PR #6327 → issue #6264 true positive closed, not merged
Issue labeled bug/sdk-python/P1-high/accepted: TS OSS default memory store's filter matcher returns on the first operator in a compound field filter, so e.g. {gte, lte} only ever applies gte. PR only adds a test.
tests/test_issue_6264.py
PR #6304 → issue #6230 true positive closed, not merged
Issue labeled bug/P2-medium: XAILLM forwards its already-built httpx.Client into the http_client_proxies argument, crashing whenever a proxy is configured. PR only adds a test.
tests/test_issue_6230.py
PR #6247 → issue #6183 true positive closed, not merged
Issue labeled bug/sdk-python/P1-high/vector-store/accepted: adding the first memory raises MilvusException on Milvus versions below 2.5. PR only adds a test.
tests/test_issue_6183.py
PR #6201 → issue #6013 true positive closed, not merged
Issue labeled bug/P2-medium: TS OSS Redis vector store builds an invalid RediSearch query string ('' instead of '*') when filters are empty or all-null. PR only adds a test.
tests/test_issue_6013.py
PR #6198 → issue #6065 true positive closed, not merged
Issue labeled bug/P2-medium: the Claude Code mem0 plugin's auto_search:false setting is dead — no hook script reads MEM0_AUTO_SEARCH, so disabling it has no effect and quota keeps burning. PR only adds a test.
tests/test_issue_6065.py
PR #6197 → issue #6085 true positive closed, not merged
Issue labeled bug/P1-high/accepted: OSS default config is internally inconsistent — model=None resolves to gpt-5-mini, which rejects the default temperature=0.1, so every add() silently fails LLM extraction. PR only adds a test.
tests/test_issue_6085.py

run-llama/llama_index

3 true positives · 2 false positives
PR #22696 → issue #22695 false positive open, not merged
Linked issue is about llama-dev's own test suite or CI matrix failing, not product behavior.
llama-dev/tests/test/test_test.py
PR #21967 → issue #15165 true positive open, not merged
Issue labeled bug/P1: MetadataFilters broken for MultiModalVectorStoreIndex. PR body says 'Adds regression coverage' — never touches the filter logic.
llama-index-core/tests/indices/multi_modal/test_retriever.py
PR #22576 → issue #22543 true positive open, not merged
Issue labeled bug/triage: LanceDB delete predicates built via unescaped string concatenation of doc IDs. PR body confirms it only touches tests/test_index.py.
llama-index-integrations/indices/llama-index-indices-managed-lancedb/tests/test_index.py
PR #22306 → issue #22068 true positive open, not merged
Issue labeled bug/triage: ag-ui adapter fabricates a random uuid4() tool_call_id instead of using the real one. PR only adds tests for the (still-wrong) handling.
llama-index-integrations/protocols/llama-index-protocols-ag-ui/tests/test_utils.py
PR #21608 → issue #21607 false positive closed, not merged
Linked issue is about llama-dev's own test suite or CI matrix failing, not product behavior.
llama-dev/llama_dev/test/__init__.pyllama-dev/tests/test/test_test.py

fastify/fastify

1 true positive · 0 false positives
PR #6875 → issue #6836 true positive open, not merged
Real TypeScript .d.ts type-definition bug. PR adds a type-assertion test confirming the wrong type, never touches the actual declaration.
test/types/logger.tst.ts

expressjs/express

0 true positives · 2 false positives
PR #7394 → issue #7348 false positive closed, not merged
Linked issue (#7348) is explicitly labeled 'tests' and asks for test coverage of already-working behavior.
test/req.is.js
PR #7349 → issue #7348 false positive closed, not merged
Linked issue (#7348) is explicitly labeled 'tests' and asks for test coverage of already-working behavior.
test/req.is.js

pallets/flask

0 true positives · 5 false positives
PR #6095 → issue #6071 false positive merged
Linked issue (#6071) is a pytest-version compatibility break in Flask's own test suite, not a product defect.
tests/conftest.pytests/test_cli.py
PR #6099 → issue #6071 false positive closed, not merged
Linked issue (#6071) is a pytest-version compatibility break in Flask's own test suite, not a product defect.
tests/conftest.pytests/test_cli.py
PR #6088 → issue #6071 false positive closed, not merged
Linked issue (#6071) is a pytest-version compatibility break in Flask's own test suite, not a product defect.
tests/conftest.pytests/test_cli.py
PR #6072 → issue #6071 false positive closed, not merged
Linked issue (#6071) is a pytest-version compatibility break in Flask's own test suite, not a product defect.
tests/conftest.pytests/test_cli.py
PR #6087 → issue #6071 false positive closed, not merged
Linked issue (#6071) is a pytest-version compatibility break in Flask's own test suite, not a product defect.
tests/conftest.pytests/test_cli.py

streamlit/streamlit

0 true positives · 2 false positives
PR #16122 → issue #8154 false positive merged
The PR touches lib/streamlit/testing/v1/app_test.py — the implementation of Streamlit's own AppTest product feature, misclassified as a test file by the filename heuristic.
lib/streamlit/testing/v1/app_test.pylib/tests/streamlit/testing/app_test_test.py
PR #15675 → issue #8154 false positive closed, not merged
The PR touches lib/streamlit/testing/v1/app_test.py — the implementation of Streamlit's own AppTest product feature, misclassified as a test file by the filename heuristic.
lib/streamlit/testing/v1/app_test.py

Honesty guardrails