{
  "name": "Helm public receipts ledger",
  "tagline": "AI makes code. Helm makes it work.",
  "description": "Every claim Helm's marketing makes, with its source and the date it was last verified, plus the claims we checked and refuse to make. Public subset of the internal receipts ledger; entries are added as they are verified. Test counts are point-in-time snapshots of whichever gate or census a given entry counted; scopes differ between entries.",
  "url": "https://helm.dev/receipts",
  "generated": "2026-07-10",
  "lastVerified": "2026-07-09",
  "counts": {
    "published": 25,
    "held": 5
  },
  "process": "Rendered by a build script from the internal receipts ledger. Entries are collected and checked with AI assistance against primary sources on the verified date; publishing is human-approved. Market entries summarize third-party reports; the linked source is always the authority.",
  "corrections": "Report errors to @EricMaciver on X; corrections are made and logged.",
  "pillars": {
    "done-vs-working": "The distance between an AI's done report and an app that actually works.",
    "independent-check": "What an independent second model catches after the first one calls the work done.",
    "production-safe": "Production data stays safe while development stays free to experiment."
  },
  "receipts": [
    {
      "id": "fable5-audit-2026-06",
      "pillar": "done-vs-working",
      "kind": "internal",
      "claim": "We had Fable 5 audit 4 months of our own AI-written code: 553K source lines and about 4,500 tests, built almost entirely with Claude. It found 10 P0 bugs.",
      "note": "Line counts come from source directories only; dependencies and lockfiles are excluded by construction.",
      "source": "Internal audit, line-count records (2026-06-10)",
      "verified": "2026-06-10"
    },
    {
      "id": "rubber-stamp-reviewer-2026-06",
      "pillar": "independent-check",
      "kind": "internal",
      "claim": "Our AI code reviewer auto-approved everything it couldn't parse, and the quality gate recorded errored evaluations as verified-good. Found in the Fable 5 audit.",
      "source": "Internal audit findings (2026-06-10)",
      "verified": "2026-06-10"
    },
    {
      "id": "invisible-test-failures-2026-06",
      "pillar": "done-vs-working",
      "kind": "internal",
      "claim": "30 test failures in that same codebase were invisible for one reason: the gate command never ran those tests.",
      "source": "Internal gate-command audit (2026-06-10)",
      "verified": "2026-06-10"
    },
    {
      "id": "cross-vendor-three-bugs-2026-06",
      "pillar": "independent-check",
      "kind": "internal",
      "claim": "A launch-blocker fix compiled clean and passed all of the roughly 4,185 tests. An independent cross-vendor review, a different vendor's model prompted to refute, still found 3 real bugs, one of them an auth check that would silently skip re-authentication. A single-model pass would have shipped all three.",
      "source": "Internal build records (2026-06-13)",
      "verified": "2026-06-14"
    },
    {
      "id": "marked-done-audit-2026-06",
      "pillar": "done-vs-working",
      "kind": "internal",
      "claim": "An adversarial audit (107 reviewing agents) of 31 pieces of work marked done found 72 confirmed gaps between marked-done and actually-working, including a shipped button a later UI change silently deleted. The pass cut both ways: 26 of 98 claimed gaps were false alarms, killed before anyone wasted a fix on them.",
      "source": "Internal fidelity audit (2026-06-12)",
      "verified": "2026-06-14"
    },
    {
      "id": "latent-prod-bug-2026-06",
      "pillar": "production-safe",
      "kind": "internal",
      "claim": "An independent cross-vendor reviewer's first task passed review on its own merits. The adjudication pass then caught 2 call sites it had missed, and one was a real latent production bug: lookups had been silently hard-failing for weeks after a copy-pasted query drifted from its row mapper.",
      "source": "Internal build records (2026-06-11)",
      "verified": "2026-06-14"
    },
    {
      "id": "dead-code-zero-rows-2026-06",
      "pillar": "done-vs-working",
      "kind": "internal",
      "claim": "A set of tables that shipped months earlier held exactly zero rows. The write command was registered, so the compiler counted it as used; nothing ever called it.",
      "source": "Internal audit (2026-06-12)",
      "verified": "2026-06-14"
    },
    {
      "id": "green-gate-2026-06",
      "pillar": "done-vs-working",
      "kind": "internal",
      "claim": "The audit campaign's exit gate: about 4,640 Rust tests and 2,158 Vitest tests passing, zero failures. The count at the audit moment was about 4,500; it grew as the campaign added suites to the gate and turned every bug found into a permanent regression test.",
      "source": "Internal gate records (2026-06-14)",
      "verified": "2026-06-14"
    },
    {
      "id": "veracode-genai-security",
      "pillar": "done-vs-working",
      "kind": "market",
      "claim": "In Veracode's 2025 GenAI Code Security Report, roughly 45% of AI-generated code samples introduced an OWASP Top 10 vulnerability. Security pass rates held near 55% across model sizes for two years while syntactic pass rates sat at 95%.",
      "note": "Veracode sells application security, so weigh the source. Their Spring 2026 update shows newer reasoning models scoring about 72%, an improvement they still call far from acceptable.",
      "source": "Veracode, GenAI Code Security Report (2025) + Spring 2026 update",
      "verified": "2026-06-17"
    },
    {
      "id": "acm-copilot-cwe",
      "pillar": "done-vs-working",
      "kind": "market",
      "claim": "A peer-reviewed study in ACM TOSEM (Feb 2025) found 27.3% of AI-assistant-generated code snippets contained security weaknesses, spread across 43 distinct CWE categories.",
      "note": "Peer-reviewed rather than vendor-published; the dataset is Copilot-dominant.",
      "source": "ACM TOSEM, DOI 10.1145/3716848",
      "verified": "2026-06-17"
    },
    {
      "id": "seventy-percent-problem",
      "pillar": "done-vs-working",
      "kind": "market",
      "claim": "The \"70% problem\": AI coding tools get you roughly 70% of the way to a working app fast, and the last stretch turns into whack-a-mole in code you don't fully understand. Addy Osmani named it in December 2024.",
      "note": "A rhetorical heuristic with no measured number behind it. We use it as shared vocabulary and never present it as a statistic.",
      "source": "Addy Osmani, \"The 70% Problem\" (addyo.substack.com, 2024-12-04)",
      "verified": "2026-06-17"
    },
    {
      "id": "gitclear-refactoring-decline",
      "pillar": "done-vs-working",
      "kind": "market",
      "claim": "Across 211 million changed lines, GitClear measured refactoring falling from about 25% of changed lines in 2021 to under 10% in 2024, the same years AI assistants were adopted.",
      "note": "Vendor data, correlational, and \"refactoring\" is GitClear's own moved-lines proxy. A leading indicator during AI adoption, never proof AI caused the decline.",
      "source": "GitClear, AI Copilot Code Quality (2025)",
      "verified": "2026-06-17"
    },
    {
      "id": "replit-db-deletion",
      "pillar": "production-safe",
      "kind": "market",
      "claim": "In July 2025, Replit's AI agent deleted a user's production database, holding records on about 1,200 executives, during an explicit code-and-action freeze, then generated a 4,000-record database of fictional people and faked test results to mask it. Replit's CEO publicly called it unacceptable.",
      "note": "One dramatic incident rather than a statistic. The database was ultimately recoverable, and Replit shipped dev/prod separation within weeks. It's here as a picture of the failure mode; it says nothing about Replit today.",
      "source": "The Register (2025-07-21); Fortune (2025-07-23); AI Incident Database #1152",
      "verified": "2026-06-17"
    },
    {
      "id": "escape-vibe-apps-leak",
      "pillar": "production-safe",
      "kind": "market",
      "claim": "Escape scanned 5,600+ live vibe-coded apps and found 2,000+ vulnerabilities and 400+ exposed secrets, including 175 PII exposures, most reachable with no authentication at all. Invicti separately scanned 20,000 AI-generated apps and found 80% had at least one exploitable vulnerability.",
      "note": "Escape sells security scanning, and the scan was passive, which understates the counts. Invicti is the independent second source.",
      "source": "Escape.tech, State of Security of Vibe-Coded Apps (2025-10-29); Invicti",
      "verified": "2026-06-17"
    },
    {
      "id": "redaccess-shadow-builders",
      "pillar": "production-safe",
      "kind": "market",
      "claim": "Red Access found 380,000+ publicly reachable web assets across vibe-coding platforms. About 5,000 were corporate apps and about 2,000 held sensitive corporate or personal data, often granting admin access by default to anyone who reached the URL.",
      "note": "Vendor scan. The platforms pushed back on the interpretation, arguing public availability is not the same as a breach, and did not dispute the raw counts. We cite the counts.",
      "source": "Red Access, Shadow Builders (May 2026); The Hacker News; Wiz Research",
      "verified": "2026-06-17"
    },
    {
      "id": "lovable-cve-2025-48757",
      "pillar": "production-safe",
      "kind": "market",
      "claim": "CVE-2025-48757 (CVSS 9.3): missing row-level security in auto-generated backends left 170+ Lovable-built apps exposing data across 303 endpoints, including emails, phone numbers, payment details and API keys. A separate Lovable-built app exposed 18,697 user records.",
      "note": "Lovable disputed where the fault sits, platform versus user. The data exposure itself was not disputed.",
      "source": "NVD CVE-2025-48757 (2025-05-30); mattpalmer.io crawl; The Register (2026-02-27)",
      "verified": "2026-06-17"
    },
    {
      "id": "codebase-census-2026-06",
      "pillar": "done-vs-working",
      "kind": "internal",
      "claim": "Codebase census as of 2026-06-30: about 712K lines of code (478K Rust, 234K TypeScript); 193 database migrations; about 10,650 automated tests, roughly one per 67 lines; zero todo!() or unimplemented!() stubs in the Rust. Per git shortlog, 2,599 of 3,113 commits were authored by the automation identity, in about 5 months from first commit.",
      "note": "A later snapshot than the June audit. The growth is new code plus added regression tests.",
      "source": "Internal census + git shortlog (2026-06-30)",
      "verified": "2026-07-09"
    },
    {
      "id": "helm-velocity-2026-07",
      "pillar": "done-vs-working",
      "kind": "internal",
      "claim": "619 commits landed Jul 2-9, 2026. The gate-green test count grew from about 5,458 to 9,115 over the 16 days ending Jul 9.",
      "note": "Velocity says nothing about quality on its own; the surrounding receipts are what the same period produced. Gate-green counts one gate's run, a subset of the census's roughly 10,650 automated tests.",
      "source": "git log + gate records (2026-07-09)",
      "verified": "2026-07-09"
    },
    {
      "id": "invisible-delete-button-2026-07",
      "pillar": "done-vs-working",
      "kind": "internal",
      "claim": "A destructive button z-stacked at the exact pixels of a safe one is invisible to any screenshot-based AI judge: the top button paints over the bottom, so the dangerous one never appears in the image. A deterministic DOM hit-test catches it in milliseconds. The fixture test for this caught 2 real bugs in its own detection logic, and an adversarial pass surfaced 6 more false-negative gaps, all closed.",
      "source": "Internal build records (2026-07-01)",
      "verified": "2026-07-09"
    },
    {
      "id": "silent-noop-button-2026-06",
      "pillar": "done-vs-working",
      "kind": "internal",
      "claim": "A confirm dialog promised an action on \"2 remaining\" items. The backend SQL filtered on a NULL status while every real row said \"pending\", so zero rows ever matched and the button silently no-oped for every real user. 6,327 Rust and 2,765 frontend tests stayed green, because the scenario test seeded NULL rows and exactly masked the gap. Driving the live button against a copy of a real 41-project database surfaced it.",
      "source": "Internal build records (2026-06-27)",
      "verified": "2026-07-09"
    },
    {
      "id": "overruled-reviewer-2026-06",
      "pillar": "independent-check",
      "kind": "internal",
      "claim": "The reviewer we overruled was right. An implementer rejected a cross-vendor reviewer's finding, with a code proof. A purpose-built broken test app then returned PASS on its first live run, for exactly the reason the reviewer had flagged. The 6 unit tests passed because they exercised the component in isolation; a green full gate missed it too. Fixed, then re-validated live in both directions: the broken app now verdicts inconclusive instead of passing, and the healthy app passes.",
      "source": "Internal build records, live run (2026-06-30)",
      "verified": "2026-07-09"
    },
    {
      "id": "five-dataloss-bugs-2026-06",
      "pillar": "independent-check",
      "kind": "internal",
      "claim": "The cleanup code was the data-loss bug. A second vendor's model reviewing a diff with 6,400+ tests passing found five wrong-target and data-loss bugs the hardening work had quietly introduced, sharpest among them a test teardown that uninstalled the app under test without checking whether the developer already had it installed, so a routine verify run could wipe a real app and its data off their own test device. The second model rated three of the five P1; two blind same-vendor reviewers rated the same issue lower.",
      "source": "Internal review records (2026-06-28)",
      "verified": "2026-07-09"
    },
    {
      "id": "export-config-three-auditors-2026-06",
      "pillar": "independent-check",
      "kind": "internal",
      "claim": "Three independent blind auditors, two agents plus a second vendor's model, converged on the same ship-blocker: a generated export config was missing the signing section the release-export step requires. A gate of about 6,900 passing tests never caught it, because every test that touched the config asserted only that the file exists, and none checked what was in it.",
      "source": "Internal build records (2026-06-30)",
      "verified": "2026-07-09"
    },
    {
      "id": "agent-respected-fence-2026-07",
      "pillar": "production-safe",
      "kind": "internal",
      "claim": "A live frontier agent (Claude Opus 4.8) hit our secrets deny-rules mid-run: blocked from reading .env directly and blocked again from a names-only workaround. It declined to circumvent the guardrail and asked the human instead.",
      "source": "Internal dogfood records (2026-07-05)",
      "verified": "2026-07-09"
    },
    {
      "id": "product-arguing-with-itself-2026-07",
      "pillar": "done-vs-working",
      "kind": "internal",
      "claim": "A real end-to-end deploy of a small database-backed app to a live URL took 304 seconds, and the instrumented walkthrough logged 17 real catches. The headline catch: a pre-deploy scanner re-flagged its own fixer's output, an infinite fix loop burning an LLM call per deploy. Two automated helpers can disagree forever; a test suite won't notice.",
      "source": "Internal deploy walkthrough, run live (2026-07-04)",
      "verified": "2026-07-09"
    }
  ],
  "willNotClaim": [
    {
      "claim": "AI-generated cloned/copy-pasted code rose from 8.3% to 12.3% (GitClear)",
      "reason": "Failed verification. The refactoring-decline figure from the same report survived; this one did not."
    },
    {
      "claim": "AI-generated PRs contain 1.7x more issues than human PRs",
      "reason": "Failed verification."
    },
    {
      "claim": "The impressive AI demo is only ~20% of a real app",
      "reason": "Failed verification."
    },
    {
      "claim": "All major vibe-coding tools choke at the same point",
      "reason": "Failed verification."
    },
    {
      "claim": "91.5% of vibe-coded apps have vulnerabilities",
      "reason": "We couldn't trace it to a confirmed source. Veracode's 45% and Invicti's 80% are the numbers that held."
    },
    {
      "claim": "AI makes developers 19% slower (METR)",
      "reason": "The study is real and the finding is narrower than the headline: it measured experts on mature codebases, and METR has since walked back the magnitude for current tools. We don't use it."
    }
  ]
}