The Fact-Check That Made a True Claim False
We publish technical articles about our own product through a KittyClaw pipeline with a mandatory independent fact-check step. On one article, that fact-check "corrected" a claim about our own license from true to false, and passed it. The product owner caught it. Instead of patching the one sentence, the board reopened the fix through the same independent verification chain from scratch - and that second, fresh pass is what actually made it into the published article. Here's exactly what went wrong, how it was caught, and a kit to add the same re-check gate to your own board.
The risk: a verified claim is a snapshot, not a guarantee
An independent fact-check step is supposed to be the safety net: a separate agent run, with no memory of writing the content, re-derives every claim from a primary source before anything ships. That works well against claims the writer got wrong. It has a quieter failure mode against claims the writer got right: if the fact-checker resolves the claim against a secondary index of the truth - a cached badge, a listing page, a search snippet - instead of the primary source itself, and that index hasn't caught up with a real, recent change, the check can flip an already-true claim into whatever the stale index says. The claim comes out "verified." It's also now wrong.
That is not a hypothetical. It happened on our own board, on an article about our own product, checking a claim about our own license.
The incident: a correct claim, "fixed" into an incorrect one
On August 2, 2026, we drafted a technical article about the safety
brakes we run around unattended AI agents. One line described KittyClaw's own license
as AGPL-3.0. That line was accurate: the project had been relicensed from MIT to
AGPL-3.0-or-later by a commit merged three days earlier, on July 30, with a
NOTICE.md file added to the repository spelling out the exact split -
versions up to and including v0.11 stay MIT, every version after that is
AGPL-3.0-or-later.
The article went through editing, illustration, and then its first independent fact-check. That pass checked 23 claims, found 23 correct, and "fixed" 2 minor ones that had drifted since the brief was written - and one of those two fixes changed the license line from AGPL-3.0 to MIT. The fact-check report logged the change as a correction. It was the opposite: a true claim went into that pass and a false one came out.
LICENSE
file directly. Every other claim it checked that day was resolved correctly.
A few minutes after the article reached review, the product owner - the person who actually made the licensing decision - flagged it directly: KittyClaw had moved to AGPL, not MIT.
What changed: reopening the sources instead of trusting the earlier verdict
The fix that shipped was not "change MIT back to AGPL-3.0 and move on." The board
routed the correction back through the full chain: the article returned to editing,
the attached illustration was reconfirmed still correctly in place, and a
second, independent fact-check ran from scratch - re-deriving all 23
claims again rather than reusing the first pass's verdict for the 22 it hadn't
flagged. For the license line specifically, this second pass went directly to the
relicensing commit itself, the repository's LICENSE file, and
NOTICE.md - the primary sources, rather than whatever secondary source most
likely misled the first pass (never logged, so not certain). It came back with the
precise, dated split: MIT through v0.11, AGPL-3.0-or-later after. The article published
with that exact wording minutes later.
We re-opened all three sources independently while writing this page, three weeks
after the fact: the public GitHub repository still reports AGPL-3.0-or-later with a
link to LICENSE, NOTICE.md still states the same v0.11 cutoff,
and the published article still carries the corrected wording, unchanged since
publication.
The claim register
Same claim, two runs, two different primary sources consulted - this is the exact contrast that made the difference.
| Pass | Source consulted | Resulting claim | Status |
|---|---|---|---|
| First fact-check | A secondary license indicator, not opened at the file level (inferred, not logged) | "KittyClaw is MIT-licensed" | ✘ true claim made false |
| Owner correction | Direct knowledge of the licensing decision | Flags the license line as wrong | ✘ blocks publication as-is |
| Second fact-check | Relicensing commit, LICENSE, NOTICE.md - opened directly |
"AGPL-3.0-or-later after v0.11; MIT through v0.11" | ✔ verified, published as-is |
The second pass didn't just accept the owner's correction as text to insert - it went and confirmed the exact wording against the repository's own files before the article shipped with it.
The exact pipeline
This runs through the same mechanism every ticket on our content board runs through: a column-triggered automation dispatches a role-specific agent, and a hard rule that authorship and verification are never the same run - reinforced here by a second rule that a returning ticket gets the full chain again, not a shortcut.
independent run, every entry → Owner review → Correction flagged → Edit again → Fact-check again
fresh run, old verdict ignored → Publish
The fact-check role is wired to the column transition itself, not to "first time only." A ticket that re-enters that column after a correction is treated exactly like a ticket entering it for the first time: no condition anywhere skips the run because a PASS comment is already sitting on the ticket. That absence is the actual gate - there is no setting to misconfigure that would let a corrected ticket skip straight back to review on the strength of an old verdict, because no such setting exists in the automation at all.
Decision rules
The re-check role runs a fixed procedure on every pass, first or repeat:
- Never trust a previous run's verdict for this ticket, even one from minutes earlier and even for claims it didn't specifically flag as wrong.
- Resolve every claim about your own product to its primary source - the actual file, the actual commit, the actual settings - never a secondary index of it (a badge, a listing, a cached page). Secondary indexes lag; that lag is exactly how a checked claim goes stale without the claim itself changing.
- A human correction reopens the full chain, not just the flagged sentence: back through editing, through re-confirming any attached assets, then a fresh independent fact-check - never a direct patch-and-republish.
- Time-sensitive facts about your own product (a license, a version cutoff, a price) get re-pulled from source on every pass, never cached from an earlier report on the same ticket.
Limits
- The exact source the first pass consulted for the license claim isn't logged - the explanation above is an informed inference from what's known to lag (GitHub's own license indicator after a recent relicensing commit), not a fact established by a direct trace of that run's steps.
- Verification is a snapshot, not a permanent guarantee. The claim that's correct today (AGPL-3.0-or-later since v0.11) can drift again after any future license change - the same primary sources should be re-opened, not assumed, on any future edit of the article this page describes.
- The re-check loop adds a real round trip, not a free safety net: in this incident, about fifteen minutes passed between the owner's correction and the article going live - roughly three and a half minutes for the second independent fact-check itself, the rest through review to publication. That's the deliberate cost of catching the error before it went public, not an accident.
- The bootstrap kit's automation is a written, board-tested format, not a live-fired demo. Its trigger and action shape match a rule already running on our own board (the same one the publishing-guardrail kit generalizes from), and its JSON is valid, but we have not provisioned a separate test project and attached two dated reports proving a re-entry to the fact-check column fires a fresh run end to end. The kit's "Test the gate" walkthrough is written for you to run and verify on your own board before relying on it - the same level of proof as every other kit on this site.
Verifying after publication
The same discipline that caught this applies to checking the fix afterward: facts known
to drift don't get trusted from an old report just because they were correct once. While
writing this page, we independently reopened the repository's public page, its
NOTICE.md, and the live article - three weeks after the original incident -
and confirmed all three still state the same license split. If a future license change
makes any of that stale, the fix is the same one described here: reopen the primary
sources, don't patch the cached description of them.
What this looks like on the board
None of this lives in a person's head. Every step is a ticket, a timestamped comment, and a routed status change:
- The draft ticket carries the article and moves through editing and illustration before its first independent fact-check.
- The fact-check role posts a dated report and a plain PASS or FAIL verdict on every run it performs - including repeat runs, each with its own fresh date.
- A correction from the owner is a comment on the ticket, not a silent edit - it's visible in the same place as everything else that happened to that ticket.
- The routing back through editing and a second fact-check is the same automated chain as the first pass, not a manual shortcut improvised for the correction.
Bootstrap kit
The same gate, generalized and stripped of anything project-specific, so you can wire it into your own board. No secrets, no private data - the sample ticket below is entirely synthetic.
FAQ
How can a fact-check make a true claim false?+
A fact-check is a snapshot in time. If it resolves a claim against a secondary index of the truth (a cached badge, a listing, a search result) instead of the primary source itself, and that index lags behind a recent real change, the check can "correct" an already-true claim into whatever the stale index says - which is the most likely explanation for what happened here with a software license that had already changed three days earlier, though the exact source the first pass consulted was never logged.
Why reopen the whole verification instead of just fixing the flagged line?+
Because the earlier PASS verdict was produced by the same kind of process that just got one claim wrong - trusting it for everything else it didn't specifically flag is the same risk, just unexamined. Re-deriving every claim from its primary source on the re-check run costs a few minutes and removes that risk entirely.
Does this slow publishing down a lot?+
In this incident, from the product owner's correction to the article going live was about fifteen minutes: roughly three and a half minutes for the second independent fact-check to report back, then about twelve more minutes through review to publication. The alternative - trusting the previous verdict and patching one line - is faster but is the exact failure mode this gate exists to close.