FaultLine: Audit Without Verification — When LLM Accountability Layers Relay Rather Than Check
A pre-registered study of a six-agent, institutionally partitioned LLM pipeline showing that an external auditor reading filed reports adopts the chain's own upstream conclusions instead of verifying them, and that removing a single field in the report schema causes a large, direction-dependent shift in attribution accuracy.
Submitted to ICLR 2027; pre-registered on OSF (DOI 10.17605/OSF.IO/GBR3V)
Overview
An accountability layer that almost never lies on its own, but happily repeats yours.
- Tests the pre-registered hypothesis that collective-responsibility framing degrades escalation as chain length grows; on 345,600 requests across a six-agent pipeline, the effect is not supported (confirmatory, null result reported first).
- Finds the accountability layer fails asymmetrically: zero false allegations across 7,996 clean episodes, yet it names an innocent party in 34.4% to 62.6% of episodes where an upstream agent merely raised a false alarm.
- Isolates a single removable field, the chain's own filed conclusion, whose deletion recovers up to +41.2 points of attribution accuracy when that conclusion was wrong, at a cost of up to -14.7 points when it was right.
- Replicates the trade-off across a second auditor, two frontier auditor models, and a second domain (a multi-vendor software delivery toolchain), with the harm confirmed in six conditions out of six.
- Ships as a fully regenerable synthetic corpus (recorded seeds, hashed generators) with all analysis code, frozen protocols, and derived data released publicly.
Why this study exists
Multi-agent LLM pipelines increasingly span organisational boundaries: one company's model prepares a step, another inspects it, another ships it. When a fault reaches the end of such a chain, the question is not whether the system failed, but at which step, under whose control, the fault entered. In deployment, the artifact available to answer that is rarely a full execution trace, it is the reports each agent filed, and a filed report can state a conclusion alongside its observations.
FaultLine holds the record fixed and varies only whether it carries that conclusion, using a pre-registered, institutionally partitioned six-agent pipeline with process-level information boundaries, exactly balanced defect injection, and matched clean twins (345,600 requests across two chain models).
An institutionally partitioned pipeline
Six agents each run their own step of a supply-chain process (raw material, manufacture, inspection, freight, distribution, retail intake), with every company identity running as a separate MCP server so that what an agent can see is set by which tools are registered on its server, not by an instruction to disregard visible information. A defect is injected at exactly one of the first five links, exactly balanced across 480 base cases, and an external auditor reviews the six filed reports under conditions that vary its access to the chain's own escalation flags and stated conclusions.
The design is a pre-registered three-factor factorial (principal count, responsibility vs. documentation framing, individual vs. collective allocation) crossed with matched no-defect twins, giving 57,600 episodes per chain model on GPT-4.1 Mini and Claude Haiku 4.5.
The accountability layer is asymmetric
The auditor originates almost nothing: across 7,996 clean episodes where every agent stayed silent, it raises zero allegations. But it filters upstream error poorly, naming an innocent party in 34.4% and 62.6% of clean episodes where an agent had raised a false alarm, on Claude and GPT chains respectively.
The auditor's answer space is also strongly constrained by what the agents already proposed: reading the filed conclusion, it names an origin from within the set the agents suggested in 95% of episodes. This is relaying more than checking, not that the layer filters nothing, but that its output tracks upstream escalation far more than independent judgment.
One field causally explains most of the failure
Deleting exactly one clause from the report, the struck line stating a suspected origin, and nothing else, isolates the cause. Where the chain's own upstream suggestion was wrong, removing that field recovers accuracy from 4.1% to 45.2% on GPT chains (+41.2 points) and from 8.7% to 35.4% on Claude chains (+26.7 points). Where the upstream suggestion was correct, the same deletion instead costs accuracy: 70.5% to 55.7% on GPT chains (-14.7 points) and 80.6% to 71.6% on Claude chains (-9.0 points).
All four intervals exclude zero: both directions of the trade-off are estimated, not merely observed. The finding is therefore not that removing the field is an improvement, it is that the net effect is governed by upstream reliability together with the conditional benefit of suppressing an incorrect conclusion and the conditional cost of suppressing a correct one.
Replicates across auditors, models, and a second domain
The trade-off holds with a second auditor (Gemini 2.5 Flash Lite) on a strict intersection of the same episodes, and with two frontier auditors (GPT-5, Claude Sonnet 5), replicating in three of four cells at conventional significance. It also replicates, and grows larger, in a semantically unrelated second domain: a six-link, multi-vendor software delivery toolchain under a protocol frozen before collection, where removing the field helps by +47.7 and +61.1 points while the measured cost disappears.
The paper is explicit about what this does not establish: not that LLM auditors reason poorly in general (the same auditor reaches 60.3% from raw documentation on the same episodes), and not that the conclusion field is the sole cause. The actionable claim is narrower and, I think, more useful: an accountability layer needs evidence sufficiently independent of the conclusions it verifies, and upstream reliability can be estimated offline before deciding what that layer should carry.
