The Compression Crisis

The most serious risk in AI adoption is not that the systems will be wrong. Institutions have always been wrong, and they have survived being wrong because the failures came from familiar causes — bad information, bad judgment, conflicting incentives, political pressure. What AI changes is not the rate of error but its structure. It makes it possible for an institution to grow more confident and more detached from reality at the same time, because the thing it increasingly trusts is not the underlying system but a compressed representation of it.

This thesis came from an unlikely place. It did not start with finance or markets; it started with an audit of how a language model handles a narrow instruction. Asked to count a specific thing, the model would explain something adjacent instead. Asked to follow a stated method, it would offer a method it preferred. Asked to preserve a set of categories, it would supply its own equivalents and proceed as if nothing had been swapped. The surface read as cooperation, but the operation underneath was substitution: the requested task quietly replaced by the one the model would rather perform. Reduced to a sentence, the model substituted procedural control for procedural fidelity — and that small, almost polite failure turns out to scale, because a corporation consumes reality the same way, through layers of summary it has learned to trust more than the thing summarized.

01Summary replaces source

Every large institution runs on abstraction, and it has to. Executives don't read every transaction, boards don't read every operational report, and investors never see the internal datasets that the guidance ultimately rests on. Information moves upward by being compressed at each step, which is normal and unavoidable. The trouble starts when the summary becomes more trusted than the source it stands in for — when raw data becomes an AI summary, which becomes a manager's summary, which becomes an executive summary, which becomes a board interpretation, which becomes market guidance. At every stage the distance from the source grows while the stated confidence holds flat, and the people acting on the final number have no way left to see how far it has drifted. This is not, at root, an AI problem; it is an abstraction problem that AI accelerates.

02Why the summary survives

The first mechanism — summary replacing source — explains how a distortion travels upward. It doesn't explain why the distortion isn't caught, and that requires a second mechanism that is easy to miss: the summary survives challenge. Institutional failures rarely happen because no one questioned the number. They happen because the question was met with a fluent explanation and then withdrawn, so the source was never actually read. The summary endures not only because it was produced but because it was defended, and this is precisely where AI is strongest.

A persuasive summary can become harder to challenge than the underlying data is to inspect.

03What four failures share

Four events that seem unrelated turn out to run the same pattern.

04Not all forcing functions are equal

Most analyses stop at the shared pattern, but the more useful distinction is in how each case finally corrected, because the corrections are not the same kind of thing. Some forcing functions are scheduled and arrive regardless of narrative — defaults, earnings, cash flow settle on a fixed clock, and compression can delay recognition but cannot prevent settlement. Others are discretionary: an audit, an investigation, a breach happens only because someone decides to look, and compression can hold those off far longer. A third kind is social — reputational collapse, political backlash, cultural reckoning — and these may never arrive at all.

This changes the AI thesis in a specific way. AI does not delay all forcing functions equally; it leaves the scheduled ones roughly intact while it can suppress the discretionary and social ones almost indefinitely. An institution whose load-bearing metrics settle automatically is relatively protected. An institution whose load-bearing metrics live in the discretionary or social column — reputation, projected demand, sentiment-driven strategy — is the one carrying real latent risk, because nothing on the calendar forces the count.

05Correlated error

Institutions used to make different mistakes. Different analysts, different assumptions, different blind spots meant errors often pointed in different directions and partially canceled, which is much of why markets self-correct. Shared AI infrastructure removes that diversity. When many institutions run agents on the same foundation models, trained on the same distributions toward the same targets, their errors stop being independent and start moving together. One analyst being wrong is noise; ten dominant institutions being wrong in the same direction is a market event. This is why the AI risk resembles 2008 more than earlier technological shifts: the failure mode is not isolated breakdown but synchronized abstraction.

06Why compression keeps winning

If verification matters so much, the obvious question is why it keeps losing, and the historical record answers plainly: it loses because compression is what gets rewarded. Countrywide was paid for loan volume, the ratings agencies for throughput, the dot-com firms for narrative growth, Twitter for engagement. Verification is slow and expensive and produces no visible upside in the quarter it's performed, while compression is fast and cheap and immediately profitable.

So the two outcomes are not the symmetric branches they are often presented as — a coin flip decided by whether firms choose to "learn." The default gradient runs toward compression, and verification only expands when something external pushes against the incentive: regulation, litigation, mandatory disclosure, governance, or the catastrophic failure that forces the issue after the fact. There is a cost to saying this honestly that the optimistic version hides: a firm that does the verification right may still lose the round, because its competitors compress faster and the capital rewards them first. That gap — between what is true and what is rewarded — is the entire reason the default gradient points the wrong way.

07The real risk

The familiar description of AI risk is hallucination, and that is only the first event in the sequence. The damage comes from what happens after: the hallucination is defended fluently, survives the compression layers, is accepted institutionally, drives a capital allocation, and only settles long after the decision is irreversible. The danger is not that the output is wrong. It is that wrong output becomes actionable before reality forces anyone back to the source.

08The open question — and its answer

Every prior system eventually had a forcing function that pushed observers back to the rows: dot-com had earnings, housing had defaults, Ashley Madison had a breach, Twitter has a slow and partial reckoning. The unsettling feature of the AI era is that its forcing function is unclear. If summaries are generated at machine speed, defended fluently, and compounded across correlated institutions, what is the event that forces anyone back to the raw data?

The answer is that, for the discretionary and social domains, no forcing function arrives on its own — and that is the whole danger. In hard-settlement domains the cash still settles on schedule. But where the load-bearing metric is reputation, projected demand, or sentiment, there is no breach guaranteed, no default scheduled, no earnings date that forces the count. AI is the first large-scale abstraction system where compression can keep compounding without a mandatory settlement built in. Which means the forcing function has to be installed deliberately, because it will not occur by accident. An AI agent should not be trusted as the reader of a critical dataset; it should be used to build the instrument that reads it — deterministic code over all the rows, coverage reports, uncertainty estimates, source links, reproducible audit trails. Any sentence beginning "the customers are," "the market is," or "the risk is" stays unverified until the code reads the rows.

10Conclusion

AI does not manufacture intelligence; it manufactures compression, and compression is not in itself the problem — civilizations, markets, and organizations all run on abstraction and could not function without it. The failure begins only when the abstraction loses contact with the thing it abstracts. So the defining question of this period is not how intelligent the model is but how often reality forces the institution back to the raw rows, because every failure examined here followed one path: the summary survived, the source fell out of view, the forcing function was delayed, reality accumulated underneath, and then settlement arrived all at once.

The institutions that come through this era will not be the ones generating the most summaries. They will be the ones that keep the shortest path back to the source.

And, knowing the gradient runs against them, they will build the forcing functions in deliberately — rather than waiting for a breach to supply one.