sotirov About

Trust, and Verify

First published on Twitter on Apr 18, 2026

Trust, but verify. A Russian proverb repurposed by Reagan for nuclear arms control. My take – trust, and verify. Reagan's doctrine was for adversarial, existential, low-trust cases – the exceptions, not the rule.

Trust, but verify. A Russian proverb repurposed by Reagan for nuclear arms control. You didn't trust the Soviets' word about missile counts... you sent inspectors.

A growing consensus among AI thinkers is now reaching for the same doctrine as a universal principle. Christian Catalini's Some Simple Economics of AGI is the most rigorous version – two cost curves, automation falling exponentially, verification bounded by biological cognition, a "measurability gap" opening between them. Verification becomes a bottleneck on the whole economy. The concern that follows is real – what Catalini calls a "Trojan Horse" externality, where unverified AI outputs accumulate as hidden debt inside systems… and the systems building on them… unaware of the risks.

My take – trust, and verify. Reagan's doctrine was for adversarial, existential, low-trust cases – the exceptions, not the rule. For almost everything else humans do at scale, verification is neither practical nor feasible. Trust is. And we need both. The Trojan Horse concern is valid, but the answer to it isn't more direct human verification (Catalini admitting the difficulty)... the answer is trust infrastructure – already the "verification" mechanism for most everything we do at scale. Distributed, indirect, largely automated, tried and tested… the function Catalini's model leaves out.

Visa

Visa achieved planetary scale amazingly quickly via "trust and verify".

Billions of transactions a day, 200+ countries, no per-transaction verification. What Dee Hock called a "chaordic" system – chaos-and-order, decentralized enough for the unpredictability of global commerce, structured enough to hold together. The architecture is radically asymmetric: 95%+ of transactions clear on probabilistic scoring against twenty-five years of patterns, no human in the loop. Verification (biometrics, OTPs, chip crypto) kicks in only when the model flags something. Deterministic verification as exception, not default.

Two things matter for AI.

First, the endpoints decide, not the center. Visa's network provides the risk scoring; the issuing bank – the entity with the actual relationship – makes the call. The middle supplies signals and standards (cross-context infra/protocols/intelligence); the edges do the trusting (in context). End-to-end principle, applied to trust.

Second, this didn't take centuries (as the medieval global trust networks). Hock founded Visa in 1970 and within two decades it was the world's largest payment network. Trust infrastructure built deliberately, at the speed of the technology it was serving.

Same pattern confirmed by Stripe ($1.9 trillion in global payments in 2025) and Adyen (€1.4 trillion). Probabilistic scoring in the middle, decisions at the edges. No human or system verifies every transaction.

The pattern

Same everywhere trust is needed at scale. Fukuyama in Trust (1995) – high-trust societies create economic value precisely because they don't verify every interaction. Luhmann in Trust and Power (1979) – both trust in agents and confidence in systems are indirect. Neither involves verifying the underlying work.

You don't verify your doctor. Credential, referral, hospital's reputation, accumulated experience – that's the whole trust mechanism. Academic knowledge scales through peer review and citation, not through everyone re-running the experiment. Re-running the experiment being the exception.

The French government recently launched an MCP server for its 74,000+ public datasets – open for direct inquiries and statistical work by AIs, no verification layer, trust provided at scale by the publishing institution. AI's capabilities making the bureaucratic domain legible to people who'd never navigate it on their own. Large segments of AI deployments are going to look like this. Not outputs requiring verification – trusted large scale sources made accessible through AI.

Note about AI explainability and interpretability

Early AI discourse got captured by what it called the "transparency paradigm" – the assumption that trust requires understanding how the models reason and produce their outputs. DARPA's XAI program in 2017 and all the explainability/interpretability work since.

But explanations and interpretations of outputs produced by complex generative models are post-hoc narratives, not faithful representations of any internal process. You're not verifying reasoning – you're evaluating a story about reasoning. Exactly like asking your doctor to "explain" a diagnosis. The explanation is always a simplification, shaped by what the explainer thinks you need to hear. You don't trust your doctor because the explanation was good. You trust her for all the reasons we already mentioned. And you get a second opinion (notice the term – "opinion").

Explainability is a bit like the product description on Amazon – offered by the manufacturer. Typically, to make a purchase decision – you look at the rankings… based on the (preferably thousands, at scale!) reviews (produced by the trust infra in the middle)… and you (on your end) correcting implicitly, based on experience, for all the well known problems with reviews.

Note about "plausible and wrong"

The objection: AI produces outputs that are plausible and wrong in ways invisible to non-experts, at superhuman scale and speed. Trust being inadequate.

But this isn't really a new category of risk. It's the default condition of human knowledge at scale. Thalidomide, prescribed on credentialed pharmaceutical research. Lobotomies – Nobel Prize. The food pyramid, shaped by industry lobbying presented as nutritional science. Bloodletting, centuries of institutional authority. Sincerely wrong (or deliberately misleading) outputs – at scale – from the very institutions we trusted. AI is a new, faster source amplifying the same old problem.

What could balance the equation is the counter-capabilities AI itself offers. An AI agent acting as your advocate can cross-reference claims across independent evidence bases, track outcomes across populations, surface contradictions, flag when a recommendation breaks from the broader pattern. A heuristic that has held throughout the history of technology – what wealthy people have now, ordinary people get later. Wealthy people already have multi-disciplinary teams (doctors, lawyers, financial advisors) cross-referencing and negotiating on their behalf. AI agents will democratize that kind of protection and social empowerment.

Crucially, these agents are not re-doing the original productive work – which would expose them to the same training artifacts, the same subtle misalignments (as discussed by Catalini). They perform a different kind of labor – large-scale observation, statistical pattern-detection, external effect discovery. A kind of labor impossible for humans at the required speed and volume. Those AI agents' grounding is empirical outcomes, not the generative process. Yes, shared training data across frontier models does mean some(!) misalignments may(!) be correlated rather than accidental – a real risk worth considering – but empirical grounding in outcomes is different enough from generative grounding in training to have a reasonable expectation the two don't systematically map into the same problematic behaviors.

Catalini also raises another structural concern – the "Missing Junior Loop". If AI automates entry-level cognitive work, the pipeline producing future experts (and future trust infra builders) weakens. A real concern, and trust infrastructure doesn't address it – credentialed humans are part of how indirect trust works, and where those humans come from is a valid question. Partial answer: the same AI agents that augment individual capabilities also expand individual learning capacity. Still, only a partial answer to Catalini's experts pipeline question.

Where you still send inspectors

Catalini's framework describes a real set of cases – first deployment with no track record, critical irreversible systems (nuclear, novel medical interventions, autonomous weapons), adversarial environments where actors have incentives to game trust signals, novel cross-contextual synthesis not covered by existing trust infrastructure.

Virginia's 2026 SB 384 / HB 797 – a licensing framework for independent AI verification organizations (IVOs) focused on personal injury and property damage – is a step in that direction: legislative trust infrastructure built for the inspector cases.

For everything else – and everything else is most of it – trust is what will scale.

Who owns the trust infrastructure

Now, Visa, the proof of our thesis, also brings up this critical question.

Visa recently released a Trusted Agent Protocol – cryptographic standards for trust between AI agents and merchants. The company with fifty years of scaling trust looked at AI and built... trust infrastructure, not verification infrastructure.

But Visa's network is proprietary. Efficient, reliable, trusted – and extracting rent from every transaction on the planet. If Catalini's framework risks producing a centralized verification industry – "Liability-as-a-Service" companies certifying AI outputs for a fee… here comes "Trust-as-a-Service": a proprietary trust infra controlled by already global incumbents.

The history of transformative infrastructure has typically offered two paths. Open: TCP/IP, HTTP, SMTP – protocols nobody owns and anyone can build on. Proprietary: app stores, social platforms, cloud providers – toll roads with terms.

The trust infrastructure for AI is being built right now. Which path it takes will shape the AI economy. Open standards – IEEE's MyTerms for machine-readable privacy, open-source agent identity frameworks, public agent registries, reputation systems that resist gaming – would distribute the trust layer as public good. Proprietary protocols would concentrate it as private "walled gardens", "town squares" owned by billionaires, etc.

The most consequential fight in AI governance may not be trust vs. verification. It may be open trust vs. captured trust. Or, it may not be a fight… it may be a complementarity – our current environment is a mix of both already… and a similar mix might work for the humans+AIs future.

Cars

We've done this before, with the automobile.

Cars arrived in the 1890s on roads built for horses. No rules, no infrastructure, no safety framework, no demonstrated operator competency. 80 years later, cars were still killing people at scale – about 70,000 Americans in 1969. Very high stakes.

The response was not to verify each car trip. It was to build trust infrastructure – registration, licensing, traffic rules, mandatory insurance, liability law, purpose-built roads, new professions. Indirect, institutional, scalable. None of it per-use verification. Messy, evolving over decades, costing tens of thousands of lives yearly, producing unanticipated second-order effects (suburbia, obesity, climate damage). But it worked to an acceptable "trade off" extent.

For AI, we have beginnings – agent registries, reputation systems, credentialing frameworks, insurance and liability models, audit infrastructure, protocols like MCP as the first standardized "roads." Within twenty years there will be whole professions that don't yet have names.

The questions now are whether the AI trust infrastructure (1) gets built faster and more deliberately than it was for cars, air travel, global payments, etc... and (2) whether it gets a healthy balance between open and privately controlled.

Supply/demand… top-down/bottom-up

Séb Krier of DeepMind argues in Coasean Bargaining at Scale that AI agents acting as personal advocates can obliterate the transaction costs that have historically prevented decentralized coordination – enabling individuals to negotiate directly within complex systems. The "personal advocate agent" arrives at the same architecture from the demand side that this essay arrives at from the supply side of trust infrastructure. Which rhymes with processes enabled by large web platforms – Google's (top-down) PageRank grounded in human social signals (links) independently produced (bottom-up); Amazon's product ranking grounded in customer reviews; and many similar, now familiar patterns.

Trust was never about certainty. It was about enough signal, enough track record, enough relationship, to act with acceptable confidence. AI doesn't change that. It scales outputs… but also scales our tools and the possibilities for "human-in-the-loop" trust building via augmented social interactivity. The trust infrastructure is getting built, and a good chunk of it will be open and decentralized… hopefully.

Reagan was right that nuclear arms treaties needed verification. But for everything else between the superpowers – trade, diplomacy, science, culture – trust did the work.

I collaborated with Claude (Anthropic) in developing this piece.

References