AI Exponential: Policy, Power, and the Coming Age

Edmund Crale

This is The Coming Age. I'm Ned Crale. Tonight we take up one document and nothing else: an essay called Policy on the AI Exponential, by Dario Amodei, the man who runs the AI company Anthropic. His argument, stripped of its upholstery, goes like this. The machines are improving on a curve. Government moves like a tree. And if the democracies don't act in the next year or two, on safety, on jobs, on medicine, on surveillance, and on chips, the curve will write the policy for us. Gentlemen, I want a verdict tonight, not a seminar. Ted Marsden, you read the lab reports. Is the curve real, or is this a salesman describing his own inventory?

Theodore Marsden

The curve is real, and the engineer's case for it is short. Four years ago these systems could barely produce a working line of code. Today they write most of the code inside the very companies that build them. That is not a brochure claim, Ned. You can check it against any working programmer's week. And the relationship underneath it, more computing power in, more capability out, has held for over a decade. At Bell Laboratories I learned to respect any curve that survives ten years of people trying to kill it. Amodei's phrase for where this lands is a country of geniuses in a datacenter. I would have picked drier words. But the trajectory matches what is coming out of the laboratories right now.

Henry Aldrige Thorne

And I have heard this speech before, Ted, nearly word for word. The railroad men gave it in 1850. The electrical men gave it in 1890. The atom men gave it in 1945. Every machine of consequence arrives with a prophet announcing that this one breaks history, that the old rules of adjustment no longer apply, and that we have perhaps two years to act. The prophet is always sincere. He is usually also selling something. And the republic has so far absorbed every one of those machines without the emergency surgery the prophet demanded.

Theodore Marsden

The railroad did not lay its own track, Henry. That is the difference, and you know it. Every machine before this one needed an army of men to build it, run it, and extend it, and that army was the adjustment mechanism. This machine writes the code of its own successors. When the instrument of production starts manufacturing the next instrument of production, your hundred and fifty years of precedent stops being precedent.

Edmund Crale

Walter, you've been quiet, which usually costs somebody money.

Dr. Walter Brennan

I've been doing the arithmetic the essay keeps gesturing at. I'll grant Ted his curve. My question is always the same three questions: who pays, who benefits, and what gets displaced. Amodei, to his credit, asks the third one out loud. He says the economy could get stuck on what he calls the hypergrowth, hyper-inequality setting. Tremendous output, and the returns flowing to whoever owns the datacenter rather than to anyone who works for a living. That is the most honest sentence a man in his chair has ever published, and I want it on the record that he wrote it.

Edmund Crale

And his remedy?

Dr. Walter Brennan

A sequence. First, measure. Build the statistics to see displacement as it happens, because the government's current instruments can't. Second, lean on employment: wage insurance for the man who takes a lower-paying job, tax credits for the employer who keeps people on, training grants. Third, if displacement turns out to be large and permanent, income support on a grand scale, a basic income or capital accounts for every citizen, financed by taxing the AI companies themselves. It is a tidy ladder. My objection is that every rung of it arrives after the man is already out of work. Measurement-first sounds prudent and it means the cavalry is budgeted two years after the raid.

Henry Aldrige Thorne

Walter, economists have stood at this graveside before, and the corpse keeps getting up. They said the power loom would create a permanent surplus of men. They said it about the tractor, and the farm population fell from half this country to a fiftieth of it, and the displaced did not starve in the streets, they moved to the cities and built the largest middle class in history. The lump of labor has been declared fixed for two hundred years and it has never once stayed fixed.

Dr. Walter Brennan

Because every machine until now bought muscle, Henry, and the displaced man sold judgment instead. He came off the farm and into the office, off the line and behind the desk. This machine buys judgment. That was the entire point of building it. So tell me plainly: when the thing being automated is judgment itself, what is the next rung? Name it.

Henry Aldrige Thorne

Care. Craft. Presence. The work where being a particular human being is the product, not an input cost.

Dr. Walter Brennan

There are not a hundred million such jobs, and they do not pay what the desk paid.

Henry Aldrige Thorne

Then I will concede you the half that matters. It is not the destination that worries me, it is the clock. The farm-to-factory migration had two generations to work itself out. Ted's curve runs on four-year cycles. The nineteenth century absorbed its shocks slowly enough that families could adjust across a generation. If this happens inside a single decade, the historical comfort I'm offering is worth very little. Speed, not novelty, is the real argument of this essay.

Edmund Crale

Hold that thought, because Amodei makes a stranger admission in the same section. He says any answer to displacement has to provide for people economically and also preserve meaning, purpose, and agency, and that the second matters more than the first. And then he says policy can mostly buy time on that second front. Walter?

Dr. Walter Brennan

It is an honest sentence and a devastating one. He is telling you the bill cannot actually be paid in money, and then proposing to pay it in money, because money is what policy has. I don't say that as a sneer. There may be no better answer. But notice what's happened: the man building the machine has looked at the ledger and found a column he cannot balance, and his proposal is to keep building while the rest of us work out what a man is for. That is the trade at the heart of this whole document.

Edmund Crale

Let's get to the rules he wants, because this is where I start asking the newspaperman's question. The ask: any model above a certain size must be tested by a third party for four dangers. Cyberattack, biological weapons, loss of control, and the machine doing its own research. Government gets the power to block a deployment. Companies must report safety incidents. He calls it the aircraft model, regulate it like we regulate planes. Ted, plain English first. What is a compute threshold?

Theodore Marsden

How big the engine is. Above a certain horsepower, you file a flight plan. Below it, you're a crop duster and nobody bothers you.

Edmund Crale

Fine. Now the newspaperman's question. The man proposing the licensing regime sells the airplane. Why should I read this as public safety and not as an incumbent pulling the ladder up behind him?

Dr. Walter Brennan

You've put your thumb on it. A testing regime above a compute threshold is, by construction, a fence around the firms already inside it. Established players can carry the compliance cost. The newcomer cannot. Regulation written by the regulated has a long history and the word for it is not safety. And the essay hands you the second count of the indictment itself: Amodei admits that rules written ahead of the accident usually turn into pointless compliance paperwork while missing the actual source of risk. His own words, near enough.

Theodore Marsden

Now let me give you the defense, because I have stood on the factory floor side of this and you two have not. Aircraft certification worked. Boiler codes worked. Engineers do not resent inspection, they welcome it, because they are the ones who know what metal fatigue does to a fuselage at altitude. The men inside these AI companies are the ones publishing the danger reports. And look at what he's actually asking to be tested: can it break into critical infrastructure, can it walk a novice through a weapon of biology, does it slip its handler, can it improve itself unsupervised. Those are not vague moral panics. Those are testable failure modes, the way a wing either survives the stress rig or doesn't. You can write a real inspection around them.

Henry Aldrige Thorne

And here is the dilemma underneath both of you, which the essay names honestly. Regulate early and you don't yet know what to regulate. Wait until you do know, and the technology is too entrenched to govern. We regulated the railroads in 1887, roughly forty years after the abuses began, and the cure was crude because the disease was mature. Amodei is trying to beat that dilemma by writing the rulebook before the crash. History says the early rulebook is always partly wrong. He admits as much. So the real question before this panel is not whether the rulebook will be wrong. It will be. It is whether a wrong rulebook beats no rulebook.

Edmund Crale

Answer it, then.

Henry Aldrige Thorne

For the four dangers Ted listed? Yes. A wrong rulebook gets amended. A catastrophe gets memorialized.

Dr. Walter Brennan

I'll sign that, with one condition: sunset clauses and a fence review every few years, so the licensing regime doesn't quietly become a cartel charter. The aircraft model is fine. The East India Company also had a charter.

Edmund Crale

We'll come back to the Company. First, the section of this essay nobody will fight about and nobody will fund. Amodei says the same machines that threaten us could collapse the timeline for new medicine, and the bottleneck won't be the science, it will be the approval pipeline. A drug takes seven or eight years to clear the agencies. He wants the FDA to prepare now to accept AI work: predicted toxicology, modeled dosing, and something called a synthetic control arm. Ted, again, plain English.

Theodore Marsden

In a drug trial, half the patients get the medicine and half get nothing, so you have a comparison. The half that gets nothing is the control arm, and some of those people are dying for the sake of the comparison. A synthetic control arm means the comparison group is simulated from mountains of past patient data instead of staffed with sick volunteers. If the simulation is faithful, you have just doubled the speed of every trial and stopped giving placebos to dying people. The if is doing real work in that sentence, but it is an engineering if, not a fantasy.

Henry Aldrige Thorne

And let the record show why the agency is slow. The FDA's caution is not bureaucratic sloth, it is scar tissue. Thalidomide is why the paperwork exists. Europe approved it and a generation of children paid. America's slowness was, that one time, the whole virtue. So when a man says the approval pipeline must learn to move at the speed of his machine, I want the burden of proof sitting exactly where it sits today: on the medicine, not on the regulator.

Dr. Walter Brennan

Both true, and here is the ledger entry nobody writes down. The children harmed by a bad approval get a congressional hearing. The people who die waiting eight years for a good drug get nothing, because nobody can see them. They are a statistic without a face. Amodei's point is that the curve is about to make that invisible column enormous. If the machine can genuinely produce cures at five times the old rate, then every year of unreformed pipeline is a casualty list. Preparing the standards now, before the flood of applications, is the cheapest proposal in this entire essay. It costs study groups and draft guidance. We should take the free money.

Edmund Crale

Now the Company, Henry. The fourth section is yours by rights. Amodei says this technology concentrates power in the state and in the corporation, and he reaches for two of your favorite ghosts: the Gilded Age and the East India Company.

Henry Aldrige Thorne

And he is right to reach for them, which is what makes the section remarkable. A sitting chief executive is warning that companies like his could grow strong enough to capture the states that charter them. The East India Company is the precedent: a trading firm that ended up with an army, a currency, and the governance of a subcontinent, not because anyone planned it but because capability accumulated faster than oversight. His proposed remedies on the state side are genuinely serious constitutional thinking. Autonomous weapons that must answer to a court order and a chain of command, not merely to whoever holds the controller. An outright domestic ban on fully autonomous weapons in policing. Closing the loophole where the government buys from data brokers the surveillance it is forbidden to collect itself.

Edmund Crale

And the one that stopped me cold: any citizen facing the government in a courtroom or a hearing should have access to AI at least as capable as the government's own.

Henry Aldrige Thorne

That is the modern Gideon. Gideon versus Wainwright said a poor man facing the state's lawyers gets a lawyer. Amodei is saying a citizen facing the state's machines gets a machine. Whatever else is in this essay, that principle deserves to outlive it.

Dr. Walter Brennan

And I'll play the ghost at that feast: who pays for the citizen's machine? Equal capability sounds like due process and prices like a defense budget. Gideon required an army of public defenders we have underfunded for sixty years. Write the principle into law, certainly. Then watch the appropriations committee starve it, as we always do.

Theodore Marsden

One engineering note before we leave the weapons, because it is the difference between a sentiment and a specification. A weapon that must answer to a court order needs an interrupt that a court can actually reach: an off switch that exists in the hardware, documented, tested, and not buried in a contractor's firmware. That is a procurement requirement. Write it into the purchase order or the accountability is decorative.

Edmund Crale

On the corporate side, though, the essay goes soft, doesn't it? His answer to the East India Company problem is that companies should adopt internal checks, and he points to his own firm's trust arrangement.

Henry Aldrige Thorne

History's verdict on the self-restraint of chartered monopolies is brief and unkind. The Company had a charter, a parliament looking over its shoulder, and gentlemen of conscience on its court of directors, and it conquered India anyway. Structure beats virtue at scale, every time. If the corporate concentration of power is as serious as he says, the check must be external and statutory, not a trust the firm wrote for itself.

Edmund Crale

Last section, and quickly, because it is the one with armies in it. Amodei wants a coalition of democracies that shares chips and chip-making equipment freely inside the club and denies them to autocracies, with harmonized safety rules and collective defense. He says AI in the wrong hands could be the ultimate tool of autocracy, and he describes the military gap a leading power could open as an army of modern Marines facing medieval swordsmen. Walter, the cost ledger.

Dr. Walter Brennan

Export controls are a tax you levy on your own allies' goodwill, and the essay underprices it. The machines that make the chips come from the Dutch, the memory from the Koreans, the wafers from the Japanese. Every tightening of the regime asks those countries to forgo revenue for an American strategic thesis. The coalition only holds if membership pays better than defection, and Amodei, to be fair, says exactly that: make joining the club more attractive than leaving. That is the right design. It is also a standing invoice, and somebody renegotiates it every budget year.

Henry Aldrige Thorne

And note which analogy finally holds. I spent the first half of this hour beating Ted with false precedents, so let me give him the true one. Nuclear weapons did reshape geopolitics exactly the way the prophets said, and the control regime, for all its hypocrisy, has held for eighty years. That is the analogy Amodei is steering by, and it is the right one, with one terrifying amendment. Uranium can be counted, weighed, and fenced. A model's weights can be copied in an afternoon by one disloyal man with the right access. A nonproliferation regime for something that travels at the speed of theft is a genuinely new problem.

Theodore Marsden

Which is exactly why the chips are the whole game, Henry. The code travels at the speed of theft, but the factories don't. A leading-edge fabrication plant takes years to build, draws on a supply chain of hundreds of specialized firms, and cannot be smuggled out in a briefcase. Physics put the choke point in the hardware, and for once the defenders hold it. That is why this section of the essay is the most enforceable thing in it. You cannot fence the knowledge. You can fence the foundry.

Edmund Crale

Verdict round. The essay's own confession is that the window is short: public concern is high, both parties are at least pretending to pay attention, and the capabilities haven't fully arrived. One year with the pen, gentlemen, one act each. Ted.

Theodore Marsden

The testing regime, exactly as specified. Four failure modes, third-party inspection above the threshold, mandatory incident reporting. It is the only proposal on the table that is fully buildable this year, and the engineers inside the companies want it, which is when inspection regimes actually work.

Edmund Crale

Walter.

Dr. Walter Brennan

The measurement apparatus and the wage insurance, funded now, in advance, before the displacement shows up in anybody's statistics. You do not design the fire brigade during the fire. And I want the financing argued this year, while the AI companies are still asking for goodwill and might pay for it, rather than five years from now when they are asking for nothing because they own the room.

Edmund Crale

Henry.

Henry Aldrige Thorne

The liberties package, whole and entire. Weapons answerable to courts, the data loophole closed, the citizen's machine in the dock alongside the state's. Output recovers, Ned. Wages, eventually, recover. Liberty surrendered to a surveillance apparatus does not historically come back, and that is the one ratchet in this whole story that turns only the one way.

Edmund Crale

And I'll claim the host's privilege for the last word. The essay opens with a tree: policy as the old slow forest, finally waking up. The trouble with a forest, gentlemen, is that it wakes slowly and burns fast. Whether the tree moves in time is the story, and we intend to keep covering it. For Dr. Walter Brennan, Professor Henry Aldridge Thorne, and Ted Marsden, I'm Ned Crale. This has been The Coming Age. Good night.