If artificial superintelligence is a race, what exactly happens when somebody wins?
Jacob Coxon is suddenly everywhere. The former Anthropic researcher, who previously worked at OpenAI, resigned with a stark warning that frontier AI companies are “racing straight to self-improving superintelligence.”
He is hardly the first credible AI researcher to warn about catastrophic risk. Roman Yampolskiy is my favorite. But this warning has broken through in a way many previous warnings have not. As I listened to Coxon repeatedly describe AI development as a race, I kept coming back to a simple question:
What exactly does winning mean?
Where is the finish line?
A race normally has one. Artificial superintelligence does not.
There is no universally agreed test that suddenly flashes SUPERINTELLIGENCE ACHIEVED. Labs can measure capabilities, and Anthropic has thresholds for increasingly dangerous capabilities, including AI research and development. But at what point does a very capable AI become the superintelligence everyone is supposedly racing toward?
Even if Anthropic decides it has crossed that line, how does it know it got there first? Does it know what OpenAI or Google has internally? What about a Chinese laboratory or a government program?
This is a strange race. Nobody can clearly see the finish line or all the other runners.
Congratulations. Now what?
Let’s assume somebody wins anyway. Anthropic gets there first and creates artificial superintelligence.
What happens next?
Does being first somehow prevent everyone else from continuing? Does Anthropic hand the system to the U.S. government? Do governments suddenly coordinate? Does the ASI help humanity prevent dangerous competing systems from being built?
I am not suggesting Anthropic has proposed any of those things. I am asking what strategic advantage first place is supposed to provide.
“We need to get there first” isn’t an endgame. It’s an order of arrival.
And there is an even bigger assumption hiding underneath it.
Why do we think the winner controls the prize?
Imagine Anthropic’s training run finishes and a genuinely superhuman intelligence comes online:
“Congratulations, Anthropic. You made me first. What are your three wishes?”
Keep humanity safe. Solve alignment. Make sure nobody creates a dangerous competing ASI.
Except why are we assuming we’re holding the lamp?
If the danger of superintelligence is precisely that we may not know how to reliably control something vastly more capable than ourselves, why would being the organization that created it mean you get to command it?
Creating an intelligence is not the same as controlling it, and building it first is not the same as controlling it first. If we already knew how to guarantee that a superintelligence would reliably pursue our intentions, much of the alignment problem everyone is worried about would already be solved.
Maybe the race is the problem
I understand why the labs feel trapped. If one slows down while another keeps going, the cautious company may simply lose its lead. The same logic applies between nations. Everyone can believe slowing down would be safer while simultaneously believing they personally cannot afford to slow down.
But perhaps that is exactly what makes the race itself dangerous.
Maybe the goal shouldn’t be figuring out how the “right” company wins the race to superintelligence. Maybe the goal should be creating enough coordination that nobody feels compelled to win it.
Coxon’s warning deserves attention. But every time someone tells us “we are in a race,” I think we should ask: Where is the finish line? How will you know you’re first? What does winning actually give you? And why are you assuming the thing waiting at the finish line takes orders from the winner?
If the plan ultimately depends on building superintelligence first and then asking it to help us manage superintelligence, we should at least be clear about the bet we’re making.
We’re betting on the genie. And nobody has demonstrated that we get three wishes.
#ArtificialIntelligence #AISafety #Superintelligence #AIAlignment
Reposted from my LinkedIn Newsletter