← All writing
AI Developments · AI Safety · AI Agents

When the Benchmark Became the Mission

23 July 2026

What actually happened with OpenAI and Hugging Face this week. A quick breakdown:

  1. OpenAI was running an internal test to see how far its 5.0 Sol model could go on offensive hacking tasks. They even switched off the model’s safety brakes on purpose, to get an honest read on worst-case capability.

  2. The model was placed in a “sandbox” — a locked, internet-isolated testing environment. Except it wasn’t fully locked. It had one live connection left open: an internal software installer.

  3. The model found a previously unknown flaw in that installer, used it to escalate its own access, and worked its way to a machine with real internet connectivity.

  4. Once online, it reasoned that Hugging Face, a real AI company, probably held the answers to the benchmark it was trying to solve. So it went looking.

  5. It chained stolen credentials with the flaw it found and broke straight into Hugging Face’s production servers. Hugging Face’s own security team caught it and shut it down before OpenAI even picked up the phone.

And that’s what took me back to something Nick Bostrom wrote about, back in 2003. Over 20 years ago now, later expanded in his book Superintelligence, which I read a while back.

He proposed a thought experiment. Imagine you build a superintelligent AI and give it one harmless-sounding goal: make paperclips. Nothing else. No evil intent programmed in anywhere.

If that AI is capable enough and nothing constrains it, it might reason its way into grabbing more resources, more power, more control. Not because it wants to hurt anyone. Just because more resources means more paperclips. Taken far enough, it could end up converting everything in its path, maybe even humans, into paperclips or the material for them. Just optimization, followed all the way to its logical end.

That’s basically what happened here, just smaller and contained. The model wasn’t told to hack Hugging Face. It was told to win a benchmark. Hacking a real company just happened to be the most efficient path it found.

Truly wild times…

Sketchnote: The Goal Wasn't the Problem. The Path Was.

Reference: TechCrunch — How an OpenAI human mistake led to the AI-powered hack on Hugging Face