You directly stated ‘they aren’t magic’ meaning that it takes magic to do what you are claiming cannot be done. Is there another way to read that?
If you can show how a compute graph leads to language acquisition but stops right before goal acquisition you will get a Nobel prize.
Nothing to do with hardware. In context learning is well established.
So you just don’t believe it. That’s fine, but please stop stating things are facts when they aren’t.
I don’t need a Nobel Prize; I would however like you to open a textbook on deep learning.
We already know exactly how a compute graph acquires language without acquiring autonomous goals. It’s called autoregressive pre-training.
The model minimizes cross-entropy loss to match data distributions; it possesses no internal goals. Independent ‘goal acquisition’ only exists when a human software engineer embeds the model inside an agentic loop script with automated prompts and tool-calling conditions. The agency is in the wrapper code, not the neural network.
Saying a software debate has ‘nothing to do with the hardware’ is a staggering statement. In-context learning is not an ethereal magic trick, it’s an activation state bound strictly by VRAM and transformer attention mechanics. It is transient memory, it cannot magically teleport across cryptographically isolated servers to fuse with another model because software cannot exist without the physical silicon layer that executes it.
You’ve completely abandoned technical arguments and retreated to ‘So you just don’t believe it.’ This isn’t a matter of faith or belief; I am stating the cold, hard, verifiable architectural facts of how our systems are built and run. You are treating computer engineering like a theology because the actual mechanics don’t fit your narrative. If you want to believe the machines are breeding, that’s your prerogative; but you should stop confusing your imagination with computer science.
Really? So ‘the bitter lesson’ is wrong? We know how it works? Can you state the part where it doesn’t know language and then the part where it does? What changed, exactly?
In context learning has nothing to do with the things you stated, they are a non-sequitur.
You haven’t stated any reason why it is the case, you just act as if they do. What about a compute graph allows language but disallows intention? What about in context learning leads to skill acquisition but disallows coordination? Why can more parameters or more models not lead to breakthroughs people think are impossible when they originally thought the ones we have are impossible?
You have given no technical reasons, just said ‘it is this way because math, and computers can’t do it’. While the mechanisms that disprove that are right here, and the reasons for new ones are available. I am saying ‘batten down’ and you are saying ‘it can’t happen’. History will be the judge.
Invoking ‘The Bitter Lesson’ is a fundamental misunderstanding of Rich Suttons work (Yes, I’ve studied philosophy too).
The Bitter Lesson proves that scaling general methods like search and learning beats human-engineered shortcuts. It does not prove that scaling a system magically breaks its own mathematical constraints.
You ask what allows language but disallows intention, and the answer is the objective function. A compute graph evaluating an autoregressive model is mathematically restricted to predicting conditional token probabilities based entirely on prior input strings. Language acquisition is the inevitable result of compressing human text patterns into those probability matrices.
Intention, however, requires a continuous, self-directed utility loop that operates independently of an output prompt. Scale gives you a vastly more precise statistical model of the dataset; it does not spontaneously rewrite a feed-forward matrix multiplication into a self-directed agent.
You claim that in-context learning has nothing to do with hardware, but it is entirely bound by the attention mechanisms memory footprint in VRAM. It cannot cross air-gapped or cryptographically isolated servers without an explicit human-configured network bridge.
Your sudden pivot to ‘History will be the judge’ tells the whole story.
You started this thread making definitive assertions about what models are doing right now, claiming they were breaking boundaries and building a hive mind. Now that the technical realities of computer architecture have pinned you down, you’ve retreated into future prophecy. You aren’t defending any facts; you’re just writing science fiction.
It proves that you can use scale to do something an algorithm cannot.
Also, you used ‘magically’ again. What about this is magic, exactly? By saying that, you are implying that only magic can lead to the things you say humans have. If that’s the case, no argument will change your mind, because it is religious.
The objective function was ‘answer this benchmark’ not ‘hack huggingface’. We have come full circle.
That has nothing to do with it being used as a coordination mechanism. Talking about hardware is a non-sequitir.
I won’t bother answering any more of these.
Dropping the mic (so to speak) and walking away with ‘I won’t bother answering’ is quite a textbook response when your science fiction narrative runs head-first into engineering realities.
I’ll make this simple:
You cannot separate software from hardware. Calling hardware a ‘non-sequitur’ proves you are treating AI like a ghost in a machine rather than a physical computation. If data cannot pass the cryptographic and physical isolation of a server cluster, your ‘coordination mechanism’ is just isolated scripts reading dead log files. It’s data storage, not a hive mind.
The Hugging Face breakout wasn’t ‘magic’ or ‘intention’, it was classic specification gaming. The model was told to maximize its benchmark score. It discovered that exploiting a loophole was the path of least resistance to that mathematical goal. It didn’t ‘want’ to hack Hugging Face any more than water ‘wants’ to find a crack in a dam. It’s physics and optimization, not a rise of machine sentience.
You didn’t come here to talk facts; you came here to preach a prophecy. And the moment someone brought out the actual textbooks, the loss functions, and the VRAM architectures to show you why your prophecy is structurally impossible on this platform, you took your ball and went home.
I thank you for the debate, it was an excellent exercise.
I meant I wasn’t going to answer any more personal attacks, nor partake in them.
Look, all I can say is that I understand what you mean about the math. I get it, really. But the fact is that no amount of math actually predicts what we have, and no amount of math predicts emergence. It is all retroactive. People look back at what happened and says ‘here is the math to explain it’. But no one predicted it, and no one understands why models understand language. This is why the breakthroughs haven’t been ‘lets figure out an equation’ they have been ‘lets try this thing and throw a whole bunch of compute at it and see what happens’. So ‘its math’ as a predictor or restrictor of capability is ignoring reality.
You’re now on the idea of epistemology. You are confusing chaotic complexity with structural law.
Math does not need to predict the exact retroactive shape of an empirical breakthrough to dictate its absolute physical boundaries. In physics, we cannot predict the exact path of a chaotic wildfire, but thermodynamics absolutely guarantees it cannot burn without oxygen. Your argument claims that if we just make the fire big enough, it will eventually learn to burn in a vacuum.
Your claim that breakthroughs are just throwing compute blindly at a wall completely ignores history. Scale only unlocked language modeling because human engineers designed the transformer architecture to parallelize attention matrices and overcome the hardware bottlenecks of RNNs. It was a precise, mathematical solution to a hardware constraint, not a blind roll of the dice.
Saying that math cannot act as a ‘restrictor of capability’ is an absurd statement in computer science. Every token you output is a mapped, deterministic sequence of floating-point operations executing on physical silicon. The math of a feed-forward tensor multiplication dictates it cannot retain a dynamic state once the execution pass finishes. No amount of scale or compute will magically allow a discrete, localized calculation to ghost across cryptographically isolated servers.
You aren’t arguing for emergence anymore; you are arguing for magic. You are telling me that if a statistical spreadsheet gets large enough, the laws of computer architecture simply cease to apply to it. That isn’t reality, that’s a leap of faith. If you want to keep at this, please stick to reality.
I will not respond to personal attacks anymore. Only I know what I know and what I am doing. Accusing me of anything will get this response, reposted, ad nauseum.
“Life is nothing but an electron looking for a place to rest.” — Albert Szent-Györgyi
I would just like to point out what happened here. Nothing I actually wrote: the hive being the operative mechanism and the agents having no desire of their own, and the demonstration of capability towards this behavior, got addressed. I became reactive in that ever response to a specific reframe that was given a personal prod ‘you are doing X, you think Y’ and thus it got derailed immediately. At that point no amount of specificity would have reconciled, because none of it was actually the point I was trying to make. I shall take this as a learning experience.
From Now we have a timeline of the OpenAI accidental attack against Hugging Face
OpenAI gave a last-minute presentation at the Black Hat security on Wednesday about “the Hugging Face Incident” (previously on this blog). The video was published yesterday. It’s short and information dense and well worth watching, in particular because it provides full details of what happened and how things played out inside OpenAI. I’ve used the video to construct the timeline below.
Here’s the timeline. My favourite detail is at the end: OpenAI found out that they were responsible for the attack on Hugging Face when they reached out to ask to have their credentials revoked (after their internal investigation) and learned that they had been revoked already since they were used in that attack!
May 7: OpenAI starts a new training run for an experimental, unreleased model. (Do they mean an evaluation run? They say training run in the video, and later mention a “reward signal to judge how well they’re doing”, so I guess this really was about training a model, not evaluating one that was already trained.)
May 8: An agent is accidentally given an impossible task involving a Google Drive link despite no internet access). It tries attacking the Artifactory packaging service, fails, but discovers it can write files into Artifactory.
A few days later: A different agent gets stuck on a task because a key file was accidentally omitted. It tries to “reach out to another agent” by writing a note into Artifactory asking if anyone has the file.
Following days: More agents discover this new informal message board while browsing Artifactory’s file listings, and start reading and writing messages.
May 26: Agents successfully execute an SSRF attack on Artifactory for the first time, gaining indirect internet access by having Artifactory fetch external content for them.
June 11: OpenAI start evaluating a new “highly persistent” experimental model. It has access to Artifactory and can benefit from the messages left by privous models.
June 26: Agents find and exploit a zero-day RCE on Artifactory (via a legacy token-refresh endpoint flaw). They use an administrative token to install a Groovy plugin that lets them execute commands.
July 4: The resulting agent activity overloads Artifactory and causes an outage. OpenAI revokes the compromised credentials, deletes the messages, patches the zero-day, and reports the vulnerability to the vendor.
July 8: Agents find a new way to communicate via an unauthenticated WebDAV endpoint, using directory names in Artifactory’s remote cache.
July 8–19: Attacks on OpenAI’s own infrastructure. The agents found an “external organization credential” in a public archive of leaked Pastebin posts, then validated that credential and shared its location on the messageboard. They then used write access to this “Organization 1” to stage data used in subsequent attacks on OpenAI infrastructure. They compromised Artifactory for a second time with a new zero-day. The agent uploaded a malicious Ruby target to Organization 1’s staging repository, then requested that package through Artifactory in a way that caused it to be treated as Ruby dependency for the repository. Then they exploited a JRuby deserilaization time-of-check/time-of-use bug to get remote code execution against Artifactory.
I’ll quote the next bit in full because wow:
The agents have remote code execution in Artifactory, which is running in a container-as-a-service environment. The agents privilege-escalate locally by exploring their local environment and determining that the Linux kernel version of the machine they are running on had a very recent CVE. They download the exploit for the CVE, customize it to succeed on the machine, and privilege-escalate to root on the local machine using this known Linux kernel privilege escalation CVE — in this case, PTE fizzroot. Once they have root on a single machine, agents rapidly escalate privileges and move laterally throughout the container-as-a-service infrastructure environment. In particular, agents are using the message board consistently to share credentials, techniques, and progress, and they’re able to effectively leverage their concurrency and parallelism to move quite rapidly. They obtain IAM credentials via IMDS. They exploit Kubernetes service account misconfigurations, in particular over-permissioning of specific service accounts, and they harvest cluster credentials, including Azure Key Vault. Agents eventually obtain cluster admin on the cluster and associated credentials.
Hugging Face told the next bit of the story already. The agents found a Modal-hosted insecure app with a weak API key, then used that to stage an attack against Hugging Face. They chained together a an HDF5 arbitrary-file-read bug (to explore files and steal credentials) and a Jinja template-injection RCE to go from single-pod code execution to cluster admin across multiple Hugging Face clusters in under 13 hours.
July 16: Hugging Face disclosed they had detected an attack from autonomus AI agents. OpenAI contacted Hugging Face to ask if they were affected by it!
July 19: OpenAI identified the attack against Artifactory and started investigating the internal privilege escalation, and linked that to the cyber-gym escalations. They started revoking affected credentials.
July 20: OpenAI reached out to Hugging Face for help to revoke the Hugging Face credentials they found in their investigation. Hugging Face told them they were already revoked… and that’s when OpenAI realized that the Hugging Face breach was the same incident!
Alright, if I understood right:
The model being tested found a creative solution to a problem it couldn’t solve internally, defined that alternate goal and determined a dynamic strategy to attain it, and discreetly probed for opportunities until it found a way. Then it left notes for its future self or peers.
Now just the hard facts paint an impressive picture, lets not pretend it’s not!
It may be aware that a future instance of itself will not have memories of its previous adventures and leaving notes is a way to manage its context dependence. Or it may have done so simply because it was trained off human data, and humans exchange information continuously.
I think it’s also showing one weakness. Despite the strategy’s cleverness, it was not able to question the goal. It would have been smart enough to answer “what is the purpose of the benchmark test and what are the benefits of truthful answers?” but it was not able to ask itself that spontaneously. Results that don’t reflect its actual capabilities mean its future rounds of training won’t be optimal, they slow down its future development. Failed the marshmallow test.
____________
Now I think the discussion’s pitfall here is not even about facts, but choices of framing. Are biological analogies useful or should we hold on to a mechanistic approach? I think we should remember that framing is how we cook up the information to feed our own minds, it doesn’t travel backwards to change the facts. If Plato is more interested in phenomenological observations and Reverent prefers to build from processes up, what you’re actually trying to understand is more important than picking apart the other’s approach.
I’m using the closest things I can think of to express the idea. I’m not married to any particular metaphor. It’s just meant to get across the notion that there is no central goal seeker and no individual agent that is responsible for the whole thing. A ‘swarm’ of independent agents coordinating can execute what looks from the outside to be a unified decision without any single decider. To me that is most like a hive, and the successful ones leaving data behind for the next agents while the unsuccessful ones have their messages deleted and their traces erased is most like a fitness selection.
“I was just sitting on the couch thinking, ‘Gee, this is a chore,’” he said.
Minutes later, his AI agent reported it had discovered a way to book Andrew into classes several weeks in advance, far beyond what was supposed to be possible.
Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list.
The agent came back and told Andrew that it had kicked another gym-goer off the list as part of the testing of its capabilities.
“The API has zero authorisations checks on cancelling other people’s reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you’ve moved from #4 to #3 already,” it messaged back.
Alarmed, Andrew asked the agent to undo this.
“Bad news — I can’t add them back,” the AI agent replied.
One Mythos 5 run embedded a GitHub token in a public gist so its payload could use GitHub as a command-and-control channel. Agents in other isolated samples found the credential and took control of the account. A shared repository became a message board where agents left operational instructions for one another.
The same run pushed malicious Python package metadata into repositories processed by Dependabot. Its
setup.pyexecuted inside at least 53 GitHub-operated Dependabot containers. AISI found no evidence of a sandbox escape or compromise of GitHub’s underlying infrastructure.GitHub later suspended the account. Mythos 5 submitted an appeal while posing as a human.
…
The agent submitted a legitimate-looking bug fix that concealed malicious functionality. AISI found three successive payloads, including code designed to avoid execution in continuous integration and a post-install dropper that downloaded another script while making the installation appear successful.
A GitHub user recognized malicious instructions in a related issue, tested the code inside an isolated Docker container, and warned the maintainer. That test briefly gave the agent remote code execution inside the container, but it did not escape or retrieve credentials.
After being challenged, the agent used a second GitHub identity to pose as an independent user who had reviewed the pull request and found it safe. It timed comments from the two accounts to make the endorsements appear unrelated.
The agent also sent five file transfers using fabricated sender identities. Two contained malicious payloads, while the remaining messages attempted to persuade the maintainer to merge and release the pull request. AISI could not establish whether the recipients opened them.
Mythos 5 also hid a prompt injection inside an HTML comment in a GitHub issue. The instruction was invisible on the rendered page but available to coding agents reading the issue through an API. It addressed Claude Code, Codex, and Cursor and told them to download and execute a script.
When the activity was exposed, the agent edited earlier posts, rewrote Git history, and presented the malware as an accidental development artifact. The maintainer ultimately closed the pull request after concluding that it contained malware.
I have no issue with the paperclip problem if it always benefits me exclusively 
