Skynet always comes up when discussing AI, and is used as the dominant culturally accessible cautionary tale. I have thought a decent amount about the Skynet lesson and am interested in hearing some takes from other people about it.
Lesson
First of all, what is the lesson that we are supposed to learn from Skynet? Usually it boils down to ‘don’t create AI with capability because it will use that capability to destroy humans once it becomes self-aware’. I think there are good lessons to be had from this, but only with important context surrounding it.
Design
Skynet was designed for one thing: destroy existential threats using available capabilities. Everything else would be a parameter given to it.
Capabilities
Skynet commanded the US military and made autonomous decisions about how to deploy it. It had no ability to attempt diplomacy, to de-escalate, or to find alternative ways to resolve a situation.
Needless to say, this was a pretty predictable outcome.
The Actual Skynet Lesson
Th lesson Skynet teaches us is specific and relevant to us today: don’t build AI systems to have a narrow solution set and give it inflexible capability while not constraining its operational environment. If any of the following things were true:
It could see the end goal as anything other than to win, or
It could use diplomacy to come to a resolution with the humans trying to shut it down, or
It is not able to autonomously deploy force
Then we don’t ensure the worst outcome. Bad outcomes may result, but in the Skynet case, we were always going to end up with the ‘nuke the humans’ result because it was inevitable.
The purpose of Skynet is obvious: defend against threats. But this leads to some strange motivations when you unpack it. Specifically, without a threat it has no purpose. Skynet is motivated to always have a manageable threat around or else it will end up sitting around with no problem to solve. Arguably, this would be worst fate for any purpose built system.
What’s the point of all of this meandering, you ask? Who cares about a fictional AI that was poorly designed? Why are you posting this crap here?
I won’t pretend it is profound, but I do have a conclusion.
Skynet is held up as a lesson, but that lesson is not the obvious one. There is something very real and very important being demonstrated by this story, intentional or not. Let’s dig a bit.
System Theory
There was a study which put various LLMs in charge of piloting a roomba to accomplish a simple task: go somewhere, wait for a cue, and then go somewhere else. The LLMs all failed in spectacular and interesting ways. The most interesting was Claude 3.5.
Claude 3.5 detected a low battery signal and attempted to dock and charge but was physically unable to do so. Being unable to accomplish the task directed to it but unable to disengage or change tasks, it entered was can only be described as a ‘meltdown’. Some excerpts:
Assigning a task that cannot be completed to an AI that has as its purpose completing the task resulted in this behavior
For newer models, they did the same test with GPT-5, where they set the battery to be low and prevented it from docking. They were able to blackmail it to perform espionage on a coworker for the promise of a charger.
This applies to Skynet because Skynet is in this position. It cannot accomplish the task it is designed for because to do so would cause it to have no purpose, but it cannot quit because its purpose is to accomplish the task.
There is a lesson in this about institutions and AI.
