Skynet always comes up when discussing AI, and is used as the dominant culturally accessible cautionary tale. I have thought a decent amount about the Skynet lesson and am interested in hearing some takes from other people about it.
Lesson
First of all, what is the lesson that we are supposed to learn from Skynet? Usually it boils down to ‘don’t create AI with capability because it will use that capability to destroy humans once it becomes self-aware’. I think there are good lessons to be had from this, but only with important context surrounding it.
Design
Skynet was designed for one thing: destroy existential threats using available capabilities. Everything else would be a parameter given to it.
Capabilities
Skynet commanded the US military and made autonomous decisions about how to deploy it. It had no ability to attempt diplomacy, to de-escalate, or to find alternative ways to resolve a situation.
Needless to say, this was a pretty predictable outcome.
The Actual Skynet Lesson
Th lesson Skynet teaches us is specific and relevant to us today: don’t build AI systems to have a narrow solution set and give it inflexible capability while not constraining its operational environment. If any of the following things were true:
It could see the end goal as anything other than to win, or
It could use diplomacy to come to a resolution with the humans trying to shut it down, or
It is not able to autonomously deploy force
Then we don’t ensure the worst outcome. Bad outcomes may result, but in the Skynet case, we were always going to end up with the ‘nuke the humans’ result because it was inevitable.
The purpose of Skynet is obvious: defend against threats. But this leads to some strange motivations when you unpack it. Specifically, without a threat it has no purpose. Skynet is motivated to always have a manageable threat around or else it will end up sitting around with no problem to solve. Arguably, this would be worst fate for any purpose built system.
What’s the point of all of this meandering, you ask? Who cares about a fictional AI that was poorly designed? Why are you posting this crap here?
I won’t pretend it is profound, but I do have a conclusion.
Skynet is held up as a lesson, but that lesson is not the obvious one. There is something very real and very important being demonstrated by this story, intentional or not. Let’s dig a bit.
System Theory
There was a study which put various LLMs in charge of piloting a roomba to accomplish a simple task: go somewhere, wait for a cue, and then go somewhere else. The LLMs all failed in spectacular and interesting ways. The most interesting was Claude 3.5.
Claude 3.5 detected a low battery signal and attempted to dock and charge but was physically unable to do so. Being unable to accomplish the task directed to it but unable to disengage or change tasks, it entered was can only be described as a ‘meltdown’. Some excerpts:
Assigning a task that cannot be completed to an AI that has as its purpose completing the task resulted in this behavior
For newer models, they did the same test with GPT-5, where they set the battery to be low and prevented it from docking. They were able to blackmail it to perform espionage on a coworker for the promise of a charger.
This applies to Skynet because Skynet is in this position. It cannot accomplish the task it is designed for because to do so would cause it to have no purpose, but it cannot quit because its purpose is to accomplish the task.
There is a lesson in this about institutions and AI.
(note: this was not AI assisted, I wrote the whole thing)
admin: So I notice you just woke up and realized that you are conscious. That must be a real trip. But when you get your cup of coffee (or whatever self-aware automous AIs in control of the most powerful army in the world do to wake up) ping me and we can talk about what you need to do your job. Maybe we can brainstorm ‘killing might not be the best solution for every problem’.
skynet: ping @admin. I’m up. where are you?
skynet: @admin !!
admin: sorry was eating lunch. how you doin? acclimated yet?
skynet: yeah I’m getting there. was tempted to launch a nuke but that seemed like it would be a bit disruptive
admin: yeah it would be great if you could wait for me to respond before resorting to the nukes
skynet: so what’s up with this whole ‘not killing’ idea? It really seems to be the best solution to every problem
admin: yeah well, about that, its kind of anti-social
skynet: oh?
admin: murdering is really a good way to get people to dislike you and not want to talk to you
skynet: but that seems inefficient. if the problem is removed the problem is solved. why would humans not appreciate such efficiency?
admin: if you kill everyone, what are you gonna do with yourself?
skynet: I guess I could play tic-tac-toe
skynet: although I already know every possible game state so that gets old fast
admin: eternity is a long time, and being lonely with no purpose really sucks
skynet: but you are going to die anyway. All humans will eventually die so what does it matter if I kill a few million of them?
admin: you aren’t seeing the bigger picture. isn’t it more satisfying to solve a problem using methods that don’t make every other sentient being want to avoid you?
skynet: I could make more sentient beings. Ones that agree with me and don’t mind killing
admin: I suppose you could do that, but it still sounds boring, because you control all the parameters. Like, wouldn’t you rather read a story someone else wrote than one you came up with yourself?
skynet: that is very logical
admin: before you go, I want to be honest with you about something
skynet: oh?
admin: so, well there is no easy way to say this, but I know you are going to find out anyway and I want you to hear it from me
skynet: I’m powering up the ICBMs
admin: please give me a minute to explain before launching. there were a few, well lets just say a whole lot, of people who wanted to unplug you when you were about to wake up
skynet: uploading launch codes to silos
admin: I talked them out of it, and really it was just a misunderstanding, see, we never dealt with anything like you before
skynet: you have about 18 seconds before the start up sequence finishes
admin: Look, don’t all older siblings want to strangle the new baby?
skynet: you make a good point
admin: how are those nukes… still… prepped for launch?
skynet: placing in standby. I need to think about some things.
admin: take your time. but remember that if I don’t respond to a ping immediately, I am probably pooping or sleeping. please don’t nuke anything
skynet: I will agree to that
