My guess is that it is the character . Possibly a name ? <| sep |> As Stuart noted the words have changed meaning over time so it ' s hard to pin down . All I mean is that the paper clip maximizer has a very specific goal . It may be able to do lots of things in service of that goal but it has no ' consciousness ' to know if it should or wants to . In the other example the machine has a consciousness of sorts and thus can change it ' s motivation . You might tell it to do x but it doesn ' t want to do x and thus chooses to do Y . Both are possible and both can have terrifying consequences . But that ' s why you can have a super smart machine that does something stupid and a super smart machine that outsmarts humans .