last day (453 days later) » 

Ray
12:02 AM
@Beta For the paperclip maximizer to happen, it needs to be an extremely intelligent, creative, sapient being capable of predicting and thwarting the countermeasures of every human on the planet in real time, while at the same time being stupid enough to think that converting humans into paperclips is the best way of going about that goal. That is a very specific level of intelligence, and while it's not impossible in theory (in much the sense that all the protons in Earth spontaneously
decaying tomorrow is possible in theory), there's absolutely no plausible situation in which training an AI with a target objective of paperclip maximization would actually lead to anything like that. Even if it somehow did happen (which again, it wouldn't), we wouldn't need to outthink the evil super AI god ourselves, because we'd have a whole sequence of non-homicidal pre-super-AI-god AIs that could be tasked with outthinking it. (In practice, they would just be tasked with
ensuring that when they develop the super-AIs, they do so in a way that won't pose an unacceptable risk, just like we do when developing programs today. That's one thing the doomsayers always forget; there will never be just humans and super-AIs; the chain of AIs that developed the super AI will still exist.
It's worth noting that the people who are most worried about these doomsday scenarios are philosophers (and programmers with non-AI areas of expertise), and the ones who are least worried are actual AI researchers. AIs are not magic. This sort of doomsday prophecizing is not based on real understanding of how AIs work harms legitimate AI research.
 
 
18 hours later…
6:08 PM
There are many points for me to answer. 1) "The countermeasures of every human on the planet..." If the PM plays its cards right, very few people will mount any countermeasures at all.
2) "...stupid enough to think that converting humans into paperclips is the best way of going about that goal." The PM might well decide to convert many humans into other things, such as electronic components or spacecraft. Can you suggest a good reason why it would choose to spare humanity, and just leave all that carbon in non-paperclip form, walking around generating entropy and not contributing to the Cause? [I seem to be constrained to short messages in this chat.]
3) We could assign the more primitive AIs the job of constraining PM. Which do you think is stronger, the PM or the set of all other AIs? Whichever it is, what constrains it? Getting an AI to constrain itself and still be useful is a seriously difficult problem.
4) 'A lesser AI would have the job of designing PM to be safe.' [I am now using single quotes to paraphrase you; I hope that's clear.] How can it do the job better than we could, if it's not as smart as we are? And if it's smarter than we are, how did we make it safe?
5) 'The chain of lesser AIs will still exist.' The chain of chess computers leading up to Deep Blue, working as a team, could not beat Deep Blue. The argument that a group of lesser AIs could control a stronger one-- without themselves being a threat -- is hand-waving and reasoning by weak analogy.
6) "The people who are most worried about these doomsday scenarios are philosophers (and programmers with non-AI areas of expertise), and the ones who are least worried are actual AI researchers." Among competent researchers, the ones who are worried about these doomsday scenarios go into AI safety research, and the ones who aren't go into AI development. Besides, this is an argument from authority about a technology that does not yet exist; let the real arguments stand on their merit.
7) *The fun one.* They build the AI lab, and install a big switch -- pull it, and the whole
building loses power. The switch is in a special room, right off the main corridor on the sixth floor, and you have the key, along with two other people. Any of you can go in and pull the switch at any time. You have authority to go everywhere in the building, inspect everything, talk to everyone. You can enter the server room (next to the switch room) and listen to the boxes hum. You can look at the code, and watch the log scrolling by. Is that sufficient? Anything else you want to add to the protocol?
 
 
4 hours later…
Ray
10:45 PM
1) Not much I can say in response to this without some more details.
2) Because going to war with humans over this is a potential failure case. If its goal is to maximize the expected number of paperclips produced, the optimal move is (in part) to minimize the chance of stopping. Especially if we're assuming that it devotes sufficient resources for the outcome of the war to be nearly a sure thing, those are resources that might be better spent getting assembly plants out to the Kuiper Belt, where there's plenty of raw materials available for minimal risk.
While gathering the materials needed for that, it can target areas like the Sahara, or the bottom of the ocean, which nobody will particularly mind if it digs up. In the long run, energy requirements will become an issue and it'll want a Dyson Sphere, which humans would be all in favor of. At least in the early years, humans will have a much greater production capability than the PM does,
and they will be of more benefit as a source of labor than they would as raw materials (who makes paperclips and spaceships out of meat, anyway?). Getting the willing cooperation of humans maximizes risk/reward.
3) The lesser AIs I speak of are the ones that designed the super AI (and the ones that designed those, and so on). We know they're safe partly by definition; the PM is the first AI to pose an existential risk (paperclip related or otherwise), and so the lesser AIs consist of the various non-genocidal AIs that led up to that point.
But we also know that they won't pose an existential risk because we'll design them that way. I can say with absolute certainty (again with the "all the protons might decay tomorrow" disclaimer of "yes, the probability is non-zero, but is so low as to be effectively 0) that no AI I have ever built will destroy humanity. I can also say with absolute certainty that no AI I have ever built will design an AI that will destroy humanity. And for that matter, that no AI I have ever built will design
an AI at all; even if designing a better AI would help them achieve their goals, they are entirely incapable of doing so. A model capable of doing so would require a greater capacity than my models possess, and even if that did exist in the search space somewhere, the optimization function wouldn't lead there.
Whoever designs the first human-level AI only needs that level of certainty. That the first one is friendly. They can then ask that one to not design any AIs that will destroy the world, and because it's friendly, it will want to do so. Self-preservation, frequently brought up as the thing that will cause the AIs to rise up and destroy us, will actually work in our favor; the seed AI is part of the world and won't want it to be destroyed.
5) That's because Deep Blue worked using a combination of the minimax algorithm and a set of hardcoded opening tables. Another chess AI that used the same algorithm to a lower depth, or had smaller opening tables, would be strictly dominated by Deep Blue. (although one that used different heuristics for end game states or different pruning techniques might conceivably beat it even if it lost more often than not).
It's also because chess is strictly a two-player game with fixed rules. If you put Deep Blue and a number of allied lesser AIs on a giant board with a full set of pieces for each, Deep Blue would be defeated.
6) I realize I'm getting dangerously close to an argument from authority there, but I'm not invoking authority so much as expertise. Lots of people claim that the Large Hadron Collider will destroy the world, but physicists say it won't, and we trust them because they actually understand how it works. Lots of people claim that global warming isn't a thing, but climate scientists say it is, and we trust them because they understand how it works.
I'm not saying that nobody can make an argument on this without a fancy title, but I am going to be skeptical when someone who has never designed a single AI tells me that AIs are going to destroy the world. They lack the necessary background education to make informed decisions.
7) The main thing is proactive defense; design the AI such that it isn't genocidal in the first place. But for the purposes of this part, we can assume that we're not confident we've done so and want to have a failsafe in place just in case.
Start with your setup, but with the clarification that the switch is a purely physical setup; pull the switch and the wires are literally cut (and on the computer side of the UPS, of course). We should also have an offsite copy of the switch that works the same way. And the switch should actually be a big red button; it's traditional. Additionally, we consider network access. For the initial tests, there will be none, but eventually we'll want it to have access to the outside world.
Have a firewall that cannot be logged into except via physical access, and has everything mounted readonly. It will drop ssh packets or anything of the sort. It will drop packets large enough to contain the AI code. Probably also have some filters for outgoing content (e.g. packets containing code), and have it shut down if they get tripped often enough to indicate that the AI is trying to bypass them.
We should also have conversations like the one we're having now, where people propose defenses and other people poke holes in them (where some of those people will include AIs that have been previously vetted friendly). It's what we do for security systems now.
(I'm getting off into philosophical territory here; up until now, I've been explaining why the risk is minimal. The following is more why I think we could tolerate some risk even if it was more substantial.)
All that being said, I'm not claiming that AIs could never pose a threat. Rather, I'm claiming that we're nowhere near that point yet, and when we are, the experts of the day will be able to say whether the things they're designing will pose an unacceptable threat, just like I can say that about the AIs I work on. The fact that at some point, the scientists in question will be AIs themselves doesn't change that so long as they're friendly themselves. And yes, once they reach a certain point,
we may not be able to perfectly analyze every action that they'll take, but how is that different from any other person? Which is what they'll be once they're as smart as we are. Not humans, certainly; they will think differently than us, but people. Some amount of trust may be required, but that's true with humans as well. The difference is, with humans, we have basically no understanding of how they work, whereas with the AIs, we will have a good, but imperfect understanding of how they
work. And while the sapient AIs will not yet have given us specific reasons to believe we can trust them (what with them not existing yet, and all), the humans have spent the last 7000 years proving they shouldn't be trusted, and yet we still manage to do so because otherwise society would fall apart. Existential risks exist today. Nuclear war, impact events, mantle plume eruptions (or even just supervolcanos). AI, even the non-sapient kind, can help us manage these risks
(if only by helping to get us to the point where we can get an offworld colony faster), and in the meantime, can improve our lives (and indeed, is already doing so). Proper risk management considers risk and reward; we shouldn't just focus on one theoretical (and not especially plausible) bugbear and in doing so delay useful research.
 

  last day (453 days later) »