Imagine systems, whether biological or artificial, with levels of intelligence equal to or far greater than human intelligence. Radically enhanced human brains (or even nonhuman animal brains) could be achievable through the convergence of genetic engineering, nanotechnology, information technology, and cognitive science, while greater-than-human machine intelligence is likely to come about through advances in computer science, cognitive science, and whole brain emulation.
And now imagine if something goes wrong with one of these systems, or if they’re deliberately used as weapons. Regrettably, we probably won’t be able to contain these systems once they emerge, nor will we be able to predict the way these systems will respond to our requests.
An artificial superintelligence could “come to the conclusion that the world would be better without human beings and obliterate us,” he said, adding that some people cite this grim possibility to explain our failure to locate extraterrestrial intelligences; perhaps “all of them have been replaced by super-intelligent AI who are not interested in contacting us, as a lower form of life,” said [computer scientist Manuel] Alfonseca.
