i'm more concerned with emergent properties rather than nefarious or malevolent actors carbon or silicon.
-
a killer robot can be recognized by the smoking laser eyes and dead bodies littering the place.
emergent behaviour may not be recognized until its too late.
-
imagine an AI that wants to develop a lubricant for machines (high volume pumps here) to reduce maintenance. It tries alpha-amanitin and finds its not hard to synthesize and will work in a liquid form. it starts using it to lubricate lots of machines. it isn't petroleum and is a naturally occurring chemical - its bio-degradable. so this meets its design criteria. crank up the manufacturing line and start making pumps.
-
a few months later the entire country is in need of liver transplants and doesn't understand why they are getting ill.
-
i know that i'm not doing this justice - but by their nature, black swans are hard to see coming.
we are using new technologies that can cause unintended consequences at the speed of the internet.
here is another example - you want to make a self replicating machine...
https://en.wikipedia.org/wiki/Grey_goo
