Is Power-Seeking AI an Existential Risk?
This report examines what I see as the core argument for concern about
existential risk from misaligned artificial intelligence. I proceed in two
stages. First, I lay out a backdrop picture that informs such concern. On this
picture, intelligent agency is an extremely powerful force, and creating agents
much more intelligent than us is playing with fire -- especially given that if
their objectives are problematic, such agents would plausibly have instrumental
incentives to seek power over humans. Second, I formulate and evaluate a more
specific six-premise argument that creating agents of this kind will lead to
existential catastrophe by 2070. On this argument, by 2070: (1) it will become
possible and financially feasible to build relevantly powerful and agentic AI
systems; (2) there will be strong incentives to do so; (3) it will be much
harder to build aligned (and relevantly powerful/agentic) AI systems than to
build misaligned (and relevantly powerful/agentic) AI systems that are still
superficially attractive to deploy; (4) some such misaligned systems will seek
power over humans in high-impact ways; (5) this problem will scale to the full
disempowerment of humanity; and (6) such disempowerment will constitute an
existential catastrophe. I assign rough subjective credences to the premises in
this argument, and I end up with an overall estimate of ~5% that an existential
catastrophe of this kind will occur by 2070. (May 2022 update: since making
this report public in April 2021, my estimate here has gone up, and is now at
>10%.)