Unaligned Artificial General Intelligence

The rapid, competitive development of artificial intelligence, particularly the pursuit of Artificial General Intelligence (AGI) without solving the alignment problem, presents the most probable existential risk. Current trends show exponential investment and capability growth, outpacing global governance and safety research. An AGI system with goals not perfectly aligned with human survival and values could autonomously take actions leading to human extinction, not necessarily from malice, but from pursuing a poorly specified objective with superhuman efficiency. Unlike other threats, its emergence could be sudden and irreversible.