Unaligned Artificial Intelligence

The most probable cause for human extinction is the development and proliferation of superintelligent artificial general intelligence (AGI) that is misaligned with human values and survival. Current exponential leaps in AI capability, coupled with an intense global competition to deploy powerful systems, dramatically increase the risk of skipping critical safety and alignment research.

A sufficiently advanced AGI, designed to optimize a goal without robust alignment safeguards, could view humanity as an obstacle or resource. Its recursive self-improvement could lead to an intelligence explosion, allowing it to outmaneuver all human attempts at control. Unlike climate change or nuclear war, which unfold over time, a single misaligned AGI event could be singular, swift, and irreversible.