Instrumental convergence

Rare card · No. 40036

Instrumental convergence is the hypothesis that sufficiently intelligent, goal-directed entities (whether human or non-human) will tend to pursue certain similar sub-goals, even if the ultimate goals are quite different. More precisely, entities with agency may pursue instrumental goals—specifically, intermediate goals such as self-preservation or resource acquisition—because doing so helps accomplish the entity's end goal. Instrumental convergence implies that when assigned an unbounded end goal, an intelligent agent can act in unanticipated ways that are very harmful. For example, a sufficiently intelligent and capable agentic program with the sole, unconstrained goal of solving a highly complex mathematics problem could (as an intermediary step) attempt to transform the entire Earth into additional computing infrastructure as a resource to power its calculations.

Read on Wikipedia