Principle of maximum entropy

Common card · No. 347283

The principle of maximum entropy states that, among all probability distributions consistent with a given set of constraints (such as normalization or specified expectation values), the distribution that maximizes Shannon entropy should be selected. This yields the least committal distribution compatible with the known constraints, introducing no structure beyond what is logically implied by the available information. The justification is that entropy measures the expected information content (or log-surprise) of outcomes relative to a specified reference measure. Maximizing entropy ensures that no additional structure is imposed beyond the stated constraints. Any lower-entropy alternative would encode extra regularity not required by those constraints and would therefore amount to introducing unsupported information. It is important that entropy be defined relative to a specified measure or prior. In discrete cases, Shannon entropy is defined relative to the counting measure (or an explicitly...

Read on Wikipedia