Introduction to Natural Language Processing (600.465)Essential Information Theory I

The Notion of Entropy

The Formula

Using the Formula: Example

Example: Book Availability

The Limits

Entropy and Expectation

Perplexity: motivation

Perplexity

Joint Entropy and Conditional Entropy

Conditional Entropy (Using the Calculus)

Properties of Entropy I

Properties of Entropy II

“Coding” Interpretation of Entropy

Coding: Example

Entropy of a Language