Introduction to Natural Language Processing (600.465)Maximum Entropy

Maximum Entropy??

The Maximum Entropy Principle

Example

Using Non-Maximum Entropy Distribution

Things in Perspective: n-gram LM

Features and Constraints

Additional Constraint (Ensuring Probability Distribution)

The Model

Loglinear (Exponential) Model

Getting the Lambdas: Setup

Generalized Iterative Scaling

Comments on Features

Feature Selection

References