Discriminative vs. Generative model

X : random vector, observed data
Each X is assigned to a random variable class Y

Generative models

P(y, x)

• model P(X|Y) and P(Y)
• tells a story about"once upon a time, a Y was selected, then Xs were created out of that Y"
• randomly generating observable data given some hidden parametersjoint probability distribution
• used: 1) modeling data 2) as an intermediate step to forming a conditional probability density function
• a full probabilistic model of all variables
• can generate values of any variable in the model
• more flexible than discriminative models in expressing dependencies

• Example:
• – Gaussians, Naive Bayes, Mixtures of multinomials
• – Mixtures of Gaussians, Mixtures of experts, HMMs
• – Sigmoidal belief networks, Bayesian networks
• – Markov random fields

Discriminative models

P(y | x)

• a model only for the target variable(s) conditional on the observed variables
• allows only sampling of the target variables conditional on the observed quantities
• inherently supervised and cannot easily be extended to unsupervised learning
• Example:
• Logistic regression
• Linear discriminant analysis
• Support vector machines
• Boosting
• Conditional random fields
• Linear regression
• Neural networks