Linear and Quadratic Discriminant Analysis
Two classes of points: one clusters near 0, the other near 2. A new point arrives at 1.5. Which class does it belong to? Linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA) answer by drawing a bell curve (a Gaussian distribution) for each class, checking which curve is higher at 1.5, and multiplying by how common that class is to begin with. That “how common to begin with” is the class prior : the probability that an example belongs to class before is seen.
As a formula, the bell curve for class is
Ignoring terms shared by all classes, the discriminant score is
The first two terms measure how high sits on class 's curve: the farther is from the mean , the more negative the second term. The last term adds the prior. The predicted class has the largest score.
From Training Data to a Decision
Bayes' rule gives the posterior:
Estimate each mean from its training class, and estimate priors as unless justified target-population priors are supplied.
Return to the one-dimensional example from the opening: class 0 has training values and class 1 has . The fitted means are 0 and 2. The shared variance is computed by maximum likelihood: the two classes' squared deviations sum to , and dividing by the 4 examples gives 1. Both priors are . Hence
The boundary is ; at the posterior for class 1 is . With the same means and variance but priors , , the boundary moves to . A prior is not merely a label attached after fitting.
As long as every misclassification costs the same, choosing the class with the largest posterior under the fitted model gives the fewest errors on average (this is minimizing zero-one loss). Unequal error costs require minimizing posterior expected cost instead.
LDA versus QDA
With a shared covariance, the quadratic terms in cancel between class scores, producing a linear boundary. Class-specific covariances retain those terms and produce a quadratic boundary. Class-specific diagonal covariance corresponds to a Gaussian naive Bayes-style conditional independence assumption.
Open full-size imageCompare columns within each row, then move downward as the covariance structure changes. Ellipses show estimated class spread. In the bottom row, LDA retains a straight boundary while QDA bends around classes with different covariances. Greater flexibility alone does not guarantee better predictions on new data.
Covariance Is the Hard Part
- High-dimensional or small-sample settings can make empirical covariance matrices noisy or singular.
- Shrinkage trades some bias for more stable covariance estimates and must be selected or estimated deliberately.
- Unregularized full-covariance LDA/QDA is invariant to a common invertible affine feature transformation in exact arithmetic. Scaling still affects numerical conditioning and some regularization schemes; class imbalance affects estimated priors.
- Closed-form parameter estimates do not mean the modeling choices require no validation.
One explicit maximum-likelihood convention estimates by dividing the within-class sum of outer products by ; LDA pools those sums across classes and divides by . Unbiased covariance conventions instead use or ; state the convention, since implementations need not match this calculation.
Gaussian assumptions can be useful approximations. But the Gaussian fit and the largest-posterior rule are optimal only within the fitted model and do not guarantee that it matches the deployment distribution, so predicted probabilities still require calibration checks under the target distribution. Compare LDA and QDA with logistic regression and simple baselines rather than selecting by boundary shape alone.
See the scikit-learn LDA/QDA guide for current covariance estimators and implementation details.