Linear and Quadratic Discriminant Analysis

Linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA) model each class with a Gaussian distribution and combine the class-conditional density with a class prior $\pi_k$.

For class $k$,

$$p(x\mid y=k) = \mathcal{N}(x;\mu_k,\Sigma_k).$$

Ignoring terms shared by all classes, the discriminant score is

$$\delta_k(x) = -\frac{1}{2}\log|\Sigma_k| -\frac{1}{2}(x-\mu_k)^{\top}\Sigma_k^{-1}(x-\mu_k) +\log\pi_k.$$

The predicted class has the largest score.

LDA versus QDA

Model	Covariance assumption	Boundary	Main trade-off
LDA	one shared $\Sigma$	linear	fewer covariance parameters, stronger assumption
QDA	one $\Sigma_k$ per class	quadratic	more flexible, much more data needed for stable estimation

With a shared covariance, the quadratic terms in $x$ cancel between class scores, producing a linear boundary. Class-specific covariances retain those terms and produce a quadratic boundary. Class-specific diagonal covariance corresponds to a Gaussian naive Bayes-style conditional independence assumption.

Covariance Is the Hard Part

• High-dimensional or small-sample settings can make empirical covariance matrices noisy or singular.
• Shrinkage trades some bias for more stable covariance estimates and must be selected or estimated deliberately.
• Feature scaling, redundant variables, and class imbalance affect the fit and priors.
• Closed-form parameter estimates do not mean the modeling choices require no validation.

Supervised Projection

LDA can also project data onto directions that separate class means relative to within-class variation. This supervised dimensionality-reduction view is related to, but distinct from, using LDA as a classifier. With $K$ classes, at most $K-1$ discriminant directions carry between-class separation.

Gaussian assumptions can be useful approximations, but predicted probabilities still require calibration checks under the target distribution. Compare LDA and QDA with logistic regression and simple baselines rather than selecting by boundary shape alone.

See the scikit-learn LDA/QDA guide for current covariance estimators and implementation details.