Multivariate Gaussian Distribution¶
Suppose random vector
with probability density function:
where:
is the mean vector is the covariance matrix is the determinant of .
Parameter Estimation¶
Suppose we have a dataset:
where each sample
The mean vector is calculated by averaging the samples:
where
The covariance matrix is estimated by summing the outer products of the centered samples:
Matrix Form¶
Let
The centered design matrix is computed as:
The covariance matrix can be written as:
Properties of the Covariance Matrix¶
Theorem
Covariance matrix
Proof:
Therefore,
Eigenvalue Interpretation¶
Without giving a formal proof, an equivalent condition to symmetric and positive semi-definite property is that all eigenvalues of the covariance matrix are non-negative [15].
An eigenvalues of the covariance matrix can be interpreted as the amount of variance explained by the corresponding eigenvector, which cannot be negative.
A zero eigenvalue indicates that one or more variables can be expressed as a linear combination of other variables, which effectively makes the covariance matrix singular and not invertible. Also, along the direction represented by the corresponding eigenvector, there is no variability in the data.
In case of a singular covariance matrix:
Apply Tikhonov regularization by adding a small multiple of the identity matrix to the covariance matrix [20].
Use the pseudo-inverse of the covariance matrix.
Perform dimensionality reduction.
Back to Statistical Learning.