Gaussian Mixture Model Tutorial
A Gaussian Mixture Model (GMM) is a probability distribution. Where basic distributions like the Gaussian or Cauchy distributions model a single peak, GMMs can model distributions with many peaks. This is achieved by adding several Gaussiand together. By using a sufficient number of Gaussians, and by adjusting their means and covariances as well as the weights, almost any continuous density can be approximated to arbitrary accuracy. A mixture of Gaussians can be written as:
where is a dimensional vector, is the weight of the gaussian component, is the dimensional vector of means for the gaussian component and is the by covariance matrix for the gaussian component. is a dimensional gaussian of the form:
where is the determinant of .
GMMs are commonly used to model speaker characteristics in automatic speaker recognition systems.
The Expectation Maximisation Algorithm §
Finding the optimal gaussian mixture parameters for given a set of observations is performed using the Expectation Maximisation (EM) algorithm. The EM algorithm is a maximum likelyhood approach similar in structure to the k-means algorithm. It is an iterative algorithm with 2 steps per iteration: the expectation (E) step and the maximisation (M) step. The update of the gaussian mixture parameters from an E step followed by an M step is guaranteed to increase the log likelyhood function (the likelyhood that , and are the mixture parameters that generated the given set of observations).
The steps involved in the EM algorithm are listed below.
- Initialise the means , covariances and mixing coefficients , and the initial value of the log likelyhood. The k-means algorithm is commonly used to cluster the given observations, these clusters are then provided as the starting point for the EM algorithm.
- E step Evaluate the responsibilities using the current parameter values
- M step Re-estimate the parameters using the current responsibilities
- Evaluate the log likelyhood
Check for convergence of either the parameters or the log likelyhood. If the convergence criterion is not satisfied, return to step 2.