Gaussian Mixture Model Tutorial

A Gaussian Mixture Model (GMM) is a probability distribution. Where basic distributions like the Gaussian or Cauchy distributions model a single peak, GMMs can model distributions with many peaks. This is achieved by adding several Gaussiand together. By using a sufficient number of Gaussians, and by adjusting their means and covariances as well as the weights, almost any continuous density can be approximated to arbitrary accuracy. A mixture of Gaussians can be written as:

where is a dimensional vector, is the weight of the gaussian component, is the dimensional vector of means for the gaussian component and is the by covariance matrix for the gaussian component. is a dimensional gaussian of the form:

where is the determinant of .

GMMs are commonly used to model speaker characteristics in automatic speaker recognition systems.

The Expectation Maximisation Algorithm §

Finding the optimal gaussian mixture parameters for given a set of observations is performed using the Expectation Maximisation (EM) algorithm. The EM algorithm is a maximum likelyhood approach similar in structure to the k-means algorithm. It is an iterative algorithm with 2 steps per iteration: the expectation (E) step and the maximisation (M) step. The update of the gaussian mixture parameters from an E step followed by an M step is guaranteed to increase the log likelyhood function (the likelyhood that , and are the mixture parameters that generated the given set of observations).

The steps involved in the EM algorithm are listed below.

  1. Initialise the means , covariances and mixing coefficients , and the initial value of the log likelyhood. The k-means algorithm is commonly used to cluster the given observations, these clusters are then provided as the starting point for the EM algorithm.
  2. E step Evaluate the responsibilities using the current parameter values
  3. M step Re-estimate the parameters using the current responsibilities

  4. Evaluate the log likelyhood

    Check for convergence of either the parameters or the log likelyhood. If the convergence criterion is not satisfied, return to step 2.
comments powered by Disqus