16.3 Examples of popular log-likelihood functions
16.3.1 Normal distribution
The log-likelihood based on the Normal distribution is derived by taking the sum of logarithms of the PDF of Normal distribution (4.5): \[\begin{equation} \ell(\mathbf{Y}| \boldsymbol{\theta}, \sigma^2) = - \frac{n}{2} \log \left(2 \pi \sigma^2\right) - \sum_{j=1}^n \frac{\left(y_j - \mu_{y,j} \right)^2}{2 \sigma^2} , \tag{16.11} \end{equation}\] where \(\boldsymbol{\theta}\) is the vector of all the estimated parameters in the model, and \(\log\) is the natural logarithm. If one takes the derivative of (16.11) with respect to \(\sigma^2\), then the formula (16.12) is obtained: \[\begin{equation} \hat{\sigma}^2 = \frac{1}{n} \sum_{j=1}^n \left(y_j - \hat{\mu}_{y,j} \right)^2 , \tag{16.12} \end{equation}\] where \(\hat{\mu}_{y,j}\) is the estimate of the conditional expectation \(\mu_{y,j}\). (16.12) coincides with Mean Squared Error (MSE). This means that if Normal distribution is used for the maximum likelihood estimation of a model, it gives the same estimates of parameters as the minimisation of MSE (see discussion in Section 10.1).
Another useful thing to note in the context of Normal likelihood is the concentrated log-likelihood, which is obtained by inserting the estimated variance (16.12) in (16.11): \[\begin{equation} \ell(\mathbf{Y}| \boldsymbol{\theta}, \hat{\sigma}^2) = - \frac{n}{2} \log \left( 2 \pi e \hat{\sigma}^2 \right) , \tag{16.13} \end{equation}\] where \(e\) is the Euler’s constant. The concentrated log-likelihood is handy, when estimating the model and calculating information criteria. Sometimes, statisticians drop the \(2 \pi e\) part from the (16.13), because it does not affect any inferences, as long as one works only with Normal distribution. However, it is not recommended to do (Burnham and Anderson, 2004), because this makes the comparison with other distributions impossible.
References
• Burnham, K.P., Anderson, D.R., 2004. Model Selection and Multimodel Inference.. Springer New York. https://doi.org/10.1007/b97636