10.3 Residuals of model estimated via OLS
OLS applied to any model guarantees two important properties about its residuals:
- \(\mathrm{E}(e_j) = \frac{1}{n} \sum_{j=1}^n e_j = 0\),
- \(\mathrm{E}(e_j x_{i,j}) = \frac{1}{n} \sum_{j=1}^n e_j x_{i,j} = 0\) for any \(i\).
The first property means that the in-sample mean of residuals is always equal to zero, while the second implies that the estimation is done in a way that the in-sample correlation between the residuals and any explanatory variable in the model is equal to zero. We start by proving the first property.
Proof. Consider the sum of residuals of a simple linear regression model estimated using OLS: \[\begin{equation} \sum_{j=1}^n e_j = \sum_{j=1}^n (y_j - b_0 - b_1 x_j) = \sum_{j=1}^n y_j - n b_0 - b_1 \sum_{j=1}^n x_j \tag{10.10} \end{equation}\] Inserting the formula for \(b_0\) from (10.6) in (10.10) we get: \[\begin{equation} \sum_{j=1}^n e_j = \sum_{j=1}^n y_j - n \frac{1}{n}\sum_{j=1}^n y_j + n b_1 \frac{1}{n}\sum_{j=1}^n x_j - b_1 \sum_{j=1}^n x_j \tag{10.11} \end{equation}\] which after some cancelations leads to: \[\begin{equation} \sum_{j=1}^n e_j = \sum_{j=1}^n y_j - \sum_{j=1}^n y_j + b_1 \sum_{j=1}^n x_j - b_1 \sum_{j=1}^n x_j = 0 \tag{10.12} \end{equation}\] Given that the sum of errors is equal to zero, its mean will be equal to zero as well.
The second property is less straightforward, but it can be proven as well, using similar logic:
Proof. For the same simple linear regression, estimated using OLS, consider: \[\begin{equation} \sum_{j=1}^n e_j x_j = \sum_{j=1}^n (y_j x_j -b_0 x_j -b_1 x_j^2) = \sum_{j=1}^n y_j x_j -b_0 \sum_{j=1}^n x_j -b_1 \sum_{j=1}^n x_j^2 . \tag{10.13} \end{equation}\] Inserting (10.6) in (10.13) leads to: \[\begin{equation} \begin{aligned} \sum_{j=1}^n e_j x_j = & \sum_{j=1}^n y_j x_j - \frac{1}{n}\sum_{j=1}^n y_j \sum_{j=1}^n x_j + b_1 \frac{1}{n}\sum_{j=1}^n x_j \sum_{j=1}^n x_j - b_1 \sum_{j=1}^n x_j^2 = \\ & \sum_{j=1}^n y_j x_j - \frac{1}{n}\sum_{j=1}^n y_j \sum_{j=1}^n x_j + b_1 \left( \frac{1}{n} \left(\sum_{j=1}^n x_j \right)^2 - \sum_{j=1}^n x_j^2 \right) . \end{aligned} \tag{10.14} \end{equation}\] Now we insert the formula for \(b_1\) from (10.6) in (10.14) to get: \[\begin{equation} \begin{aligned} \sum_{j=1}^n e_j x_j = & \sum_{j=1}^n y_j x_j - \frac{1}{n}\sum_{j=1}^n y_j \sum_{j=1}^n x_j + \\ & \frac{n \sum_{j=1}^n y_j x_j - \sum_{j=1}^n y_j \sum_{j=1}^n x_j}{n \sum_{j=1}^n x_j^2 - \left(\sum_{j=1}^n x_j \right)^2} \left( \frac{1}{n} \left(\sum_{j=1}^n x_j \right)^2 - \sum_{j=1}^n x_j^2 \right) . \end{aligned} \tag{10.15} \end{equation}\] The ratio in the right-hand side of (10.15) can be regrouped and rewritten as: \[\begin{equation} -\frac{n \sum_{j=1}^n y_j x_j - \sum_{j=1}^n y_j \sum_{j=1}^n x_j}{n \left( \sum_{j=1}^n x_j^2 - \frac{1}{n}\left(\sum_{j=1}^n x_j \right)^2 \right)} \left( \sum_{j=1}^n x_j^2 - \frac{1}{n} \left(\sum_{j=1}^n x_j\right)^2 \right) , \tag{10.16} \end{equation}\] which after inserting in (10.15) and the cancelation of elements leads to: \[\begin{equation} \sum_{j=1}^n e_j x_j = \sum_{j=1}^n y_j x_j - \frac{1}{n}\sum_{j=1}^n y_j \sum_{j=1}^n x_j - \sum_{j=1}^n y_j x_j + \frac{1}{n} \sum_{j=1}^n y_j \sum_{j=1}^n x_j = 0 \tag{10.17} \end{equation}\] Given that the sum (10.17) is equal to zero, the mean of \(e_j x_j\) will be equal to zero as well.
In order to see that the second property implies that the correlation between the residuals and regressors is equal to zero, we need to take a step back and consider the covariance between \(e_j\) and \(x_j\) (because it is used in correlation coefficient as discussed in Section 9.3): \[\begin{equation} \mathrm{cov}(e_j,x_j) = \sum_{j=1}^n (e_j - \bar{e})(x_j - \bar{x}) \tag{10.18} \end{equation}\] The first thing to notice in (10.18) is that \(\bar{e}=0\) because of the property (1) discussed in the beginning of this subsection. This simplifies the formula and leads to: \[\begin{equation} \mathrm{cov}(e_j,x_j) = \sum_{j=1}^n e_j (x_j - \bar{x}) = \sum_{j=1}^n e_j x_j - \bar{x} \sum_{j=1}^n e_j = \sum_{j=1}^n e_j x_j , \tag{10.19} \end{equation}\] because the second sum in (10.19) is equal to zero due to the same property (1).
These two basic properties on one hand are useful for further derivations and on the other one show what to expect from the residuals of a regression model estimated via the OLS. The latter means, for example, that there is no point in testing whether the two properties hold, they will be satisfied automatically in case of OLS.