6 Outliers, Leverage, and Influence
A fitted regression line summarizes the observations, but some observations may matter much more than others. We ask three different questions: is the response unusual given the predictors, are the predictor values unusual, and does the observation substantially change the fit? These lead to outliers, leverage, and influence. We use the same small dataset throughout so that every diagnostic can be connected to a visible change in the estimates.
Learning Objectives
By the end of this chapter, students should be able to:
- distinguish a response outlier, a high-leverage observation, and an influential observation;
- compare coefficient estimates and predictions with and without one observation;
- interpret leverage, standardized residuals, and externally studentized residuals;
- explain how Cook’s distance combines response discrepancy and predictor position;
- investigate a flagged observation and report a sensitivity analysis.
NoteQuick Recap: What We Will Use
- Mean model and design — Chapter 1. Write \(\mathbf Y=\mathbf X\boldsymbol\beta+\boldsymbol\varepsilon\), with \(\mathbb E(\boldsymbol\varepsilon\mid\mathbf X)=\mathbf0\). A row describes an observation; a column multiplies a coefficient. Let \(p\) count the columns, including the intercept. We assume full column rank and \(n>p\).
- Fitted values and residuals — Chapter 2. OLS gives \(\hat{\mathbf Y}=\mathbf H\mathbf Y\) and \(\mathbf e=(\mathbf I-\mathbf H)\mathbf Y\), where \(\mathbf H=\mathbf X(\mathbf X^\top\mathbf X)^{-1}\mathbf X^\top\). A residual is an observed deviation from the fitted mean, not the unobserved error.
- Noise and uncertainty — Chapter 3. Under \(\operatorname{Var}(\boldsymbol\varepsilon\mid\mathbf X)=\sigma^2\mathbf I\), estimate \(\sigma^2\) by \(s^2=\mathrm{SSE}/(n-p)\). Here \(\mathrm{SSE}=\sum_i e_i^2\) is the unexplained variation from Chapter 4. Estimated coefficient covariance is \(s^2(\mathbf X^\top\mathbf X)^{-1}\).
- Interpret the quantity that changes — Chapter 5. An additive slope describes a mean-response difference holding other predictors fixed. To assess influence, compare the coefficients and predictions relevant to that question, using the same model specification.
6.1 See how one observation changes the fit
First look at the data and the fitted lines. We will name the three concepts after seeing how they differ, then introduce their numerical measures.
Example 6.1 (Three Points, Three Different Stories) Look at the lines first. Every panel starts with the same six green observations, whose fitted line is \(\hat y=2+x\). We add one orange point at a time. The dashed gray line omits that point; the solid purple line includes it. Compare the point’s vertical position, its horizontal position, and how much the line moves.
- A: unusual response, ordinary predictor value. At \(x=3.5\), the original line predicts 5.5, but the orange response is 10.5. This point is near the center of the predictor values and far above the line. It mainly shifts the line upward.
- B: high leverage without a change in the fitted line. The new \(x=10\) lies beyond the original range of 1 to 6. Its response, 12, lies exactly on the extension of the original line. The point has the potential to pull the fit, but here the two lines coincide.
- C: high leverage with a large change in the fitted line. The predictor is again \(x=10\), but the response is 4 instead of 12. The point pulls down the right side of the line, substantially reducing the estimated slope.
These descriptions can overlap. C is also a high-leverage point; its actual effect on the fit makes it influential. A also affects the fit, even though its effect on the slope differs from C’s.
Now put numbers on the changes. The six baseline observations are:
| Variable | Obs. 1 | Obs. 2 | Obs. 3 | Obs. 4 | Obs. 5 | Obs. 6 |
|---|---|---|---|---|---|---|
| x | 1.0 | 2.0 | 3.0 | 4.0 | 5.0 | 6.0 |
| y | 2.8 | 4.2 | 5.0 | 6.0 | 7.2 | 7.8 |
Fit lm(y ~ x) to the baseline data, then add A, B, or C separately. The table reports the intercept, slope, fitted mean at \(x=6\), and residual standard deviation \(s\). Removing the orange point from any of the three augmented datasets returns the baseline fit.
| Fit | Intercept | Slope | Fitted mean at 6 | Residual SD |
|---|---|---|---|---|
| Baseline | 2.000 | 1.000 | 8.000 | 0.200 |
| Baseline + A | 2.714 | 1.000 | 8.714 | 2.078 |
| Baseline + B | 2.000 | 1.000 | 8.000 | 0.179 |
| Baseline + C | 4.532 | 0.170 | 5.553 | 1.899 |
- A changes the intercept and the noise estimate. The intercept rises from 2.000 to 2.714, while the slope stays at 1.000. Here \(x=3.5\) is the baseline mean of \(x\), so this added point does not change the slope. The residual SD rises from 0.200 to 2.078: an outlying response can inflate estimated noise even when the slope is unchanged.
- B leaves both coefficient estimates unchanged. The fitted mean at \(x=6\) remains 8.000. High leverage alone does not imply that a point changes the estimated line; it can still affect the precision of the estimates.
- C changes the estimated relationship. The slope falls from 1.000 to 0.170 and the intercept rises to 4.532. The fitted mean at \(x=6\) falls from 8.000 to 5.553. This is a substantial change in a prediction within the original predictor range, not just at the distant added point.
These pictures motivate three distinct definitions.
Definition 6.1 (Outlier, Leverage, and Influence)
- A response outlier has an unusual response given its predictor values and the specified model. An extreme response value alone does not make an observation a response outlier.
- A high-leverage observation has an unusual configuration of predictor values. Leverage concerns the predictors jointly and does not depend on the observed responses. It measures the potential to pull the fit.
- An influential observation changes the fitted coefficients or predictions substantially when omitted. Influence concerns the sensitivity of the analysis, rather than unusualness alone.
6.2 High leverage: unusual predictor values
Leverage asks where the observation sits among the predictor rows. It does not use its response. From the hat matrix, define \(h_{ii}\) as diagonal entry \(i\). With an intercept and one predictor,
\[ h_{ii}=\frac1n+\frac{(x_i-\bar x)^2}{\sum_j(x_j-\bar x)^2}. \]
The first term is shared by all observations; the second grows as \(x_i\) moves away from the predictor center. The mean and sum in this formula use all observations in the fit being diagnosed, including the point of interest. With several predictors, leverage measures unusualness of the whole predictor combination, not just one variable at a time.
| Point | Added x | Leverage |
|---|---|---|
| A | 3.500 | 0.143 |
| B | 10.000 | 0.721 |
| C | 10.000 | 0.721 |
For A, \(x=3.5\) remains the predictor mean, so its leverage is \(1/7=0.143\). B and C have the same predictor rows, so both have leverage 0.721 even though their responses are different. B leaves the estimated line unchanged; C changes it substantially. Thus high leverage indicates potential influence, not a conclusion that the fit has changed.
Because \(\sum_i h_{ii}=p\), average leverage is \(p/n\). In this example, \(p=2\) and \(n=7\), so the average is \(2/7=0.286\). Comparing a value with \(2p/n\) is a rough screening rule, not a test or a reason to delete an observation. R returns leverages with hatvalues(fit).
Derivation: leverage and its average
1. Center the predictor without changing the fitted model. Set \(z_i=x_i-\bar x\), \(\mathbf z=(z_1,\ldots,z_n)^\top\), and \(S_{xx}=\sum_i z_i^2>0\). The columns of \(\mathbf Z=(\mathbf1,\mathbf z)\) span the same space as \((\mathbf1,\mathbf x)\), so they give the same hat matrix. Since \(\mathbf1^\top\mathbf z=0\),
\[ \mathbf Z^\top\mathbf Z= \begin{pmatrix}n&0\\0&S_{xx}\end{pmatrix}, \qquad \mathbf H=\mathbf Z(\mathbf Z^\top\mathbf Z)^{-1}\mathbf Z^\top =\frac{\mathbf1\mathbf1^\top}{n}+\frac{\mathbf z\mathbf z^\top}{S_{xx}}. \]
2. Read diagonal entry \(i\). It is
\[ h_{ii}=\frac1n+\frac{z_i^2}{S_{xx}} =\frac1n+\frac{(x_i-\bar x)^2}{\sum_j(x_j-\bar x)^2}. \]
3. Sum the leverages in a general full-rank model. By the cyclic property of the trace,
\[ \sum_i h_{ii}=\operatorname{tr}(\mathbf H) =\operatorname{tr}\!\left[(\mathbf X^\top\mathbf X)^{-1}\mathbf X^\top\mathbf X\right] =\operatorname{tr}(\mathbf I_p)=p. \]
Thus average leverage is \(p/n\). This uses the projection matrix from Chapter 2, not the responses.
A well-measured high-leverage observation can be useful: under an appropriate model it can provide information about the slope. In B, the slope estimate stays at 1.000 while its estimated standard error falls from 0.048 to 0.024. Whether the model is credible at that distant predictor value still needs scientific judgment.
6.3 Outliers: unusual responses given the predictors
An outlier is unusual relative to the mean predicted by the specified model. B has the largest response in the example, 12, yet it lies on the fitted line and has zero residual. A’s response is smaller, 10.5, but is far above the expected response near \(x=3.5\). Looking only at the distribution of \(Y\) would miss this distinction.
For the diagnostics below, write \(e_i=Y_i-\hat Y_i\) for the raw residual. We use \(r_i=e_i/\{s\sqrt{1-h_{ii}}\}\) for the standardized residual and \(t_i=e_i/\{s_{(-i)}\sqrt{1-h_{ii}}\}\) for the externally studentized residual; \(s\) uses the full fit and \(s_{(-i)}\) uses the fit without observation \(i\). Large \(|t_i|\) flags a response discrepancy to investigate, while Cook’s distance will assess its effect on the fit.
The definitions, choice of residual, worked comparison of A, B, and C, and derivations are collected in Section 6.7: Supplementary. Outlyingness and influence answer different questions: a large scaled residual does not by itself tell us how much the fitted relationship changes.
6.4 Influence: changes in estimates and Cook’s distance
Influence asks how much the fitted result changes when an observation is omitted. Compare the same model on the remaining observations: changes in the slope, intercept, or scientifically relevant prediction describe what that observation was doing. Cook’s distance summarizes movement of the fitted mean vector using both the residual and leverage.
Definition 6.2 (Cook’s Distance) Let \(\hat{\boldsymbol\beta}_{(-i)}\) be the OLS estimate after omitting observation \(i\), and let \(\hat Y_{j(-i)}=\mathbf x_j^\top\hat{\boldsymbol\beta}_{(-i)}\) be its prediction at each of the original \(n\) predictor rows, including row \(i\). Cook’s distance is
\[ D_i=\frac{\sum_{j=1}^n(\hat Y_j-\hat Y_{j(-i)})^2}{p s^2} =\frac{e_i^2}{p s^2}\frac{h_{ii}}{(1-h_{ii})^2} =\frac{r_i^2}{p}\frac{h_{ii}}{1-h_{ii}}. \]
Here \(s^2\) comes from the full fit. These formulas apply to ordinary unweighted least squares with full column rank and positive residual degrees of freedom; the deleted fit must also be estimable (\(h_{ii}<1\)).
Read the two parts. The first expression asks how much the whole fitted mean vector moves after deletion, measured relative to the noise scale and model size. The last combines response discrepancy (\(r_i^2\)) with leverage (\(h_{ii}/(1-h_{ii})\)). Thus the largest residual need not produce the largest Cook’s distance. \(D_i\) is dimensionless; it is not a p-value. Cook’s original article.
Derivation: the equivalent formulas for Cook’s distance
1. Express movement of the fitted means through the coefficient change. Use \(\boldsymbol\delta_i=\hat{\boldsymbol\beta}-\hat{\boldsymbol\beta}_{(-i)}\) from the deletion derivation in Section 6.7. Predictions are evaluated at all \(n\) original rows, so
\[ \sum_{j=1}^n(\hat Y_j-\hat Y_{j(-i)})^2 =\|\mathbf X\boldsymbol\delta_i\|^2 =\boldsymbol\delta_i^\top\mathbf A\boldsymbol\delta_i. \]
2. Substitute the deleted-coefficient identity. With \(\mathbf A=\mathbf X^\top\mathbf X\),
\[ \begin{aligned} \boldsymbol\delta_i^\top\mathbf A\boldsymbol\delta_i &=\frac{e_i^2}{(1-h_{ii})^2} \mathbf x_i^\top\mathbf A^{-1}\mathbf A\mathbf A^{-1}\mathbf x_i\\ &=\frac{e_i^2h_{ii}}{(1-h_{ii})^2}. \end{aligned} \]
3. Scale the change and rewrite using \(r_i\). Dividing by \(ps^2\) gives
\[ D_i=\frac{e_i^2}{ps^2}\frac{h_{ii}}{(1-h_{ii})^2} =\frac{r_i^2}{p}\frac{h_{ii}}{1-h_{ii}}, \qquad r_i^2=\frac{e_i^2}{s^2(1-h_{ii})}. \]
Cook’s distance uses the full-fit noise scale \(s\), so this expression uses the internally studentized \(r_i\). Substituting the externally studentized \(t_i\) into the same expression would not give Cook’s distance. The algebra does not require normal errors; the variance model supplies the interpretation of the scaling.
Apply Cook’s distance to the same three points. In each augmented fit, \(n=7\) and \(p=2\). The diagnostics for its orange point are:
| Point | Leverage | Standardized residual | Cook’s distance |
|---|---|---|---|
| A | 0.143 | 2.228 | 0.414 |
| B | 0.721 | 0.000 | 0.000 |
| C | 0.721 | -2.226 | 6.395 |
A and C have almost the same absolute standardized residual, but C has much higher leverage and a much larger Cook’s distance: 6.395 versus 0.414. B has the same leverage as C, yet its zero residual gives zero Cook’s distance. These numbers quantify what the plots and coefficient table already showed: influence depends on both the response discrepancy and the predictor position.
Exercise 6.1 (Which Point Moves the Fit More?) In a model with \(p=2\), point A has \(r_A=3\) and \(h_{AA}=0.05\); point B has \(r_B=1\) and \(h_{BB}=0.50\). Compute their Cook’s distances. Which point is more influential by this measure? Does either number establish a data error?
Solution 6.1. \(D_A=(9/2)(0.05/0.95)\approx0.237\), while \(D_B=(1/2)(0.50/0.50)=0.50\). B has greater influence despite its smaller standardized residual. Neither value establishes an error or justifies deletion; inspect the observations and the scientific consequences of the changed fit.
6.5 Summary
Keep the three questions separate: unusual response, unusual predictors, or a changed fit. A point can belong to more than one category. Use the table to choose the appropriate diagnostic and follow-up.
| Observation | How to identify it | What to do |
|---|---|---|
| Response outlier | Use the externally studentized residual \(t_i\). Rough screen: \(\lvert t_i\rvert>2\). Formal check under the normal model: one pre-specified point, \(\lvert t_i\rvert>t_{d,1-\alpha/2}\); all \(n\) points (Bonferroni), \(\lvert t_i\rvert>t_{d,1-\alpha/(2n)}\). | Verify the response and measurement conditions. Check whether the fitted mean or error model explains the discrepancy. Assess its effect on estimates separately. |
| High-leverage point | Unusual predictor values: \(h_{ii}>2p/n\) is a rough screen; average leverage is \(p/n\). High leverage alone does not establish an outlier or actual influence. | Verify predictor values and study eligibility. Examine the residual and compare fits to determine whether the point actually changes the result. |
| Influential point | Use \(D_i>4/n\) or a Cook’s distance that stands out to flag possible influence. Confirm it through meaningful changes in coefficients or predictions when the point is omitted. | Refit the same model with and without the point. Compare relevant estimates, uncertainty, and predictions; report both fits when conclusions are sensitive. |
For the formal outlier checks, use the assumptions in Section 6.7: \(d=n-p-1\), \(t_{d,q}\) is the \(q\) quantile of the t distribution, and \(\alpha\) is the chosen significance level. The leverage and Cook cutoffs are screening guides, not tests; falling below one does not prove that a point is harmless.
For all three: correct verified errors and repeat the diagnostics. Retain valid, eligible observations; exclude only for a documented data-quality or eligibility reason, not a diagnostic value alone. If valid observations reveal an inadequate model, revise it and recheck. Report the decision and its effect on the conclusion.
Chapter 7 develops transformations and variable selection.
6.6 Practice Problems
Exercise 6.2 (Practice: Diagnose and Explain)
- B has a larger response than A but a zero residual. Why is the response value alone insufficient to identify an outlier?
- Why do B and C have the same leverage but different Cook’s distances?
- In A, the slope is unchanged but the intercept and residual SD increase. Is it correct to say the point has no effect on estimation?
- A high-leverage point has a small raw residual. Explain why this alone does not establish that it is harmless.
- A student deletes every point with \(D_i>4/n\). State what information and analysis are missing.
Solution 6.2.
- Outlyingness concerns the response relative to the fitted conditional mean. B lies on the original line; A lies well above it at an ordinary predictor value.
- Their predictor rows are identical, so leverage is identical. Their responses differ: B leaves the line unchanged, while C moves it substantially.
- No. A affects the intercept, predictions, and estimated noise even though its position at the predictor mean leaves the slope unchanged.
- A high-leverage observation can pull the fit toward itself. Examine a scaled residual and compare the fitted result with and without that observation.
- The cutoff is only a screening guide. Verify the observation, examine the model, compare scientifically relevant estimates and predictions with and without the point, and justify any exclusion using evidence.
6.7 Supplementary
Raw, standardized, and studentized residuals: which should I use? This supplement expands the notation used in Section 6.3. Choose the residual according to the question; scaling does not replace checking the model or the measurements.
Definition 6.3 (Raw (Ordinary) Residual) The raw residual is \(e_i=Y_i-\hat Y_i\), the observed response minus its fitted mean from the full model. It has the same units as the response being modeled: \(e_i=5\) means the response is five units above its fitted mean. If the model response is \(\log Y\), its residual is in log-response units, not the original units of \(Y\). In R, use resid(fit) or residuals(fit).
A residual estimates an unobserved error; it is not that error itself. Its size also depends on leverage, so raw residuals alone are not a fair way to rank unusual observations.
Choose by purpose.
| Residual | When is it useful? | R function |
|---|---|---|
| Raw \(e_i\) | Describe discrepancies in response units; inspect curvature, spread, or time patterns in residual plots. | resid(fit) |
| Standardized \(r_i\) (internally studentized) | Compare residual sizes after adjusting for leverage and the common noise scale; use in general diagnostic plots and the residual form of Cook’s distance. | rstandard(fit) |
| Externally studentized \(t_i\) | Investigate an individual response outlier without allowing that observation to inflate its own noise estimate; use for a normal-model outlier test when its assumptions hold. | rstudent(fit) |
For ordinary unweighted lm fits, plot(fit, which = 1) shows raw residuals versus fitted values. R’s normal Q-Q plot (which = 2) and residual-versus-leverage plot (which = 5) use standardized residuals, not rstudent(fit). These plots assess different aspects of the same model. R plotting documentation.
NoteScaling a residual is not scaling the response
The two scaled residuals are dimensionless. Neither is obtained by calling scale(resid(fit)): each uses the observation-specific factor \(\sqrt{1-h_{ii}}\). Reporting scaled residuals does not change the original fitted coefficients or response scale, make the residuals independent, or repair nonconstant error variance. The quantities below assume a common error variance; weighted regression and GLMs require diagnostics appropriate to those models.
Shared notation and assumptions. Let \(h_{ii}\) be the full-fit leverage, \(p\) count coefficients including the intercept, and \(\nu=n-p\) be the residual degrees of freedom. Assume ordinary unweighted least squares with full column rank. Under the mean model and constant error variance,
\[ \operatorname{Var}(e_i\mid\mathbf X)=\sigma^2(1-h_{ii}). \]
Different observations therefore have different residual variances. Both residuals below account for this using \(\sqrt{1-h_{ii}}\); they differ in how the unknown error standard deviation \(\sigma\) is estimated. Assume \(h_{ii}<1\) and positive estimated error variance.
Definition 6.4 (Standardized Residual (Internally Studentized)) Estimate the error variance using all \(n\) observations:
\[ s^2=\frac{\mathrm{SSE}}{n-p} =\frac{\sum_{j=1}^n e_j^2}{n-p}. \]
The standardized residual is
\[ r_i=\frac{e_i}{s\sqrt{1-h_{ii}}}. \]
Observation \(i\) helps determine both the fitted line and \(s\). An unusual response can inflate \(s\) and make its own scaled residual appear less extreme. This residual is also called internally studentized. In R, use rstandard(fit) with its default settings.
Definition 6.5 (Externally Studentized Residual (Studentized Deleted Residual)) Refit the same model after removing observation \(i\), giving \(\hat{\boldsymbol\beta}_{(-i)}\). Estimate the error variance from the remaining \(n-1\) observations:
\[ s_{(-i)}^2=\frac{\mathrm{SSE}_{(-i)}}{n-p-1}, \qquad \mathrm{SSE}_{(-i)}=\sum_{j\ne i} \left(Y_j-\mathbf x_j^\top\hat{\boldsymbol\beta}_{(-i)}\right)^2. \]
The externally studentized residual is
\[ t_i=\frac{e_i}{s_{(-i)}\sqrt{1-h_{ii}}}. \]
Here \(e_i\) and \(h_{ii}\) still come from the full fit; only the error-scale estimate is replaced. The deleted fit must retain rank \(p\), with \(n-p-1>0\) and \(s_{(-i)}>0\). In R, use rstudent(fit).
In this chapter, standardized means internally studentized, and studentized by itself means externally studentized, matching these two R diagnostic functions. Other texts may use studentized for either type; check which variance estimate appears in the denominator.
Neither scaling makes residuals independent. With independent normal errors and a correctly specified, fixed model, each pre-specified \(t_i\) has a \(t_{n-p-1}\) distribution; \(r_i\) does not have that exact t distribution. Checking many points also requires accounting for multiple comparisons.
Compare the same three added points. Each row uses its own augmented fit. The \(s\) column uses all seven observations; \(s_{(-i)}\) comes from refitting after removing that row’s added point. The last column is the externally studentized residual.
| Point | Residual \(e_i\) | \(s\) | \(s_{(-i)}\) | Standardized \(r_i\) | Studentized \(t_i\) |
|---|---|---|---|---|---|
| A | 4.286 | 2.078 | 0.200 | 2.228 | 23.146 |
| B | 0.000 | 0.179 | 0.200 | 0.000 | 0.000 |
| C | -2.234 | 1.899 | 0.200 | -2.226 | -21.138 |
For A, the full-fit residual SD is \(s=2.078\), but deleting A gives \(s_{(-A)}=0.200\). The same full-fit residual and leverage therefore produce \(r_A=2.228\) and \(t_A=23.146\). A was inflating the noise estimate used to assess itself. For C the scales are 1.899 and 0.200, with the same effect. Deleting either point returns the six baseline observations from Section 6.1. B has zero residual, so both scaled residuals are zero regardless of the different scales. These diagnostics identify a response discrepancy under the model; they do not establish a recording error.
A and C have similar absolute standardized residuals, yet C changes the slope much more. Outlyingness and influence answer different questions.
Derivation: why divide by the square root of one minus leverage?
1. Express the residual vector using the errors. From the residual maker in Chapter 2, \(\mathbf M=\mathbf I-\mathbf H\) is symmetric and idempotent, and \(\mathbf M\mathbf X=\mathbf0\). Therefore
\[ \mathbf e=\mathbf M\mathbf Y =\mathbf M(\mathbf X\boldsymbol\beta+\boldsymbol\varepsilon) =\mathbf M\boldsymbol\varepsilon. \]
2. Compute its covariance. Under \(\operatorname{Var}(\boldsymbol\varepsilon\mid\mathbf X)=\sigma^2\mathbf I\),
\[ \operatorname{Var}(\mathbf e\mid\mathbf X) =\mathbf M(\sigma^2\mathbf I)\mathbf M^\top =\sigma^2\mathbf M^2 =\sigma^2(\mathbf I-\mathbf H). \]
The \(i\)th diagonal entry is \(\sigma^2(1-h_{ii})\), so the true standard deviation of \(e_i\) is \(\sigma\sqrt{1-h_{ii}}\). Replacing \(\sigma\) with \(s\) gives \(r_i\); replacing it with \(s_{(-i)}\) gives \(t_i\).
3. Keep variance scaling separate from independence. For \(i\ne j\), \(\operatorname{Cov}(e_i,e_j\mid\mathbf X)=-\sigma^2 h_{ij}\), generally nonzero. Dividing residuals by estimated scales does not remove their shared dependence on the fitted model. A high-leverage point has a smaller raw-residual variance, which is why raw residuals alone are not comparable.
Derivation: deleting one observation and relating the two residuals
1. Find the coefficient change. Write \(\mathbf A=\mathbf X^\top\mathbf X\) and let \(\mathbf x_i\) be the column vector whose transpose is row \(i\). The full and deleted normal equations are
\[ \mathbf A\hat{\boldsymbol\beta}=\mathbf X^\top\mathbf Y, \qquad (\mathbf A-\mathbf x_i\mathbf x_i^\top)\hat{\boldsymbol\beta}_{(-i)} =\mathbf X^\top\mathbf Y-\mathbf x_iY_i. \]
For \(\boldsymbol\delta_i=\hat{\boldsymbol\beta}-\hat{\boldsymbol\beta}_{(-i)}\), subtracting gives
\[ (\mathbf A-\mathbf x_i\mathbf x_i^\top)\boldsymbol\delta_i=\mathbf x_i e_i. \]
Since \(h_{ii}=\mathbf x_i^\top\mathbf A^{-1}\mathbf x_i\), \((\mathbf A-\mathbf x_i\mathbf x_i^\top)\mathbf A^{-1}\mathbf x_i=(1-h_{ii})\mathbf x_i\). When the deleted fit is estimable, this identifies the solution:
\[ \boldsymbol\delta_i=\frac{\mathbf A^{-1}\mathbf x_i e_i}{1-h_{ii}}. \]
2. Distinguish a deleted residual from a full-fit residual. Predict the omitted response from the deleted fit:
\[ d_i=Y_i-\mathbf x_i^\top\hat{\boldsymbol\beta}_{(-i)} =e_i+\mathbf x_i^\top\boldsymbol\delta_i =\frac{e_i}{1-h_{ii}}. \]
Thus \(d_i\) is not the numerator \(e_i\) in the displayed definition of \(t_i\).
3. Recalculate the residual sum of squares. At all original rows, the deleted-fit residual vector is \(\mathbf e+\mathbf X\boldsymbol\delta_i\). Remove its \(i\)th squared component \(d_i^2\) and use \(\mathbf X^\top\mathbf e=\mathbf0\):
\[ \begin{aligned} \mathrm{SSE}_{(-i)} &=\|\mathbf e+\mathbf X\boldsymbol\delta_i\|^2-d_i^2\\ &=\mathrm{SSE}+\boldsymbol\delta_i^\top\mathbf A\boldsymbol\delta_i-d_i^2\\ &=\mathrm{SSE}+\frac{e_i^2h_{ii}}{(1-h_{ii})^2} -\frac{e_i^2}{(1-h_{ii})^2}\\ &=\mathrm{SSE}-\frac{e_i^2}{1-h_{ii}}. \end{aligned} \]
It is not simply \(\mathrm{SSE}-e_i^2\): refitting changes the other residuals too.
4. Relate the two scales and residuals. With \(\nu=n-p\), \(\mathrm{SSE}=\nu s^2\), and \(r_i^2=e_i^2/[s^2(1-h_{ii})]\),
\[ s_{(-i)}^2=s^2\frac{\nu-r_i^2}{\nu-1}, \qquad t_i=r_i\frac{s}{s_{(-i)}} =r_i\sqrt{\frac{\nu-1}{\nu-r_i^2}}. \]
For positive deleted-fit variance, \(r_i^2<\nu\). A very unusual point can make \(r_i\) approach its finite bound while \(t_i\) becomes much larger. The two values have the same sign, but need not have similar magnitudes; \(|t_i|\) is not always larger than \(|r_i|\).
Derivation: why the externally studentized residual has a t distribution
Assume \(\boldsymbol\varepsilon\mid\mathbf X\sim N_n(\mathbf0,\sigma^2\mathbf I)\) and fix the model and index \(i\) in advance. Use the normal-theory OLS results in Chapter 3 on the deleted dataset.
1. Standardize the deleted prediction error. Because \(Y_i\) is independent of the observations used to fit \(\hat{\boldsymbol\beta}_{(-i)}\),
\[ \begin{aligned} \mathbb E(d_i\mid\mathbf X)&=0,\\ \operatorname{Var}(d_i\mid\mathbf X) &=\sigma^2\!\left[1+\mathbf x_i^\top (\mathbf A-\mathbf x_i\mathbf x_i^\top)^{-1}\mathbf x_i\right]\\ &=\sigma^2\!\left(1+\frac{h_{ii}}{1-h_{ii}}\right) =\frac{\sigma^2}{1-h_{ii}}. \end{aligned} \]
The inverse identity follows from step 1 of the deletion derivation. As a linear combination of normal observations, \(Z_i=d_i\sqrt{1-h_{ii}}/\sigma\sim N(0,1)\).
2. Use an independent variance estimate. The deleted fit has \(n-1-p=\nu-1\) residual degrees of freedom, so
\[ U_i=\frac{(\nu-1)s_{(-i)}^2}{\sigma^2}\sim\chi^2_{\nu-1}. \]
Under normality, the deleted-fit coefficient estimate is independent of its SSE. Also, the held-out \(Y_i\) is independent of that entire deleted dataset. Hence \(d_i\), and therefore \(Z_i\), is independent of \(U_i\).
3. Form the t ratio. By the definition of the t distribution,
\[ \frac{Z_i}{\sqrt{U_i/(\nu-1)}} =\frac{d_i\sqrt{1-h_{ii}}}{s_{(-i)}} =\frac{e_i}{s_{(-i)}\sqrt{1-h_{ii}}} =t_i\sim t_{\nu-1}. \]
The internal residual \(r_i\) uses the full-fit \(s\), which shares the contribution from observation \(i\) with its numerator. This independence argument does not apply to \(r_i\); its bound \(|r_i|\le\sqrt{\nu}\) also rules out an exact t distribution, whose support is unbounded.