3 Distribution Theory of OLS and Inference

In the previous chapter, we studied least squares primarily from an algebraic and geometric point of view.

We obtained

\[ \hat{\boldsymbol{\beta}} = (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top\mathbf{Y}, \]

and showed that

\[ \hat{\mathbf{Y}} = \mathbf{H}\mathbf{Y}, \qquad \mathbf{e} = \mathbf{M}\mathbf{Y}, \]

where

\[ \mathbf{H} = \mathbf{X} (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top \]

and

\[ \mathbf{M} = \mathbf{I}_n-\mathbf{H}. \]

We now move from estimation to statistical inference.

Since \(\mathbf{Y}\) is random, quantities such as

\[ \hat{\boldsymbol{\beta}}, \qquad \hat{\mathbf{Y}}, \qquad \mathbf{e}, \qquad \mathrm{SSE} \]

are also random.

Our goal is to understand their sampling distributions and use them to quantify uncertainty.

Learning Objectives

By the end of this chapter, students should be able to:

derive the expectation and variance of the OLS estimator;

state the normal linear model;

derive the distribution of the OLS estimator under normality;

explain the distribution of the residual sum of squares;

explain why the residual degrees of freedom are \(n-p\);

estimate \(\sigma^2\) using the mean squared error;

explain why \(\hat{\boldsymbol{\beta}}\) and \(\mathrm{SSE}\) are independent;

construct confidence intervals for regression coefficients;

perform hypothesis tests for regression coefficients;

perform inference for linear combinations of regression coefficients;

distinguish inference for a mean response from prediction of a future observation;

interpret standard regression output from R.

Reading

3.1 Mean and Variance of the OLS Estimator

Before introducing normality, we can already study the expectation and variance of the OLS estimator.

Assume

\[ \mathbb{E}(\boldsymbol{\varepsilon}) = \mathbf{0} \]

and

\[ \operatorname{Var}(\boldsymbol{\varepsilon}) = \sigma^2\mathbf{I}_n. \]

Recall that

\[ \mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}. \]

Substituting this into the OLS estimator gives

\[\begin{align*} \hat{\boldsymbol{\beta}} &= (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top\mathbf{Y}\\ &= (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top \left( \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon} \right)\\ &= \boldsymbol{\beta} + (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top \boldsymbol{\varepsilon}. \end{align*}\]

Thus,

\[ \boxed{ \hat{\boldsymbol{\beta}} = \boldsymbol{\beta} + (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top \boldsymbol{\varepsilon}. } \]

This representation will be useful throughout the chapter.

Proposition 3.1 (Mean and Variance of OLS) Under

\[ \mathbb{E}(\boldsymbol{\varepsilon}) = \mathbf{0} \]

and

\[ \operatorname{Var}(\boldsymbol{\varepsilon}) = \sigma^2\mathbf{I}_n, \]

the OLS estimator satisfies

\[ \boxed{ \mathbb{E} \left( \hat{\boldsymbol{\beta}} \right) = \boldsymbol{\beta} } \]

and

\[ \boxed{ \operatorname{Var} \left( \hat{\boldsymbol{\beta}} \right) = \sigma^2 (\mathbf{X}^\top\mathbf{X})^{-1}. } \]

3.1.1 Why Is OLS Unbiased?

Using

\[ \hat{\boldsymbol{\beta}} = \boldsymbol{\beta} + (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top \boldsymbol{\varepsilon}, \]

we obtain

\[\begin{align*} \mathbb{E} \left( \hat{\boldsymbol{\beta}} \right) &= \boldsymbol{\beta} + (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top \mathbb{E} \left( \boldsymbol{\varepsilon} \right)\\ &= \boldsymbol{\beta}. \end{align*}\]

Thus,

\[ \boxed{ \mathbb{E} \left( \hat{\boldsymbol{\beta}} \right) = \boldsymbol{\beta}. } \]

OLS is therefore an unbiased estimator of \(\boldsymbol{\beta}\).

Intuitively, if we repeated the experiment many times using the same design matrix, the OLS estimates would fluctuate around the true parameter vector.

3.1.2 Variance of OLS

Similarly,

\[\begin{align*} \operatorname{Var} \left( \hat{\boldsymbol{\beta}} \right) &= \operatorname{Var} \left[ (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top \boldsymbol{\varepsilon} \right]\\ &= (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top \operatorname{Var} (\boldsymbol{\varepsilon}) \mathbf{X} (\mathbf{X}^\top\mathbf{X})^{-1}\\ &= \sigma^2 (\mathbf{X}^\top\mathbf{X})^{-1}. \end{align*}\]

Normality Is Not Needed Yet

The results

\[ \mathbb{E} \left( \hat{\boldsymbol{\beta}} \right) = \boldsymbol{\beta} \]

and

\[ \operatorname{Var} \left( \hat{\boldsymbol{\beta}} \right) = \sigma^2 (\mathbf{X}^\top\mathbf{X})^{-1} \]

do not require normally distributed errors.

They follow from the first two moment assumptions

\[ \mathbb{E}(\boldsymbol{\varepsilon}) = \mathbf{0} \]

and

\[ \operatorname{Var}(\boldsymbol{\varepsilon}) = \sigma^2\mathbf{I}_n. \]

Normality becomes important when we want exact finite sample distributions and inference.

Exercise 3.1 (Which Assumptions Are Needed?) Suppose

\[ \mathbb{E}(\boldsymbol{\varepsilon}) = \mathbf{0} \]

and

\[ \operatorname{Var}(\boldsymbol{\varepsilon}) = \sigma^2\mathbf{I}_n. \]

Do we need to assume that the errors are normally distributed to conclude that

\[ \mathbb{E} \left( \hat{\boldsymbol{\beta}} \right) = \boldsymbol{\beta}? \]

Solution 3.1. No.

Normality is not required for unbiasedness.

The result follows from

\[ \hat{\boldsymbol{\beta}} = \boldsymbol{\beta} + (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top \boldsymbol{\varepsilon} \]

and

\[ \mathbb{E}(\boldsymbol{\varepsilon}) = \mathbf{0}. \]

Normality will be needed later for exact finite sample inference.

3.2 The Normal Linear Model

To obtain exact sampling distributions, we now strengthen the assumptions.

Definition 3.1 (Normal Linear Model) The normal linear model assumes

\[ \boxed{ \mathbf{Y} \sim N_n \left( \mathbf{X}\boldsymbol{\beta}, \sigma^2\mathbf{I}_n \right). } \]

Equivalently,

\[ \boxed{ \boldsymbol{\varepsilon} \sim N_n \left( \mathbf{0}, \sigma^2\mathbf{I}_n \right). } \]

Under this assumption:

each response has a normal distribution;
the errors have common variance \(\sigma^2\);
the errors are independent;
exact finite sample distributions can be obtained for many regression statistics.

Why Does Normality Matter?

Least squares itself does not require normality.

Normality is introduced because it gives us exact distributions for

\[ \hat{\boldsymbol{\beta}}, \qquad \mathrm{SSE}, \qquad T, \qquad F. \]

These distributions allow us to construct exact finite sample confidence intervals and hypothesis tests.

Exercise 3.2 (What Changes When We Add Normality?) Suppose the errors satisfy

\[ \mathbb{E}(\boldsymbol{\varepsilon}) = \mathbf{0}, \qquad \operatorname{Var}(\boldsymbol{\varepsilon}) = \sigma^2\mathbf{I}_n, \]

but are not normally distributed.

Which of the following can still hold?

OLS can still be computed.
OLS can still be unbiased.
We can still have

\[ \operatorname{Var} \left( \hat{\boldsymbol{\beta}} \right) = \sigma^2 (\mathbf{X}^\top\mathbf{X})^{-1}. \]
Exact finite sample \(t\) and \(F\) distributions automatically hold.

Solution 3.2. Statements 1, 2, and 3 can still hold.

Statement 4 does not generally hold.

Normality is not required to define OLS or derive its mean and variance. Its main role here is to provide exact finite sample distributions for inference.

3.3 Distribution of the OLS Estimator

Recall that

\[ \hat{\boldsymbol{\beta}} = (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top\mathbf{Y}. \]

Thus, \(\hat{\boldsymbol{\beta}}\) is a linear transformation of \(\mathbf{Y}\).

A fundamental property of the multivariate normal distribution is that a linear transformation of a multivariate normal vector is again multivariate normal.

Theorem 3.1 (Distribution of the OLS Estimator) Under the normal linear model,

\[ \boxed{ \hat{\boldsymbol{\beta}} \sim N_p \left( \boldsymbol{\beta}, \sigma^2 (\mathbf{X}^\top\mathbf{X})^{-1} \right). } \]

Let

\[ \mathbf{C} = (\mathbf{X}^\top\mathbf{X})^{-1}. \]

Then

\[ \operatorname{Var} \left( \hat{\boldsymbol{\beta}} \right) = \sigma^2\mathbf{C}. \]

If \(c_{jj}\) denotes the \(j\)th diagonal element of \(\mathbf{C}\), then

\[ \hat{\beta}_j \sim N \left( \beta_j, \sigma^2c_{jj} \right). \]

Therefore,

\[ \boxed{ \frac{ \hat{\beta}_j-\beta_j }{ \sigma\sqrt{c_{jj}} } \sim N(0,1). } \]

Exercise 3.3 (Reading the Covariance Matrix) Let

\[ \mathbf{C} = (\mathbf{X}^\top\mathbf{X})^{-1}. \]

If \(c_{jj}\) is the \(j\)th diagonal element of \(\mathbf{C}\):

What is \(\operatorname{Var}(\hat{\beta}_j)\)?
What happens to the uncertainty of \(\hat{\beta}_j\) if \(c_{jj}\) becomes larger?

Solution 3.3. The variance is

\[ \boxed{ \operatorname{Var}(\hat{\beta}_j) = \sigma^2c_{jj}. } \]

Therefore, a larger \(c_{jj}\) means greater sampling variability and less precise estimation of \(\beta_j\).

Thus, the precision of OLS depends not only on \(\sigma^2\), but also on the design matrix \(\mathbf{X}\).

There is still one difficulty.

The quantity

\[ \frac{ \hat{\beta}_j-\beta_j }{ \sigma\sqrt{c_{jj}} } \]

depends on the unknown parameter \(\sigma\).

Before performing inference, we need to estimate \(\sigma^2\).

Exercise 3.4 (Why Can We Not Use the Normal Statistic Directly?) We have

\[ \frac{ \hat{\beta}_j-\beta_j }{ \sigma\sqrt{c_{jj}} } \sim N(0,1). \]

Why can we usually not use this statistic directly for inference?

Solution 3.4. Because the error standard deviation \(\sigma\) is unknown.

We therefore need to estimate \(\sigma\) from the data.

Replacing \(\sigma\) with an estimator will lead naturally to the \(t\) distribution.

3.4 Residual Sum of Squares and Estimation of \(\sigma^2\)

Recall from Chapter 2 that

\[ \mathbf{e} = \mathbf{M}\mathbf{Y}, \]

where

\[ \mathbf{M} = \mathbf{I}_n-\mathbf{H}. \]

Since

\[ \mathbf{M}\mathbf{X} = \mathbf{0}, \]

we have

\[\begin{align*} \mathbf{e} &= \mathbf{M}\mathbf{Y}\\ &= \mathbf{M} \left( \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon} \right)\\ &= \mathbf{M}\boldsymbol{\varepsilon}. \end{align*}\]

Thus,

\[ \boxed{ \mathbf{e} = \mathbf{M}\boldsymbol{\varepsilon}. } \]

The residual vector therefore contains the part of the random error that lies outside the model space.

3.4.1 Residual Sum of Squares

The residual sum of squares is

\[ \mathrm{SSE} = \mathbf{e}^\top\mathbf{e}. \]

Since

\[ \mathbf{e} = \mathbf{M}\boldsymbol{\varepsilon}, \]

and \(\mathbf{M}\) is symmetric and idempotent,

\[\begin{align*} \mathrm{SSE} &= \boldsymbol{\varepsilon}^\top \mathbf{M}^\top \mathbf{M} \boldsymbol{\varepsilon}\\ &= \boldsymbol{\varepsilon}^\top \mathbf{M} \boldsymbol{\varepsilon}. \end{align*}\]

Therefore,

\[ \boxed{ \mathrm{SSE} = \boldsymbol{\varepsilon}^\top \mathbf{M} \boldsymbol{\varepsilon}. } \]

3.4.2 Distribution of SSE

Under the normal linear model,

\[ \frac{\boldsymbol{\varepsilon}}{\sigma} \sim N_n \left( \mathbf{0}, \mathbf{I}_n \right). \]

Let

\[ \mathbf{Z} = \frac{\boldsymbol{\varepsilon}}{\sigma}. \]

Then

\[ \frac{\mathrm{SSE}}{\sigma^2} = \mathbf{Z}^\top \mathbf{M} \mathbf{Z}. \]

Recall that \(\mathbf{M}\) is symmetric and idempotent and

\[ \operatorname{rank}(\mathbf{M}) = n-p. \]

Therefore,

Theorem 3.2 (Distribution of the Residual Sum of Squares) Under the normal linear model,

\[ \boxed{ \frac{\mathrm{SSE}}{\sigma^2} \sim \chi^2_{n-p}. } \]

3.4.3 Why Are the Degrees of Freedom \(n-p\)?

Recall that

\[ \operatorname{rank}(\mathbf{H}) = p \]

and

\[ \operatorname{rank}(\mathbf{M}) = n-p. \]

The fitted vector lies in

\[ \mathcal{C}(\mathbf{X}), \]

which has dimension \(p\).

The residual vector lies in

\[ \mathcal{C}(\mathbf{X})^\perp, \]

which has dimension

\[ n-p. \]

Therefore, the residual variation has

\[ \boxed{ n-p } \]

degrees of freedom.

Geometric Interpretation of Degrees of Freedom

The response space can be decomposed as

\[ \mathbb{R}^n = \mathcal{C}(\mathbf{X}) \oplus \mathcal{C}(\mathbf{X})^\perp. \]

The fitted component occupies \(p\) dimensions.

The remaining

\[ n-p \]

dimensions are available for residual variation.

This provides a geometric explanation for why estimating \(p\) regression parameters costs \(p\) degrees of freedom.

Exercise 3.5 (Degrees of Freedom) Suppose a regression model has

\[ n=20 \]

observations and

\[ p=3 \]

regression parameters.

What is the residual degrees of freedom?
Why is it not \(20\)?

Solution 3.5. The residual degrees of freedom are

\[ n-p = 20-3 = 17. \]

It is not \(20\) because fitting the regression model uses \(p=3\) dimensions of the response space.

The residual vector therefore lies in a space of dimension

\[ 20-3=17. \]

3.4.4 Estimating \(\sigma^2\)

Since a chi-square random variable with \(\nu\) degrees of freedom has expectation \(\nu\),

\[ \mathbb{E} \left( \frac{\mathrm{SSE}}{\sigma^2} \right) = n-p. \]

Therefore,

\[ \mathbb{E}(\mathrm{SSE}) = (n-p)\sigma^2. \]

This gives us an unbiased estimator of the error variance.

Definition 3.2 (Mean Squared Error) The mean squared error is defined as

\[ \boxed{ \mathrm{MSE} = \hat{\sigma}^2 = \frac{\mathrm{SSE}}{n-p}. } \]

It satisfies

\[ \mathbb{E} \left( \hat{\sigma}^2 \right) = \sigma^2. \]

The estimated error standard deviation is

\[ \boxed{ \hat{\sigma} = \sqrt{\mathrm{MSE}}. } \]

Exercise 3.6 (Estimate the Error Variance) Suppose

\[ n=20, \qquad p=3, \qquad \mathrm{SSE}=34. \]

Compute

\[ \hat{\sigma}^2 \]

and

\[ \hat{\sigma}. \]

Solution 3.6. The residual degrees of freedom are

\[ n-p = 17. \]

Therefore,

\[ \hat{\sigma}^2 = \frac{34}{17} = 2. \]

Hence,

\[ \boxed{ \hat{\sigma} = \sqrt{2} \approx 1.414. } \]

3.5 Independence of OLS and Residual Variation

A crucial result under the normal linear model is

\[ \boxed{ \hat{\boldsymbol{\beta}} \quad\text{and}\quad \mathrm{SSE} \quad\text{are independent}. } \]

This result is closely connected to the projection geometry from the previous chapter.

3.5.1 Why Does Independence Occur?

Recall that

\[ \hat{\mathbf{Y}} = \mathbf{H}\mathbf{Y} \]

and

\[ \mathbf{e} = \mathbf{M}\mathbf{Y}. \]

We also know that

\[ \mathbf{H}\mathbf{M} = \mathbf{0}. \]

Thus, the fitted component and the residual component come from orthogonal parts of the response vector.

Geometry Becomes Probability

In the previous chapter, we learned that

\[ \hat{\mathbf{Y}} \perp \mathbf{e}. \]

This is a geometric statement.

Under the normal linear model, these components are jointly normal.

For jointly normal random vectors, zero covariance implies independence.

Thus, under normality, geometric orthogonality leads to probabilistic independence.

More formally,

\[ \hat{\boldsymbol{\beta}} - \boldsymbol{\beta} = (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top \boldsymbol{\varepsilon} \]

and

\[ \mathbf{e} = \mathbf{M}\boldsymbol{\varepsilon}. \]

Therefore,

\[\begin{align*} \operatorname{Cov} \left( \hat{\boldsymbol{\beta}}, \mathbf{e} \right) &= \sigma^2 (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top \mathbf{M}\\ &= \mathbf{0}, \end{align*}\]

because

\[ \mathbf{X}^\top\mathbf{M} = \mathbf{0}. \]

Since the two random vectors are jointly normal,

\[ \hat{\boldsymbol{\beta}} \perp\!\!\!\perp \mathbf{e}. \]

Since

\[ \mathrm{SSE} = \mathbf{e}^\top\mathbf{e} \]

is a function of \(\mathbf{e}\),

\[ \boxed{ \hat{\boldsymbol{\beta}} \perp\!\!\!\perp \mathrm{SSE}. } \]

Exercise 3.7 (Does Orthogonal Always Mean Independent?) We know that

\[ \mathbf{H}\mathbf{M} = \mathbf{0}. \]

Does the orthogonality between the fitted and residual components automatically imply that they are independent?

Why is normality important?

Solution 3.7. No.

In general, zero covariance or orthogonality does not imply independence.

Under the normal linear model, the fitted and residual components are jointly normal.

For jointly normal random vectors, zero covariance implies independence.

Thus, normality is what allows us to move from orthogonality to independence.

3.6 Inference for a Single Regression Coefficient

For the \(j\)th regression coefficient,

\[ \hat{\beta}_j \sim N \left( \beta_j, \sigma^2c_{jj} \right). \]

If \(\sigma\) were known, we could use

\[ \frac{ \hat{\beta}_j-\beta_j }{ \sigma\sqrt{c_{jj}} } \sim N(0,1). \]

In practice, \(\sigma\) is unknown.

We replace it with

\[ \hat{\sigma} = \sqrt{\frac{\mathrm{SSE}}{n-p}}. \]

3.6.1 Standard Error

The estimated standard deviation of \(\hat{\beta}_j\) is

Definition 3.3 (Standard Error of an OLS Coefficient) The standard error of \(\hat{\beta}_j\) is

\[ \boxed{ \operatorname{SE} \left( \hat{\beta}_j \right) = \hat{\sigma}\sqrt{c_{jj}}. } \]

3.6.2 Why Does the \(t\) Distribution Appear?

We know that

\[ Z = \frac{ \hat{\beta}_j-\beta_j }{ \sigma\sqrt{c_{jj}} } \sim N(0,1). \]

We also know that

\[ U = \frac{\mathrm{SSE}}{\sigma^2} \sim \chi^2_{n-p}. \]

In addition,

\[ Z \quad\text{and}\quad U \]

are independent.

Therefore,

\[ \frac{ Z }{ \sqrt{U/(n-p)} } \sim t_{n-p}. \]

Substituting the regression quantities gives

Theorem 3.3 (\(t\) Distribution for an OLS Coefficient) Under the normal linear model,

\[ \boxed{ T = \frac{ \hat{\beta}_j-\beta_j }{ \hat{\sigma}\sqrt{c_{jj}} } \sim t_{n-p}. } \]

Normal vs. t Distribution

If \(\sigma\) were known, we would use a normal distribution.

Because \(\sigma\) is unknown and must be estimated from the data, we introduce additional uncertainty.

The \(t\) distribution accounts for this additional uncertainty.

As the degrees of freedom increase,

\[ t_\nu \]

approaches

\[ N(0,1). \]

Exercise 3.8 (Why Do We Use a \(t\) Distribution?) Suppose

\[ \hat{\beta}_j \sim N \left( \beta_j, \sigma^2c_{jj} \right). \]

Why do we use a \(t\) distribution rather than a standard normal distribution when performing inference for \(\beta_j\)?

Solution 3.8. Because \(\sigma\) is unknown.

We replace \(\sigma\) with the estimate \(\hat{\sigma}\) obtained from the residual sum of squares.

This additional estimation uncertainty changes the standardized statistic from a normal distribution to a \(t\) distribution.

3.6.3 Confidence Interval for \(\beta_j\)

A \(100(1-\alpha)\%\) confidence interval for \(\beta_j\) is

\[ \boxed{ \hat{\beta}_j \pm t_{1-\alpha/2,\;n-p} \operatorname{SE} \left( \hat{\beta}_j \right). } \]

Equivalently,

\[ \boxed{ \hat{\beta}_j \pm t_{1-\alpha/2,\;n-p} \hat{\sigma}\sqrt{c_{jj}}. } \]

3.6.4 Hypothesis Test for \(\beta_j\)

Suppose we want to test

\[ H_0: \beta_j = \beta_{j,0} \]

against

\[ H_1: \beta_j \neq \beta_{j,0}. \]

The test statistic is

\[ \boxed{ T = \frac{ \hat{\beta}_j-\beta_{j,0} }{ \operatorname{SE} (\hat{\beta}_j) }. } \]

Under \(H_0\),

\[ T \sim t_{n-p}. \]

For a two sided test at significance level \(\alpha\), reject \(H_0\) if

\[ |T| > t_{1-\alpha/2,\;n-p}. \]

Exercise 3.9 (Confidence Interval and Hypothesis Test) Suppose we test

\[ H_0: \beta_j=0 \]

against

\[ H_1: \beta_j\neq0 \]

at significance level

\[ \alpha=0.05. \]

What should happen if the corresponding 95% confidence interval for \(\beta_j\) does not contain zero?

Solution 3.9. We reject

\[ H_0: \beta_j=0 \]

at the 5% significance level.

A two sided hypothesis test at level \(\alpha\) and a \(100(1-\alpha)\%\) confidence interval give equivalent conclusions.

3.7 Inference for Linear Combinations

Sometimes the quantity of interest is not a single coefficient.

Suppose we are interested in

\[ \mathbf{a}^\top \boldsymbol{\beta}, \]

where \(\mathbf{a}\) is a fixed \(p\times1\) vector.

Since

\[ \hat{\boldsymbol{\beta}} \sim N_p \left( \boldsymbol{\beta}, \sigma^2 (\mathbf{X}^\top\mathbf{X})^{-1} \right), \]

we have

\[ \mathbf{a}^\top \hat{\boldsymbol{\beta}} \sim N \left( \mathbf{a}^\top \boldsymbol{\beta}, \sigma^2 \mathbf{a}^\top (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{a} \right). \]

Therefore,

\[ \boxed{ \frac{ \mathbf{a}^\top\hat{\boldsymbol{\beta}} - \mathbf{a}^\top\boldsymbol{\beta} }{ \hat{\sigma} \sqrt{ \mathbf{a}^\top (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{a} } } \sim t_{n-p}. } \]

A \(100(1-\alpha)\%\) confidence interval for \(\mathbf{a}^\top\boldsymbol{\beta}\) is

\[ \boxed{ \mathbf{a}^\top\hat{\boldsymbol{\beta}} \pm t_{1-\alpha/2,\;n-p} \hat{\sigma} \sqrt{ \mathbf{a}^\top (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{a} }. } \]

Why Is This Useful?

Many regression questions can be expressed as linear combinations of the regression coefficients.

Examples include:

a single coefficient;
the difference between two coefficients;
the mean response at a particular predictor value.

Thus, inference for a single regression coefficient is a special case of inference for a linear combination.

Exercise 3.10 (A Single Coefficient as a Linear Combination) Suppose

\[ \boldsymbol{\beta} = \begin{pmatrix} \beta_0\\ \beta_1\\ \beta_2 \end{pmatrix}. \]

What vector \(\mathbf{a}\) would you choose so that

\[ \mathbf{a}^\top\boldsymbol{\beta} = \beta_1? \]

Solution 3.10. Choose

\[ \boxed{ \mathbf{a} = \begin{pmatrix} 0\\ 1\\ 0 \end{pmatrix}. } \]

Then

\[ \mathbf{a}^\top\boldsymbol{\beta} = \beta_1. \]

Thus, inference for an individual coefficient is a special case of inference for a linear combination.

3.8 General Linear Hypotheses

Sometimes we want to test several linear restrictions simultaneously.

Suppose

\[ H_0: \mathbf{C}\boldsymbol{\beta} = \mathbf{d}, \]

where \(\mathbf{C}\) is an \(r\times p\) matrix of rank \(r\) and \(\mathbf{d}\) is an \(r\times1\) vector.

Definition 3.4 (General Linear Hypothesis) A hypothesis of the form

\[ H_0: \mathbf{C}\boldsymbol{\beta} = \mathbf{d} \]

is called a general linear hypothesis.

It allows several linear restrictions on the regression coefficients to be tested simultaneously.

The corresponding test statistic is

\[ F = \frac{ (\mathbf{C}\hat{\boldsymbol{\beta}}-\mathbf{d})^\top \left[ \mathbf{C} (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{C}^\top \right]^{-1} (\mathbf{C}\hat{\boldsymbol{\beta}}-\mathbf{d}) }{ r\hat{\sigma}^2 }. \]

Under \(H_0\),

\[ \boxed{ F \sim F_{r,\;n-p}. } \]

3.8.1 Relationship Between \(t\) and \(F\)

When only one restriction is being tested,

\[ r=1. \]

In this case,

\[ \boxed{ F=T^2. } \]

Thus, the familiar \(t\) test for one coefficient is a special case of the general \(F\) test.

We will study \(F\) tests more thoroughly in the next chapter when we develop the ANOVA decomposition and compare nested regression models.

Exercise 3.11 (\(t\) Test or \(F\) Test?) Suppose we want to test only

\[ H_0: \beta_2=0. \]

Could we use either a \(t\) test or an \(F\) test?

How are the two test statistics related?

Solution 3.11. Yes.

Because there is only one linear restriction, either test can be used.

The corresponding statistics satisfy

\[ \boxed{ F=T^2. } \]

Therefore, the two tests produce the same conclusion.

3.9 Inference for the Mean Response

Suppose we want to estimate the mean response at a predictor vector

\[ \mathbf{x}_0. \]

The true mean response is

\[ \mu(\mathbf{x}_0) = \mathbf{x}_0^\top \boldsymbol{\beta}. \]

Its estimator is

\[ \hat{\mu}(\mathbf{x}_0) = \mathbf{x}_0^\top \hat{\boldsymbol{\beta}}. \]

Since this is a linear combination of \(\hat{\boldsymbol{\beta}}\),

\[ \operatorname{Var} \left( \hat{\mu}(\mathbf{x}_0) \right) = \sigma^2 \mathbf{x}_0^\top (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{x}_0. \]

Thus,

\[ \frac{ \mathbf{x}_0^\top\hat{\boldsymbol{\beta}} - \mathbf{x}_0^\top\boldsymbol{\beta} }{ \hat{\sigma} \sqrt{ \mathbf{x}_0^\top (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{x}_0 } } \sim t_{n-p}. \]

Therefore, a \(100(1-\alpha)\%\) confidence interval for the mean response is

\[ \boxed{ \mathbf{x}_0^\top\hat{\boldsymbol{\beta}} \pm t_{1-\alpha/2,\;n-p} \hat{\sigma} \sqrt{ \mathbf{x}_0^\top (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{x}_0 }. } \]

3.10 Prediction for a New Observation

Now suppose we want to predict one new observation at the same predictor value \(\mathbf{x}_0\):

\[ Y_{\mathrm{new}} = \mathbf{x}_0^\top \boldsymbol{\beta} + \varepsilon_{\mathrm{new}}, \]

where

\[ \varepsilon_{\mathrm{new}} \sim N(0,\sigma^2) \]

and is independent of the original data.

The predictor is

\[ \hat{Y}_{\mathrm{new}} = \mathbf{x}_0^\top \hat{\boldsymbol{\beta}}. \]

The prediction error is

\[ Y_{\mathrm{new}} - \mathbf{x}_0^\top \hat{\boldsymbol{\beta}}. \]

Its variance is

\[ \operatorname{Var} \left( Y_{\mathrm{new}} - \mathbf{x}_0^\top \hat{\boldsymbol{\beta}} \right) = \sigma^2 \left[ 1+ \mathbf{x}_0^\top (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{x}_0 \right]. \]

Therefore, a \(100(1-\alpha)\%\) prediction interval is

\[ \boxed{ \mathbf{x}_0^\top\hat{\boldsymbol{\beta}} \pm t_{1-\alpha/2,\;n-p} \hat{\sigma} \sqrt{ 1+ \mathbf{x}_0^\top (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{x}_0 }. } \]

Mean Response vs. New Observation

The two intervals answer different questions.

Goal	Source of uncertainty
Estimate a mean response	Uncertainty in estimating the regression mean
Predict a new observation	Uncertainty in the regression mean plus new random variation

The additional \(1\) in

\[ 1+ \mathbf{x}_0^\top (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{x}_0 \]

represents the variability of the new observation itself.

Therefore, at the same \(\mathbf{x}_0\) and confidence level, the prediction interval is wider than the confidence interval for the mean response.

Exercise 3.12 (Mean Response or Prediction?) Consider the following questions.

What is the average stopping distance of all cars travelling at 50 mph?
What stopping distance should we expect for one particular future car travelling at 50 mph?

Which question requires a confidence interval for the mean response, and which requires a prediction interval?

Solution 3.12. The first question concerns the mean response, so we use a confidence interval for the regression mean.

The second question concerns one future observation, so we use a prediction interval.

The prediction interval is wider because it also includes the random variation associated with the future observation.

3.11 Worked Example

Consider again the simple regression dataset from Chapter 2:

\[ \begin{array}{c|cccc} x_i & 0 & 1 & 2 & 3\\ \hline Y_i & 1 & 3 & 3 & 5 \end{array}. \]

The design matrix and response vector are

\[ \mathbf{X} = \begin{pmatrix} 1 & 0\\ 1 & 1\\ 1 & 2\\ 1 & 3 \end{pmatrix}, \qquad \mathbf{Y} = \begin{pmatrix} 1\\ 3\\ 3\\ 5 \end{pmatrix}. \]

From the previous chapter,

\[ \hat{\boldsymbol{\beta}} = \begin{pmatrix} 1.2\\ 1.2 \end{pmatrix}, \]

and

\[ \hat{\mathbf{Y}} = \begin{pmatrix} 1.2\\ 2.4\\ 3.6\\ 4.8 \end{pmatrix}. \]

Therefore,

\[ \mathbf{e} = \begin{pmatrix} -0.2\\ 0.6\\ -0.6\\ 0.2 \end{pmatrix}. \]

3.11.1 Estimate the Error Variance

The residual sum of squares is

\[\begin{align*} \mathrm{SSE} &= \mathbf{e}^\top\mathbf{e}\\ &= (-0.2)^2 + 0.6^2 + (-0.6)^2 + 0.2^2\\ &= 0.8. \end{align*}\]

Since

\[ n=4 \]

and

\[ p=2, \]

the residual degrees of freedom are

\[ n-p=2. \]

Thus,

\[ \boxed{ \hat{\sigma}^2 = \frac{0.8}{2} = 0.4. } \]

Therefore,

\[ \boxed{ \hat{\sigma} = \sqrt{0.4} \approx 0.632. } \]

3.11.2 Standard Error of the Slope

We have

\[ (\mathbf{X}^\top\mathbf{X})^{-1} = \frac{1}{20} \begin{pmatrix} 14 & -6\\ -6 & 4 \end{pmatrix}. \]

For the slope,

\[ c_{22} = \frac{4}{20} = 0.2. \]

Therefore,

\[ \operatorname{SE} (\hat{\beta}_1) = \sqrt{ 0.4(0.2) } = \sqrt{0.08} \approx 0.283. \]

3.11.3 Test the Slope

Consider

\[ H_0: \beta_1=0 \]

versus

\[ H_1: \beta_1\neq0. \]

The test statistic is

\[ T = \frac{ 1.2 }{ 0.283 } \approx 4.243. \]

The residual degrees of freedom are

\[ 2. \]

For a 5% two sided test,

\[ t_{0.975,2} \approx 4.303. \]

Since

\[ |4.243| < 4.303, \]

we fail to reject \(H_0\) at the 5% significance level.

What Does This Example Show?

The estimated slope

\[ \hat{\beta}_1=1.2 \]

is relatively large.

However, there are only four observations, leaving only two residual degrees of freedom.

Therefore, there is substantial uncertainty in the estimate.

A large estimated effect does not automatically imply strong evidence against the null hypothesis.

3.11.4 Confidence Interval for the Slope

A 95% confidence interval is

\[ 1.2 \pm 4.303(0.283), \]

which gives approximately

\[ \boxed{ (-0.017,\;2.417). } \]

The interval contains zero, which agrees with the hypothesis test.

Exercise 3.13 (Check the Connection) Without performing another hypothesis test, what conclusion can you draw from the fact that the 95% confidence interval

\[ (-0.017,\;2.417) \]

contains zero?

Solution 3.13. We fail to reject

\[ H_0: \beta_1=0 \]

at the 5% significance level.

This agrees with the previous \(t\) test.

3.11.5 Mean Response at \(x_0=2\)

For simple linear regression with an intercept,

\[ \mathbf{x}_0 = \begin{pmatrix} 1\\ 2 \end{pmatrix}. \]

The estimated mean response is

\[ \hat{\mu}(\mathbf{x}_0) = \mathbf{x}_0^\top \hat{\boldsymbol{\beta}} = 3.6. \]

Also,

\[ \mathbf{x}_0^\top (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{x}_0 = 0.3. \]

Therefore,

\[ \operatorname{SE} \left( \hat{\mu}(\mathbf{x}_0) \right) = \sqrt{ 0.4(0.3) } \approx 0.346. \]

The 95% confidence interval for the mean response is approximately

\[ \boxed{ (2.110,\;5.090). } \]

3.11.6 Prediction at \(x_0=2\)

For one new observation,

\[ \operatorname{SE}_{\mathrm{pred}} = \sqrt{ 0.4(1+0.3) } \approx 0.721. \]

Therefore, the 95% prediction interval is approximately

\[ \boxed{ (0.497,\;6.703). } \]

Notice that the prediction interval is substantially wider than the confidence interval for the mean response.

3.12 R Demonstration

We can reproduce the previous calculations using R.

x <- c(0, 1, 2, 3)
y <- c(1, 3, 3, 5)

fit <- lm(y ~ x)

summary(fit)


Call:
lm(formula = y ~ x)

Residuals:
   1    2    3    4 
-0.2  0.6 -0.6  0.2 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)  
(Intercept)   1.2000     0.5292   2.268   0.1515  
x             1.2000     0.2828   4.243   0.0513 .
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.6325 on 2 degrees of freedom
Multiple R-squared:    0.9, Adjusted R-squared:   0.85 
F-statistic:    18 on 1 and 2 DF,  p-value: 0.05132

3.12.1 Estimated Coefficients

coef(fit)

(Intercept)           x 
        1.2         1.2

3.12.2 Residual Degrees of Freedom and Error Variance

df.residual(fit)

[1] 2

sigma(fit)

[1] 0.6324555

sigma(fit)^2

[1] 0.4

3.12.3 Covariance Matrix and Standard Errors

vcov(fit)

            (Intercept)     x
(Intercept)        0.28 -0.12
x                 -0.12  0.08

sqrt(diag(vcov(fit)))

(Intercept)           x 
  0.5291503   0.2828427

3.12.4 Verify the Formulas Using Matrix Operations

X <- model.matrix(fit)
Y <- matrix(y, ncol = 1)

n <- nrow(X)
p <- ncol(X)

beta_hat <-
  solve(t(X) %*% X) %*%
  t(X) %*%
  Y

e <- Y - X %*% beta_hat

SSE <- as.numeric(t(e) %*% e)

sigma2_hat <- SSE / (n - p)

V_beta_hat <-
  sigma2_hat *
  solve(t(X) %*% X)

SE_beta_hat <-
  sqrt(diag(V_beta_hat))

beta_hat

            [,1]
(Intercept)  1.2
x            1.2

SSE

[1] 0.8

sigma2_hat

[1] 0.4

V_beta_hat

            (Intercept)     x
(Intercept)        0.28 -0.12
x                 -0.12  0.08

SE_beta_hat

(Intercept)           x 
  0.5291503   0.2828427

3.12.5 Hypothesis Tests and Confidence Intervals

R automatically reports coefficient level \(t\) tests:

summary(fit)$coefficients

            Estimate Std. Error  t value  Pr(>|t|)
(Intercept)      1.2  0.5291503 2.267787 0.1514719
x                1.2  0.2828427 4.242641 0.0513167

Confidence intervals for the regression coefficients can be obtained using

confint(fit)

                  2.5 %   97.5 %
(Intercept) -1.07674982 3.476750
x           -0.01697397 2.416974

3.12.6 Mean Response and Prediction

\[ x_0=2, \]

use

newdat <- data.frame(x = 2)

predict(
  fit,
  newdata = newdat,
  interval = "confidence"
)

  fit      lwr      upr
1 3.6 2.109517 5.090483

predict(
  fit,
  newdata = newdat,
  interval = "prediction"
)

  fit      lwr      upr
1 3.6 0.497313 6.702687

The first interval estimates the mean response.

The second interval predicts one future observation.

3.13 Reading Standard Regression Output

For

summary(fit)

the coefficient table contains four important columns:

Column	Interpretation
`Estimate`	Estimated regression coefficient
`Std. Error`	Estimated standard deviation of the coefficient estimator
`t value`	\(t\) statistic for testing whether the coefficient equals zero
`Pr(>\|t\|)`	Two sided \(p\) value for the test

The output also includes:

the residual standard error;
the residual degrees of freedom;
\(R^2\);
adjusted \(R^2\);
an overall \(F\) test.

We will study \(R^2\), the ANOVA decomposition, and the overall \(F\) test more carefully in the next chapter.

Exercise 3.14 (Reading an R Regression Table) Suppose R reports

	Estimate	Std. Error	t value	Pr(>\|t\|)
\(x_1\)	2.40	0.60	4.00	0.001

Answer the following.

What is \(\hat{\beta}_1\)?
What is \(\operatorname{SE}(\hat{\beta}_1)\)?
What null hypothesis is being tested by default?

Solution 3.14. The estimate is

\[ \hat{\beta}_1 = 2.40. \]

The standard error is

\[ \operatorname{SE} (\hat{\beta}_1) = 0.60. \]

The default coefficient level test is

\[ H_0: \beta_1=0. \]

The reported \(t\) statistic is

\[ \frac{2.40}{0.60} = 4. \]

3.14 Practice at Home

Starting from

\[ \hat{\boldsymbol{\beta}} = \boldsymbol{\beta} + (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top \boldsymbol{\varepsilon}, \]

derive

\[ \mathbb{E} \left( \hat{\boldsymbol{\beta}} \right) = \boldsymbol{\beta} \]

and

\[ \operatorname{Var} \left( \hat{\boldsymbol{\beta}} \right) = \sigma^2 (\mathbf{X}^\top\mathbf{X})^{-1}. \]
Explain why normality is not required for OLS to be unbiased but is useful for exact finite sample inference.
Explain geometrically why the residual degrees of freedom are

\[ n-p. \]
Explain why

\[ \hat{\boldsymbol{\beta}} \quad\text{and}\quad \mathrm{SSE} \]

are independent under the normal linear model.
Derive the \(t\) statistic for testing

\[ H_0: \beta_j = \beta_{j,0}. \]
Explain the connection between a two sided hypothesis test and the corresponding confidence interval.
Explain the difference between a confidence interval for

\[ \mathbf{x}_0^\top \boldsymbol{\beta} \]

and a prediction interval for a future observation at \(\mathbf{x}_0\).
Fit a simple regression model in R and identify:
- the coefficient estimates;
- their standard errors;
- the \(t\) statistics;
- the \(p\) values;
- the residual standard error;
- the residual degrees of freedom.

Summary

This chapter moved from the geometry of least squares to the probability of least squares.

Under the first two moment assumptions,

\[ \mathbb{E} \left( \hat{\boldsymbol{\beta}} \right) = \boldsymbol{\beta} \]

and

\[ \operatorname{Var} \left( \hat{\boldsymbol{\beta}} \right) = \sigma^2 (\mathbf{X}^\top\mathbf{X})^{-1}. \]

Under the additional normality assumption,

\[ \boxed{ \hat{\boldsymbol{\beta}} \sim N_p \left( \boldsymbol{\beta}, \sigma^2 (\mathbf{X}^\top\mathbf{X})^{-1} \right). } \]

The residual sum of squares satisfies

\[ \boxed{ \frac{\mathrm{SSE}}{\sigma^2} \sim \chi^2_{n-p}. } \]

This leads to the unbiased estimator

\[ \boxed{ \hat{\sigma}^2 = \frac{\mathrm{SSE}}{n-p}. } \]

The independence of

\[ \hat{\boldsymbol{\beta}} \]

and

\[ \mathrm{SSE} \]

allows us to replace the unknown \(\sigma\) with \(\hat{\sigma}\) and obtain exact \(t\) inference.

The same framework extends naturally to:

linear combinations of regression coefficients;
general linear hypotheses;
confidence intervals for the mean response;
prediction intervals for future observations.

In the next chapter, we will use these results to develop the ANOVA decomposition, the overall \(F\) test, and comparisons of nested regression models.

Side Fact: Distribution Results Used in This Chapter

\[ \mathbf{Z} \sim N_n \left( \mathbf{0}, \mathbf{I}_n \right) \]

and \(\mathbf{A}\) is symmetric and idempotent with

\[ \operatorname{rank}(\mathbf{A}) = r, \]

then

\[ \boxed{ \mathbf{Z}^\top \mathbf{A} \mathbf{Z} \sim \chi^2_r. } \]

\[ Z \sim N(0,1), \qquad U \sim \chi^2_\nu, \]

and \(Z\) and \(U\) are independent, then

\[ \boxed{ \frac{ Z }{ \sqrt{U/\nu} } \sim t_\nu. } \]

\[ U_1 \sim \chi^2_r, \qquad U_2 \sim \chi^2_\nu, \]

and \(U_1\) and \(U_2\) are independent, then

\[ \boxed{ \frac{ U_1/r }{ U_2/\nu } \sim F_{r,\nu}. } \]

These results provide the distributional foundation for regression inference.