Assignment 1

This assignment focuses on the foundations of the linear model:

matrix representation,
least squares estimation,
projection
basic inference.

Instructions

Submit both your source file (.qmd or .Rmd) and the rendered HTML or PDF file.
Name your source file as A1_LastName_FirstName.qmd or A1_LastName_FirstName.Rmd.
Clearly label every question and subpart.
Type all mathematical work using LaTeX.
Include all R code used to obtain your answers.
Your document must render from beginning to end without error.
Do not submit screenshots of code, R output, or mathematical derivations.
Round numerical answers to three decimal places when appropriate.
For True/False questions, enter only T or F.
For short-answer questions, one or two clear sentences are sufficient unless a derivation is requested.

The total value of the assignment is [40].

Question 1: Matrix Representation of the Linear Model [8]

Consider the simple linear regression model

\[ Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \qquad i=1,\ldots,5, \]

with the following observed data:

\(i\)	\(x_i\)	\(Y_i\)
1	0	2
2	1	4
3	2	5
4	3	8
5	4	9

(a) [2]

Which of the following is the correct design matrix \(\mathbf{X}\)?

\[ \begin{pmatrix} 0 & 2\\ 1 & 4\\ 2 & 5\\ 3 & 8\\ 4 & 9 \end{pmatrix} \]

\[ \begin{pmatrix} 1 & 0\\ 1 & 1\\ 1 & 2\\ 1 & 3\\ 1 & 4 \end{pmatrix} \]

\[ \begin{pmatrix} 0 & 1\\ 1 & 1\\ 2 & 1\\ 3 & 1\\ 4 & 1 \end{pmatrix} \]

\[ \begin{pmatrix} 1 & 2\\ 1 & 4\\ 1 & 5\\ 1 & 8\\ 1 & 9 \end{pmatrix} \]

Solution. Solution: B

(b) [2]

Fill in the following table.

Quantity	Dimension / Rank
\(\mathbf{Y}\)	`___`
\(\mathbf{X}\)	`___`
\(\boldsymbol{\beta}\)	`___`
\(\operatorname{rank}(\mathbf{X})\)	`___`

Solution. Solution:

Quantity	Dimension / Rank
\(\mathbf{Y}\)	\(5\times 1\)
\(\mathbf{X}\)	\(5\times 2\)
\(\boldsymbol{\beta}\)	\(2\times 1\)
\(\operatorname{rank}(\mathbf{X})\)	\(2\)

(c) [2]

Which pair is correct?

\[ \mathbf{X}^\top\mathbf{X} = \begin{pmatrix} 5 & 10\\ 10 & 30 \end{pmatrix}, \qquad \mathbf{X}^\top\mathbf{Y} = \begin{pmatrix} 28\\ 74 \end{pmatrix} \]

\[ \mathbf{X}^\top\mathbf{X} = \begin{pmatrix} 5 & 5\\ 5 & 30 \end{pmatrix}, \qquad \mathbf{X}^\top\mathbf{Y} = \begin{pmatrix} 28\\ 74 \end{pmatrix} \]

\[ \mathbf{X}^\top\mathbf{X} = \begin{pmatrix} 30 & 10\\ 10 & 5 \end{pmatrix}, \qquad \mathbf{X}^\top\mathbf{Y} = \begin{pmatrix} 77\\ 28 \end{pmatrix} \]

\[ \mathbf{X}^\top\mathbf{X} = \begin{pmatrix} 5 & 10\\ 10 & 20 \end{pmatrix}, \qquad \mathbf{X}^\top\mathbf{Y} = \begin{pmatrix} 28\\ 77 \end{pmatrix} \]

Solution. Solution: A

(d) [2]

Assume

\[ \mathbb{E}[\boldsymbol{\varepsilon}] = \mathbf{0}, \qquad \operatorname{Var}(\boldsymbol{\varepsilon})=\sigma^2\mathbf{I}_5. \]

For each statement, enter T or F.

Statement	T/F
\(\mathbb{E}[\mathbf{Y}] = \mathbf{X}\boldsymbol{\beta}\)	`___`
\(\operatorname{Var}(\mathbf{Y}) = \sigma^2\mathbf{I}_5\)	`___`
\(\mathbb{E}[\mathbf{Y}] = \mathbf{0}\)	`___`
The five errors all have variance \(\sigma^2\)	`___`

Solution. Solution:

Statement	T/F
\(\mathbb{E}[\mathbf{Y}] = \mathbf{X}\boldsymbol{\beta}\)	T
\(\operatorname{Var}(\mathbf{Y}) = \sigma^2\mathbf{I}_5\)	T
\(\mathbb{E}[\mathbf{Y}] = \mathbf{0}\)	F
The five errors all have variance \(\sigma^2\)	T

Question 2: Least Squares and Projection [12]

Continue using the data and design matrix from Question 1.

(a) [4]

Starting from

\[ S(\boldsymbol{\beta}) = (\mathbf{Y}-\mathbf{X}\boldsymbol{\beta})^\top (\mathbf{Y}-\mathbf{X}\boldsymbol{\beta}), \]

derive the normal equations

\[ \mathbf{X}^\top\mathbf{X}\hat{\boldsymbol{\beta}} = \mathbf{X}^\top\mathbf{Y}, \]

and then state the OLS estimator when \(\mathbf{X}\) has full column rank.

Keep your derivation concise.

Solution. Solution:

Expanding,

\[ S(\boldsymbol{\beta}) = \mathbf{Y}^\top\mathbf{Y} - 2\boldsymbol{\beta}^\top\mathbf{X}^\top\mathbf{Y} + \boldsymbol{\beta}^\top\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta}. \]

Differentiating,

\[ \frac{\partial S(\boldsymbol{\beta})}{\partial\boldsymbol{\beta}} = -2\mathbf{X}^\top\mathbf{Y} + 2\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta}. \]

Setting the derivative equal to zero gives

\[ \mathbf{X}^\top\mathbf{X}\hat{\boldsymbol{\beta}} = \mathbf{X}^\top\mathbf{Y}. \]

Hence,

\[ \boxed{ \hat{\boldsymbol{\beta}} = (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top\mathbf{Y} }. \]

(b) [3]

Using the quantities from Question 1, compute

\[ \hat{\boldsymbol{\beta}} = \begin{pmatrix} \hat{\beta}_0\\ \hat{\beta}_1 \end{pmatrix}. \]

Fill in the blanks:

\[ \hat{\beta}_0 = \underline{\hspace{2cm}}, \qquad \hat{\beta}_1 = \underline{\hspace{2cm}}. \]

Hence,

\[ \hat{Y} = \underline{\hspace{2cm}} + \underline{\hspace{2cm}}x. \]

Solution. Solution:

\[ \hat{\beta}_0=1.4, \qquad \hat{\beta}_1=2.1, \]

\[ \boxed{\hat{Y}=1.4+2.1x}. \]

(c) [2]

For the least squares residual vector

\[ \mathbf{e} = \mathbf{Y}-\hat{\mathbf{Y}}, \]

which statement must be true?

A. \(\mathbf{X}\mathbf{e}=\mathbf{0}\)
B. \(\mathbf{X}^\top\mathbf{e}=\mathbf{0}\)
C. \(\mathbf{e}^\top\mathbf{e}=0\)
D. \(\mathbf{H}\mathbf{e}=\mathbf{e}\)

In one sentence, explain the geometric meaning of your choice.

Your explanation:
____________________________________________________________

Solution. Solution: B

The residual vector is orthogonal to the column space of \(\mathbf{X}\).

(d) [3]

For the hat matrix

\[ \mathbf{H} = \mathbf{X} (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top, \]

enter T or F.

Statement	T/F
\(\mathbf{H}^\top=\mathbf{H}\)	`___`
\(\mathbf{H}^2=\mathbf{H}\)	`___`
\(\hat{\mathbf{Y}}=\mathbf{H}\mathbf{Y}\)	`___`
\(\mathbf{H}\) is an orthogonal projection matrix	`___`

Solution. Solution:

Statement	T/F
\(\mathbf{H}^\top=\mathbf{H}\)	T
\(\mathbf{H}^2=\mathbf{H}\)	T
\(\hat{\mathbf{Y}}=\mathbf{H}\mathbf{Y}\)	T
\(\mathbf{H}\) is an orthogonal projection matrix	T

Question 3: Consequences and Geometry of Least Squares [12]

Continue using the data and fitted regression model from Questions 1 and 2.

Recall that

\[ \hat{\beta}_0=2.0, \qquad \hat{\beta}_1=1.8. \]

(a) Fitted Values and Residuals [3]

Compute the fitted values

\[ \hat{\mathbf{Y}} = \mathbf{X}\hat{\boldsymbol{\beta}} \]

and the residual vector

\[ \mathbf{e} = \mathbf{Y}-\hat{\mathbf{Y}}. \]

Fill in the blanks:

\[ \hat{\mathbf{Y}} = \begin{pmatrix} \underline{\hspace{1cm}}\\ \underline{\hspace{1cm}}\\ \underline{\hspace{1cm}}\\ \underline{\hspace{1cm}}\\ \underline{\hspace{1cm}} \end{pmatrix}, \qquad \mathbf{e} = \begin{pmatrix} \underline{\hspace{1cm}}\\ \underline{\hspace{1cm}}\\ \underline{\hspace{1cm}}\\ \underline{\hspace{1cm}}\\ \underline{\hspace{1cm}} \end{pmatrix}. \]

Solution. Solution:

\[ \hat{\mathbf{Y}} = \begin{pmatrix} 2.0\\ 3.8\\ 5.6\\ 7.4\\ 9.2 \end{pmatrix}, \qquad \mathbf{e} = \begin{pmatrix} 0\\ 0.2\\ -0.6\\ 0.6\\ -0.2 \end{pmatrix}. \]

(b) Consequences of Including an Intercept [3]

For each statement, enter T or F.

Statement	T/F
The residuals satisfy \(\sum_{i=1}^n e_i=0\).	`___`
The fitted values satisfy \(\bar{\hat{Y}}=\bar{Y}\).	`___`
The vector \(\mathbf{1}_n\) belongs to \(\mathcal{C}(\mathbf{X})\).	`___`
These properties necessarily hold for every regression model without an intercept.	`___`

Solution. Solution:

Statement	T/F
The residuals satisfy \(\sum_{i=1}^n e_i=0\).	T
The fitted values satisfy \(\bar{\hat{Y}}=\bar{Y}\).	T
The vector \(\mathbf{1}_n\) belongs to \(\mathcal{C}(\mathbf{X})\).	T
These properties necessarily hold for every regression model without an intercept.	F

Because the model contains an intercept, \(\mathbf{1}_n\) is a column of \(\mathbf{X}\). Since \(\mathbf{X}^\top\mathbf{e}=\mathbf{0}\),

\[ \mathbf{1}_n^\top\mathbf{e}=0, \]

which implies

\[ \sum_{i=1}^n e_i=0 \]

and therefore

\[ \bar{\hat{Y}}=\bar{Y}. \]

(c) Residual-Maker Matrix [3]

Define

\[ \mathbf{M} = \mathbf{I}_n-\mathbf{H}. \]

For each statement, enter T or F.

Statement	T/F
\(\mathbf{e}=\mathbf{M}\mathbf{Y}\)	`___`
\(\mathbf{M}^\top=\mathbf{M}\)	`___`
\(\mathbf{M}^2=\mathbf{M}\)	`___`
\(\mathbf{H}\mathbf{M}=\mathbf{0}\)	`___`

Solution. Solution:

Statement	T/F
\(\mathbf{e}=\mathbf{M}\mathbf{Y}\)	T
\(\mathbf{M}^\top=\mathbf{M}\)	T
\(\mathbf{M}^2=\mathbf{M}\)	T
\(\mathbf{H}\mathbf{M}=\mathbf{0}\)	T

The matrix \(\mathbf{H}\) projects onto \(\mathcal{C}(\mathbf{X})\), whereas \(\mathbf{M}\) projects onto \(\mathcal{C}(\mathbf{X})^\perp\).

(d) Rank and Uniqueness [3]

Suppose now that a design matrix \(\mathbf{X}\) does not have full column rank.

For each statement, enter T or F.

Statement	T/F
\((\mathbf{X}^\top\mathbf{X})^{-1}\) necessarily exists.	`___`
The least squares coefficient vector \(\hat{\boldsymbol{\beta}}\) may not be unique.	`___`
The fitted vector \(\hat{\mathbf{Y}}\) is still unique.	`___`
The residual vector \(\mathbf{e}\) is still unique.	`___`

Solution. Solution:

Statement	T/F
\((\mathbf{X}^\top\mathbf{X})^{-1}\) necessarily exists.	F
The least squares coefficient vector \(\hat{\boldsymbol{\beta}}\) may not be unique.	T
The fitted vector \(\hat{\mathbf{Y}}\) is still unique.	T
The residual vector \(\mathbf{e}\) is still unique.	T

Even when several coefficient vectors produce the same least squares solution, the orthogonal projection of \(\mathbf{Y}\) onto \(\mathcal{C}(\mathbf{X})\) is unique.

Question 4: Linear Regression in R [8]

For this question, use the built-in R dataset cars.

The variables are:

speed: speed of the car in miles per hour;
dist: stopping distance in feet.

Consider the model

\[ \mathrm{dist}_i = \beta_0 + \beta_1\mathrm{speed}_i + \varepsilon_i. \]

Load the data using

data(cars)

(a) [4]

Fit the model using lm() and complete the table.

fit <- lm(dist ~ speed, data = cars)
summary(fit)


Call:
lm(formula = dist ~ speed, data = cars)

Residuals:
    Min      1Q  Median      3Q     Max 
-29.069  -9.525  -2.272   9.215  43.201 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) -17.5791     6.7584  -2.601   0.0123 *  
speed         3.9324     0.4155   9.464 1.49e-12 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 15.38 on 48 degrees of freedom
Multiple R-squared:  0.6511,    Adjusted R-squared:  0.6438 
F-statistic: 89.57 on 1 and 48 DF,  p-value: 1.49e-12

Quantity	Your answer
\(\hat{\beta}_0\)	`___`
\(\hat{\beta}_1\)	`___`

Which interpretation of \(\hat{\beta}_1\) is correct?

A. For each 1 mph increase in speed, the stopping distance increases by exactly 3.9324 feet for every car.
B. For each 1 mph increase in speed, the fitted mean stopping distance increases by approximately 3.9324 feet.
C. The average speed increases by 3.9324 mph for every extra foot of stopping distance.
D. A car travelling at 0 mph has a stopping distance of 3.9324 feet.

Solution. Solution:

\[ \hat{\beta}_0=-17.5791, \qquad \hat{\beta}_1=3.9324. \]

The correct interpretation is B.

(b) [4]

Construct \(\mathbf{Y}\) and \(\mathbf{X}\) manually and reproduce the OLS coefficients using matrix operations.

Complete the following code:

Y <- matrix(cars$dist, ncol = 1)

X <- cbind(
  1,
  cars$speed
)

# Replace NULL with your matrix expression
beta_hat <- NULL

beta_hat

NULL

Then report

\[ \hat{\boldsymbol{\beta}} = \begin{pmatrix} \underline{\hspace{2cm}}\\ \underline{\hspace{2cm}} \end{pmatrix}. \]

Solution. Solution:

beta_hat <- solve(t(X) %*% X) %*% t(X) %*% Y

and

\[ \boxed{ \hat{\boldsymbol{\beta}} = \begin{pmatrix} -17.5791\\ 3.9324 \end{pmatrix} }. \]

Grading Summary

Question	Topic	Value
Question 1	Matrix representation	8
Question 2	Least squares and projection	12
Question 3	Distribution and inference	12
Question 4	Regression in R	8
Total		40