Assignment 4
STAT 8670 | Fall 2026
Due date: Friday, November 20, 2026
Total: 30 points
Coverage: Chapter 5: Monte Carlo Simulation and Variance Reduction, with a guided introduction to Chapter 6: Cross Validation. The additional practice also reviews Chapter 4: Generating Random Variables.
Submit Questions 1-4 only (30 points). Questions 1-3 cover Chapter 5 [24 points]; Question 4 introduces Chapter 6 with a supplied reading guide [6 points]. P1-P10 are additional practice: not graded and not required for submission. Optional and supplementary sections of the notes are not required.
Exam preparation: We recommend working through this assignment before Exam 2 on November 16, together with Assignment 3 and the main material in Chapters 4-6. The submission deadline is November 20, after the exam. These exercises support review; they are not an exhaustive list of exam topics or a promise of identical questions.
| Question | Topic / format | Points |
|---|---|---|
| 1 | Monte Carlo concepts (8 multiple-choice items) | 8 |
| 2 | Variance reduction (8 multiple-choice items) | 8 |
| 3 | Precision and interpretation (4 short-answer parts) | 8 |
| 4 | Guided preview: cross-validation (3 short parts) | 6 |
| Total | 30 | |
| P1-P10 | Additional Exam 2 practice | Not graded |
Assignment PDF | Response template (.qmd)
Use Quarto or R Markdown. Upload both files to the Assignment 4 folder on iCollege:
- Your rendered PDF, named
A4_LastName_FirstName.pdf. - Its matching source, named
A4_LastName_FirstName.qmdorA4_LastName_FirstName.Rmd.
The PDF must be generated from the submitted source. HTML alone does not satisfy the submission requirement.
- Put your name and course number at the beginning, and label every question and subpart. Use the response template or follow its labeling format; you do not need to copy a table.
- For Questions 1-2, enter exactly one letter for each item. For Question 3, show a brief formula or substitution with each calculation, and add one sentence of interpretation in part (b). For Question 4, follow the requested response type; part (b) needs one sentence. Equivalent correct reasoning is accepted.
- Round numerical answers to three decimal places, retaining full precision during calculations. Report counts as integers and leave exact symbolic quantities in symbolic form.
- No programming or plots are required. If you use R, include executable chunks after your answers and set
set.seed(8670)once before each independent simulation; do not reset it within repetitions. - Your source must render from beginning to end without errors. Use LaTeX for mathematics; do not submit screenshots of answers or code.
Solutions are provided below each part, including P1-P10. Expand a Solution to view the answer and explanation. Solutions are also included in the course PDF.
Question 1: Monte Carlo Concepts [8]
Select one best answer for each item. Each item is worth 1 point. Assume the expectations and variances used below are finite. Here \(s_Y\) denotes the sample standard deviation of the outputs \(Y_i\).
(a) The target [1]
Let \(U\sim\operatorname{Uniform}(0,1)\), \(Y=g(U)\), and \(\theta=E[Y]\). What does the average of many independent \(g(U_i)\) values estimate?
A. The variance of \(U\).
B. The expectation \(E[g(U)]\).
C. The largest possible value of \(g(U)\).
D. The value \(g(E[U])\), which always equals \(E[g(U)]\).
B [1]. The estimator averages the outputs \(Y_i=g(U_i)\), so its target is their expectation. In general, \(E[g(U)]\ne g(E[U])\).
(b) Output spread and simulation precision [1]
An estimate averages \(N>1\) independent, identically distributed outputs \(Y_i=g(U_i)\) with nonzero variance. Which estimates its Monte Carlo standard error (MCSE)?
A. \(s_U/\sqrt N\).
B. \(s_Y\).
C. \(s_Y/N\).
D. \(s_Y/\sqrt N\).
D [1]. MCSE concerns the average of the outputs, so it uses their SD divided by \(\sqrt N\).
(c) A larger simulation [1]
With the same output distribution, approximately how many independent draws are needed to reduce MCSE from 0.03 to 0.01 if the current run uses 2,000 draws?
A. 18,000.
B. 6,000.
C. 2,000.
D. 200.
A [1]. A threefold reduction in MCSE requires nine times the draws: \(2000(0.03/0.01)^2=18000\).
(d) An integral [1]
To estimate \(I=\int_2^5 g(x)\,dx\), generate independent \(X_i\sim\operatorname{Uniform}(2,5)\). Which estimator is correct?
A. \(\operatorname{mean}(g(X_i))/3\).
B. \(\operatorname{mean}(X_i)\).
C. \(3\operatorname{mean}(g(X_i))\).
D. \(\operatorname{mean}(g(X_i))\).
C [1]. The interval has width 3 and the uniform density is \(1/3\), so \(I=3E[g(X)]\).
(e) A probability [1]
Independent draws \(X_1,\ldots,X_N\) follow the specified model. Let \(I_i=1\) if \(X_i>8\) and \(I_i=0\) otherwise. Which expression estimates \(P(X>8)\)?
A. \(N^{-1}\sum_{i=1}^N I_i\).
B. The average of only those \(X_i\) that exceed 8.
C. \(\sum_{i=1}^N I_i\).
D. The indicator that the sample mean of the \(X_i\) exceeds 8.
A [1]. Average the event indicators. The other expressions return a conditional sample mean, a count, or a single logical value.
(f) An uncertainty interval [1]
A large simulation of independent, identically distributed outputs reports \(\widehat\theta\pm1.96\widehat{\mathrm{MCSE}}\). What uncertainty does this interval address?
A. The spread of individual simulated outputs.
B. All uncertainty in the scientific model and its assumptions.
C. Monte Carlo error in estimating the specified model expectation.
D. Only measurement error in the original dataset.
C [1]. The interval describes simulation error under the specified model, using a large-sample approximation. It does not include model misspecification or parameter-estimation uncertainty unless those are separately incorporated.
(g) Reproducibility [1]
A script sets set.seed(8670) once before a simulation. Which statement is correct?
A. The seed guarantees the estimate equals the target.
B. The seed makes the simulation error zero.
C. Every independently chosen seed gives the same estimate.
D. The seed supports reproducibility; MCSE is still needed to assess precision.
D [1]. A reproducible result can still have Monte Carlo error.
(h) A missing factor [1]
An estimate of \(\int_2^5 g(x)\,dx\) uses \(N^{-1}\sum_{i=1}^N g(X_i)\) for independent \(X_i\sim\operatorname{Uniform}(2,5)\), omitting the interval-width factor. Assume the desired integral is nonzero. What does increasing \(N\) accomplish?
A. It automatically supplies the missing factor.
B. It estimates the wrong scaled target more precisely.
C. It makes the output distribution uniform.
D. It guarantees a larger estimated MCSE.
B [1]. The average converges to one third of the integral. More draws do not repair the estimator.
Question 2: Variance Reduction [8]
Select one best answer for each item. Each item is worth 1 point. Use the conventions from Chapter 5. Assume finite second moments; \(s_A\) is the sample standard deviation of the \(K\) pair averages, with \(K>1\).
(a) Antithetic inputs [1]
For \(U\sim\operatorname{Uniform}(0,1)\), which statement about \((U,1-U)\) is correct?
A. The two values are independent.
B. Their sum is random.
C. Each is uniform, but they are dependent.
D. Only the first value is uniform.
C [1]. Reflection preserves the uniform distribution, but \(1-U\) is determined by \(U\).
(b) The independent unit [1]
From \(K\) independent uniforms, form \(A_k=\{g(U_k)+g(1-U_k)\}/2\). Which estimates the MCSE of \(\overline A\)?
A. \(s_A/\sqrt K\).
B. \(s_A/\sqrt{2K}\).
C. The SD of all \(2K\) outputs divided by \(\sqrt{2K}\), treating them as independent.
D. \(s_U/\sqrt K\).
A [1]. There are \(K\) independent pair averages. Outputs within a pair are dependent.
(c) A fair cost comparison [1]
Each call to \(g\) has the same cost. An antithetic run uses 600 pairs. Which plain Monte Carlo run has the same number of calls to \(g\)?
A. 300 independent outputs.
B. 600 independent outputs.
C. 2,400 independent outputs.
D. 1,200 independent outputs.
D [1]. Each pair uses two evaluations, giving \(2(600)=1200\) calls.
(d) When reflection helps [1]
At the same number of calls to \(g\), antithetic pairing reduces estimator variance relative to independent outputs when:
A. \(g(U)\) and \(g(1-U)\) have positive covariance.
B. \(g(U)\) and \(g(1-U)\) have negative covariance.
C. \(U\) is resampled until it exceeds 0.5.
D. \(g\) is any function, without further conditions.
B [1]. The paired variance contains the covariance term. Negative output covariance reduces it; reflection alone is not a guarantee.
(e) The sign of a control coefficient [1]
Use \(Z=Y+c(C-\mu_C)\) with known \(\mu_C=E[C]\) and \(\operatorname{Var}(C)>0\). If \(\operatorname{Cov}(Y,C)>0\), the variance-minimizing fixed coefficient \(c^*\) is:
A. Positive.
B. Zero.
C. Negative.
D. Always equal to 1.
C [1]. With this plus-sign convention, \(c^*=-\operatorname{Cov}(Y,C)/\operatorname{Var}(C)<0\).
(f) Preserving the target [1]
For a fixed coefficient \(c\), why does \(Z=Y+c(C-\mu_C)\) have the same expectation as \(Y\)?
A. \(E[C-\mu_C]=0\).
B. \(C\) is always zero.
C. \(C\) and \(Y\) must be independent.
D. \(Z\) must equal \(Y\) for every draw.
A [1]. The adjustment has mean zero, so \(E[Z]=E[Y]\).
(g) Replacing a known mean [1]
A programmer uses \(Z_i=Y_i+c(C_i-\overline C)\) with the same sample mean \(\overline C\) and the same fixed \(c\) in every row. What happens to \(\overline Z\)?
A. It always becomes zero.
B. Its variance must be smaller than that of the original sample mean.
C. It becomes the known control mean.
D. It equals the original sample mean of the outputs.
D [1]. The sample average of \(C_i-\overline C\) is exactly zero, so \(\overline Z=\overline Y\).
(h) No association with the control [1]
If \(\operatorname{Cov}(Y,C)=0\) and \(\operatorname{Var}(C)>0\), which fixed coefficient minimizes \(\operatorname{Var}\{Y+c(C-\mu_C)\}\)?
A. \(c=1\).
B. \(c=0\).
C. \(c=-1\).
D. Every value gives the same variance.
B [1]. The variance becomes \(\operatorname{Var}(Y)+c^2\operatorname{Var}(C)\), minimized at zero.
Question 3: Precision and Interpretation [8]
Two runs estimate the same \(\theta=E[g(U)]\), where \(U\sim\operatorname{Uniform}(0,1)\). Each evaluates \(g\) exactly 1,200 times, and each evaluation has the same cost. The plain run averages independent outputs; the antithetic run averages independent pair averages \(A_k\).
| Method | Independent contributions | Estimate | Sample SD of contributions |
|---|---|---|---|
| Plain | \(N=1200\) outputs | 1.721 | 0.600 |
| Antithetic | \(K=600\) pair averages | 1.718 | 0.120 |
Treat the displayed SDs as exact for these calculations. Retain full precision until the final answer.
(a) MCSE [2]
Report the estimated MCSE for plain and antithetic Monte Carlo, in that order. Each value is worth 1 point.
[1 each]. Plain: \(0.600/\sqrt{1200}=0.0173205\approx0.017\). Antithetic: \(0.120/\sqrt{600}=0.0048990\approx0.005\). Use 600, not 1,200, independent pair averages.
(b) Relative precision [2]
Compute the ratio of estimated estimator variances, \(\widehat{\operatorname{Var}}(\widehat\theta_{\mathrm{plain}})/\widehat{\operatorname{Var}}(\widehat\theta_{\mathrm{anti}})\) [1]. In one sentence, identify the more precise estimator and interpret this variance ratio at the stated computational budget (do not call it an MCSE ratio) [1].
12.500 [1]. \((0.600^2/1200)/(0.120^2/600)=12.5\). Interpretation [1]: at the same number of function evaluations, the plain estimator has about 12.5 times the estimated variance of the antithetic estimator (equivalently, the antithetic estimate has about 8% of the variance). This is not a ratio of SDs.
(c) A Monte Carlo interval [2]
Using the antithetic result, give the lower and upper endpoints of the approximate 95% interval \(\widehat\theta\pm1.96\widehat{\mathrm{MCSE}}\) (1 point each).
[1 each]. \(1.718\pm1.96(0.120/\sqrt{600})\) gives (1.708, 1.728). This is an interval for the specified expectation due to Monte Carlo error.
(d) A separate control-variate calculation [2]
In a separate simulation, \(E[C]=2\), \(\operatorname{Var}(C)=0.25\), and \(\operatorname{Cov}(Y,C)=0.50\). Use the fixed optimal coefficient in
\[Z=Y+c(C-2),\qquad c^*=-\frac{\operatorname{Cov}(Y,C)}{\operatorname{Var}(C)}.\]
The run gives \(\overline Y=3.4\) and \(\overline C=2.1\). Report \(c^*\) and the adjusted estimate \(\overline Z\) (1 point each).
[1 each]. \(c^*=-0.50/0.25=-2.000\) and \(\overline Z=3.4-2(2.1-2)=3.200\).
Question 4: Guided Preview: Cross-Validation [6]
New material. Read Chapter 6: prediction on new observations and K-fold CV.
Reading guide. Split the development data into \(K\) folds. For each candidate and each fold, fit using the other folds and predict the held-out observations. Average their squared errors to obtain the CV MSE. Use the same folds for all candidates. Select the candidate with the smallest CV MSE, refit it on all development observations, and assess it once on a separate test set that was not used for selection. Data-dependent preprocessing and feature selection belong inside each training fold.
(a) Data roles [2]
There are 100 independent development observations in five equal folds. For each fit, how many observations are used for training and for validation, respectively (1 point each)?
80 and 20 [1 each]. Each fit trains on four folds and validates on the remaining fold.
(b) Choose a candidate [2]
Three polynomial regression candidates use the same five folds and squared-error loss. Training MSE is calculated after fitting each candidate to all 100 development observations; CV MSE uses held-out predictions from the five separate training fits:
| Degree | Training MSE | CV MSE |
|---|---|---|
| 1 | 9.0 | 10.0 |
| 2 | 4.0 | 6.0 |
| 6 | 1.0 | 12.0 |
Which degree should be selected by CV [1]? In one sentence, explain why the smallest training MSE does not determine this choice [1].
Degree 2 [1]. Reason [1]: it has the smallest held-out prediction error; training error evaluates observations already used for fitting and can favor an overly flexible model.
(c) Valid assessment [2]
Enter T or F for each statement (1 point each).
- Selecting features using all development outcomes before forming the folds gives a valid CV assessment of that feature-selection procedure.
- After choosing the degree by CV, it is appropriate to refit on all development data and evaluate once on an untouched test set.
F, T [1 each]. Feature selection must be repeated inside each training fold; otherwise held-out outcomes influence the fitted procedure. The reserved test set is used after selection.
Additional Exam 2 Practice (Not Graded)
These ten questions complement Questions 1-4 and Assignment 3. They practice core concepts in new situations; the longer explanations are for learning and do not prescribe the length of an exam answer. They are not an additional submission requirement and do not change the 30-point assignment total. No programming is required; use short explanations and calculations where requested.
- P1-P5: random-variable generation and checking (Chapter 4).
- P6-P8: Monte Carlo precision and variance reduction (Chapter 5).
- P9-P10: validation design and prediction loss (Chapter 6, with reading guides).
Vectors are bold and are column vectors. Gamma parameters are shape and rate unless stated otherwise. Exclude the optional and supplementary material in the notes.
P1. Inverse transformation
A lifetime \(X\) has CDF
\[F(x)=\begin{cases}0,&x\leq0,\\x^2/4,&0<x<2,\\1,&x\geq2.\end{cases}\]
For \(U\sim\operatorname{Uniform}(0,1)\), select the correct generator. Check your choice by substituting it into \(F\).
A. \(X=2U^2\).
B. \(X=\sqrt{U}/2\).
C. \(X=2\sqrt{U}\).
D. \(X=4U\).
C. Solving \(u=x^2/4\) on \(0<x<2\) gives \(x=2\sqrt u\). Then \(F(2\sqrt U)=U\), and the generated values have the required support. This is an inverse transformation, not direct substitution of a uniform draw into the CDF.
P2. Construct a distribution from normal draws
Let \(Z_1,\ldots,Z_4\) be independent standard normal variables and define
\[T=\frac{Z_1^2+Z_2^2+Z_3^2+Z_4^2}{6}.\]
Which distribution does \(T\) have? Here \(\operatorname{Gamma}(a,b)\) uses shape \(a\) and rate \(b\).
A. \(\operatorname{Gamma}(2,3)\).
B. \(\operatorname{Gamma}(4,6)\).
C. \(\operatorname{Gamma}(2,1/3)\).
D. \(\chi^2_4\).
A. The numerator is \(\chi^2_4=\operatorname{Gamma}(2,1/2)\). Dividing by 6 multiplies the rate by 6, giving \(\operatorname{Gamma}(2,3)\), with mean \(2/3\) and variance \(2/9\). Rate and scale transform in opposite directions.
P3. Check the joint distribution
Let \(\mathbf Z=(Z_1,Z_2)^\top\sim N_2(\mathbf0,I_2)\) and define
\[\mathbf X=\begin{pmatrix}1&0\\1&2\end{pmatrix}\mathbf Z, \qquad \mathbf W=\begin{pmatrix}1&0\\0&\sqrt5\end{pmatrix}\mathbf Z.\]
Which statement is correct? Treat the vectors as columns.
A. The two vectors have the same joint distribution because their coordinate variances agree.
B. \(X_1\) and \(X_2\) are independent because \(Z_1\) and \(Z_2\) are independent.
C. \(\operatorname{Var}(X_2)=4\), whereas \(\operatorname{Var}(W_2)=5\).
D. Both vectors have coordinate variances 1 and 5, but \(\operatorname{Cov}(X_1,X_2)=1\) and \(\operatorname{Cov}(W_1,W_2)=0\).
Write the two covariance matrices as a check.
D. The covariance matrices are
\[ \operatorname{Cov}(\mathbf X)= \begin{pmatrix}1&0\\1&2\end{pmatrix} \begin{pmatrix}1&1\\0&2\end{pmatrix} =\begin{pmatrix}1&1\\1&5\end{pmatrix}, \qquad \operatorname{Cov}(\mathbf W)=\begin{pmatrix}1&0\\0&5\end{pmatrix}. \]
The shared \(Z_1\) term induces covariance in \(\mathbf X\). Matching the two univariate marginal distributions does not determine the joint distribution.
P4. Proposal support
A rejection sampler targets \(\operatorname{Uniform}(0,1)\) and proposes \(Y\sim\operatorname{Exp}(1)\) on \((0,\infty)\).
Mark T or F and give one sentence: “The proposal is unusable because it can generate values greater than 1.”
For this question, you may use the valid envelope constant \(M=e\): the target density is 1 on \((0,1)\) and zero elsewhere, while the proposal density is \(g(y)=e^{-y}\) for \(y>0\).
F. A proposal may cover more than the target support; proposals outside \((0,1)\) are rejected. For \(0<y<1\), \(f(y)/g(y)=e^y\leq e\), and for \(y\geq1\) the target density is zero. Missing part of the target support is a problem; extra proposal support is not itself a validity problem.
P5. Rejection-sampling efficiency
Two rejection samplers use the same normalized target density \(f\) and proposal density \(g\). Both envelope constants are valid: \(M_A=1.5\) and \(M_B=3\), meaning \(f(y)\leq M g(y)\) throughout the target support. Each sampler uses 600 independent proposals, with the same cost per proposal and fresh independent acceptance draws.
- Calculate each sampler’s overall acceptance probability and expected number of accepted draws.
- Which sampler gives more accepted draws per proposal, and by what factor? Does the larger valid envelope change the target distribution of accepted draws?
Guide. For normalized target and proposal densities, the overall acceptance probability is \(1/M\). An expected count need not equal the count in a particular run.
- For sampler A, \(P(\text{accept})=1/1.5=2/3\approx0.667\), so the expected count is \(600(2/3)=400\). For sampler B, \(P(\text{accept})=1/3\approx0.333\), so the expected count is \(600/3=200\).
- Sampler A gives twice as many accepted draws per proposal on average: \((2/3)/(1/3)=2\). Both samplers still produce the same target distribution among accepted draws because both envelopes are valid. Increasing a valid envelope constant lowers efficiency; it does not change that target. Actual accepted counts are random and need not equal their expectations.
P6. Combine independent simulation batches
Two independent batches contain \(n_1=100\) and \(n_2=400\) independent outputs from the same distribution, with mean \(\theta\) and variance 8. Let their means be \(\overline Y_1\) and \(\overline Y_2\). Compare
\[\widehat\theta_A=\tfrac12\overline Y_1+\tfrac12\overline Y_2, \qquad \widehat\theta_B=\tfrac15\overline Y_1+\tfrac45\overline Y_2.\]
- Are both estimators unbiased? Explain using their expectations.
- Calculate both estimator variances. Which uses the 500 outputs more efficiently, and why?
- Does the smaller estimator variance guarantee a closer estimate to \(\theta\) in every pair of realized runs? Explain briefly.
Use \(\operatorname{Var}(aV+bW)=a^2\operatorname{Var}(V)+b^2\operatorname{Var}(W)\) for independent \(V,W\).
- Both are unbiased: \(E[\widehat\theta_A]=(1/2+1/2)\theta=\theta\) and \(E[\widehat\theta_B]=(1/5+4/5)\theta=\theta\).
- Independence gives
\[ \operatorname{Var}(\widehat\theta_A)=\tfrac14(8/100)+\tfrac14(8/400)=0.025, \] \[ \operatorname{Var}(\widehat\theta_B)=\tfrac1{25}(8/100)+\tfrac{16}{25}(8/400)=0.016. \]
B is more efficient. It is the mean of all 500 outputs. Equal weighting of unequal batches overweights the less precise small-batch mean here.
- No. Smaller variance describes repeated-run precision, not an ordering of the absolute errors in every pair of realized runs. Either estimate can happen to be closer in a particular run.
P7. Evaluate a proposed pairing
Each paired run averages \(K\) independent pair averages \((Y_k+Y'_k)/2\). Both members have the correct target marginal distribution and variance 1. Their within-pair covariance differs:
| Pairing | \(\operatorname{Cov}(Y_k,Y'_k)\) |
|---|---|
| A | \(-0.6\) |
| B | \(0.4\) |
Each method uses \(2K\) function evaluations. The plain comparison averages \(2K\) independent outputs of variance 1.
- Calculate \(\operatorname{Var}(\widehat\theta_{\mathrm{paired}})/\operatorname{Var}(\widehat\theta_{\mathrm{plain}})\) separately for A and B. Which pairing improves precision?
- A paired-to-plain ratio of 0.4 for estimator variance is reported as a paired-to-plain MCSE ratio of 0.4. Is that conversion correct? Give the correct ratio.
Use \(\operatorname{Var}\{(Y+Y')/2\}=\{\operatorname{Var}(Y)+\operatorname{Var}(Y')+2\operatorname{Cov}(Y,Y')\}/4\).
For A, one pair average has variance \((2-1.2)/4=0.2\), giving estimator variance \(0.2/K\). For B, the corresponding values are \(0.7\) and \(0.7/K\). Plain Monte Carlo has variance \(1/(2K)\), so the ratios are 0.400 and 1.400. A improves precision; B worsens it at the same evaluation budget.
The MCSE ratio is the square root of the variance ratio: \(\sqrt{0.4}\approx0.632\), not 0.4. Correct marginal distributions preserve the target but do not by themselves guarantee a useful pairing.
P8. Can a larger adjustment be worse?
Suppose \(E[Y]=\theta\), \(E[C]=\mu_C\) is known, \(\operatorname{Var}(Y)=9\), \(\operatorname{Var}(C)=4\), and \(\operatorname{Cov}(Y,C)=3\). Use
\[Z_c=Y+c(C-\mu_C).\]
Compare the two fixed coefficients \(c=-0.5\) and \(c=-2\).
- State the expectation and variance of \(Z_c\) for each coefficient.
- A student argues that making \(c\) more negative must keep improving precision because the covariance is positive. Use your results to assess this claim at the same number of independent draws.
You may use \(\operatorname{Var}(Z_c)=\operatorname{Var}(Y)+c^2\operatorname{Var}(C)+2c\operatorname{Cov}(Y,C)\).
Both expectations equal \(\theta\), since the control is centered at its known mean. For \(c=-0.5\), the variance is \(9+0.25(4)-3=7\). For \(c=-2\), the variance is \(9+4(4)-12=13\). With \(N\) independent draws, the estimator variances are \(7/N\) and \(13/N\), versus \(9/N\) for plain Monte Carlo. Thus the smaller-magnitude adjustment helps, but the larger one is worse than no adjustment. Positive covariance motivates a negative coefficient under this convention; it does not justify an arbitrarily large magnitude. The optimum is \(-3/4\), although finding it is not required.
P9. Match validation to the prediction task
Reading guide. A validation split should resemble the future prediction task. For a new group, keep its observations out of training; for forecasting, train on earlier times and validate on later times. Learn preprocessing from the training portion of each split. See Chapter 6: choosing folds and avoiding leakage.
A retailer records monthly sales for several stores. Address these two different goals:
- To predict the next month’s sales at the existing stores, should validation randomly mix past and future months, or hold out later months and train on earlier months? Explain briefly.
- For the separate goal of predicting sales at a new store during the same calendar period (not forecasting future months), should rows from the same store be spread across training and validation, or kept in the same fold? Assume stores are the independent sampling units. Explain briefly.
- Within a CV fit, a predictor is standardized using a mean and SD estimated from all development rows before splitting. State where those quantities should instead be estimated and how they should be applied to the validation rows.
- Hold out later months and train only on earlier months. Random mixing can let future observations inform predictions of earlier observations, which does not represent forecasting.
- Keep each store’s rows together, holding out entire stores to assess prediction for a new store. A model trained on other records from the validation store has already seen that group.
- Estimate the mean and SD using only the current fit’s training observations, then apply those same training-derived values to validation. Do not estimate separate validation scaling parameters or use the held-out rows to learn the transformation. The central point is validation of the entire fitting procedure.
P10. Aggregate held-out loss
A candidate model specification is fixed. It is refitted separately on the other two folds each time, giving the following held-out results. Each observation appears in exactly one validation fold:
| Fold | Number of validation observations | Sum of squared prediction errors |
|---|---|---|
| 1 | 2 | 2 |
| 2 | 3 | 9 |
| 3 | 5 | 4 |
- Calculate the overall CV MSE, giving every validation observation equal weight.
- A student instead averages the three fold MSEs without weights. Calculate that result and explain why it answers a different averaging question.
Guide. Add the squared held-out errors over all observations and divide by their total count. Unequal folds require size weights if you start from fold MSEs.
The overall CV MSE is \((2+9+4)/(2+3+5)=1.500\). The fold MSEs are 1, 3, and 0.8; their unweighted average is 1.600. Equal fold weighting gives each fold the same influence, so observations in the smaller folds receive greater weight. To give every observation equal weight, use \((2/10)(1)+(3/10)(3)+(5/10)(0.8)=1.5\). Neither result is a training MSE: both use held-out predictions, but their aggregation differs.