6  Cross Validation

Monte Carlo simulation generates inputs from a specified model. Cross-validation (CV) reuses observed data to ask a different question: how well will a fitting method predict new observations? We use CV to compare models and choose their complexity.

6.1 Prediction on New Observations

For regression, we want the average squared prediction error on independent new observations from the same population.

Training error uses the observations that fitted the model. It is usually optimistic: the model has already used their outcomes to choose its coefficients. To assess prediction, evaluate observations left out of fitting.

For \(m\) held-out observations, calculate

\[ \operatorname{MSE}=\frac1m\sum_{i=1}^m(y_i-\widehat y_i)^2. \]

For classification, we could instead average \(\mathbf{1}\{y_i\ne\widehat y_i\}\) to estimate the misclassification rate. Choose the loss before comparing models.

Figure 6.1: Training fits the coefficients; validation chooses the model; a separate test set assesses the chosen procedure. CV rotates the training and validation roles within the development data.

A single training/validation split is simple, but its result depends on which observations were held out. CV spreads validation across several subsets. A final test set remains separate from this model-selection process.

6.2 One Example: Choose a Polynomial Degree

We compare polynomial regressions

\[ \widehat f_d(x)=\widehat\beta_0+\widehat\beta_1x+\cdots+\widehat\beta_dx^d, \qquad d\in\{1,\ldots,8\}. \]

The degree \(d\) is a hyperparameter: it specifies the model’s flexibility. Fitting estimates the coefficients for a chosen degree; CV helps choose the degree.

Generate one illustrative data set with a quadratic mean and random noise. Reserve 40 observations for the final test; use 80 for development. All observations are generated independently, so these disjoint groups are independent samples.

set.seed(8670)
x <- runif(120, -2, 2)
dat <- data.frame(x, y = 1 + 2 * x - x^2 + rnorm(120, sd = 0.7))
development <- dat[1:80, ]
test <- dat[81:120, ]

Figure 6.2: The same development observations are fitted three ways. The dashed curve is the known simulation mean, shown only to explain the example. CV uses held-out outcomes to compare degrees.

A rigid model may miss the mean pattern (bias). A very flexible model can react strongly to the particular sample (variance) and fit noise. For nested least-squares models, training MSE cannot increase when we add polynomial terms. Error on new observations can increase. CV checks this prediction tradeoff without knowing the true mean.

6.3 How K-Fold CV Works

Split the \(n\) development observations into \(K\) roughly equal folds. Each fold serves once as validation data; the other \(K-1\) folds supply the training data. Every observation receives a prediction from a model fitted without that observation.

Figure 6.3: In 5-fold CV, each row is a fresh model fit. Pink is held out; mint is used for training. With 80 observations, each fit trains on 64 and predicts the remaining 16.

1. Choose the candidate models and the loss. Randomly assign the development observations to \(K\) folds; use the same folds for every candidate.
2. For each candidate, and for each fold \(k\), fit a fresh model on all folds except \(k\).
3. Predict the outcomes in fold \(k\) and store their losses. Repeat until every observation has one held-out prediction.
4. Average all \(n\) held-out losses to obtain that candidate’s CV error. Select the candidate with the smallest error.

Let \(S_k\) contain the indices in fold \(k\), and let \(\widehat f_d^{(-k)}\) be the degree-\(d\) model fitted without that fold. Then

\[ \operatorname{CV}(d) =\frac1n\sum_{k=1}^K\sum_{i\in S_k} \left(y_i-\widehat f_d^{(-k)}(x_i)\right)^2. \]

What is averaged? The squared errors of the held-out predictions. If a held-out outcome is 3 and its prediction is 2.5, it contributes \((3-2.5)^2=0.25\). With \(n=80\), sum all 80 such errors and divide by 80. Equal-sized folds also allow averaging their MSEs; unequal folds require weights \(|S_k|/n\).

6.4 Select, Refit, and Test

prediction[fold == k] stores the held-out predictions. poly(x, d) uses a numerically stable basis for a degree-\(d\) polynomial.

K <- 5
fold <- sample(rep(1:K, length.out = nrow(development)))
degrees <- 1:8
cv_mse <- numeric(length(degrees))
for (d in degrees) {
  prediction <- numeric(nrow(development))
  for (k in 1:K) {
    fit <- lm(y ~ poly(x, d), data = development[fold != k, ])
    prediction[fold == k] <- predict(fit, development[fold == k, ])
  }
  cv_mse[d] <- mean((development$y - prediction)^2)
}
best_degree <- degrees[which.min(cv_mse)]

Figure 6.4: Training error keeps falling, while CV reaches its minimum at degree 2. The right panel enlarges degrees 2-8; its vertical scale differs from the left panel. Both curves use the same development data; only CV predictions exclude the scored observations.

CV selects degree 2, with MSE 0.562. We now refit that degree on all 80 development observations, then use the 40 reserved test observations once:

final_fit <- lm(y ~ poly(x, best_degree), data = development)
test_prediction <- predict(final_fit, newdata = test)
c(degree = best_degree, test_MSE = mean((test$y - test_prediction)^2))
#>   degree test_MSE 
#> 2.000000 0.413982

The test MSE estimates prediction error for this final fit. It need not equal the CV MSE: CV used smaller training sets and different held-out observations. Also, the smallest CV error was selected from several candidates, so it can be optimistic as a final performance report. Keep the test set out of model selection.

6.5 Choose Folds and Avoid Leakage

5-fold or 10-fold CV is a useful starting point for independent observations. Each fit uses a fraction \((K-1)/K\) of the development data. Leave-one-out CV (LOOCV) sets \(K=n\): fit on \(n-1\) observations and predict the remaining one, repeating for every observation.

Larger \(K\) reduces the training-size difference from the final fit but requires more fits. It does not guarantee a less variable CV estimate. A repeated-CV illustration is in Supplementary.

Two practical details determine whether the comparison is fair:

• Fit the whole procedure inside each training fold. Learn imputation, scaling, or feature selection using that fold’s training observations, then apply those choices to its validation observations. Using validation outcomes to select features before CV leaks information.
• Split according to the prediction task. Keep repeated observations from the same person together when predicting new people. For future prediction, train on the past and validate on later times; randomly shuffling time points can use future information.

6.6 Summary

Cross-validation estimates how a fitting procedure predicts observations excluded from training. It helps choose a model or hyperparameter when the population prediction error cannot be calculated directly. Its score measures predictive performance for the chosen loss and a comparable population; it does not establish that the model is the true data-generating mechanism.

The workflow is: define the prediction task and loss, hold out a test set, compare candidates by CV, refit the selected candidate, and assess it on the test set. Use the same folds for all candidates, and keep all data-dependent fitting inside the training folds.

Method	Training / validation per fit	Fits per candidate	Main consideration
One split	One training set / one holdout	1	Fast; sensitive to the split
\(K\)-fold CV	\(K-1\) folds / one fold	\(K\)	Every observation is held out once
LOOCV	\(n-1\) observations / one	\(n\)	Nearly full training size; often costly
Repeated \(K\)-fold (optional)	Repeat the fold assignment	\(RK\)	Checks sensitivity to partitioning

Here \(n\) is the development sample size and \(R\) is the number of repetitions. The separate final test set is not one of the CV folds.

6.7 Supplementary

Optional reading:

• James, Witten, Hastie, and Tibshirani (2021), An Introduction to Statistical Learning, 2nd edition. Chapter 5, especially Section 5.1 and the R lab, develops validation, LOOCV, and \(K\)-fold CV.
• Hastie, Tibshirani, and Friedman (2009), The Elements of Statistical Learning, 2nd edition. Sections 7.2-7.3 discuss bias and variance; Section 7.10 covers CV and why every model-building step belongs inside validation.
• 2025 STAT 8670 lecture: additional models and the ironslag example.

6.7.1 Repeated CV (optional)

Figure 6.5: Ten new 5-fold partitions of the same development data give slightly different CV curves. Only degrees 2-8 are shown to make nearby candidates visible. Thin curves are individual partitions; the thick curve is their average. No test observations are used.

Repeating CV checks sensitivity to the fold assignment; it does not create new observations. Here several degrees have similar scores. A tiny improvement in one partition is weak evidence for a more complex model. The repeated curves describe split sensitivity, not independent experiments or a confidence interval.