Chapter 6 asked whether individual observations change a fitted model. A different problem arises when many observations follow a pattern that the model misses. We begin with pictures and numbers: when should we change the predictor scale, and when should we change the response scale? We then make the regression assumptions explicit, use them to guide transformations, and consider which terms to retain. Finally, ridge, lasso, and elastic net show how shrinking coefficients can improve prediction when a full OLS fit is unstable.
Learning Objectives
By the end of this chapter, students should be able to:
distinguish correct mean form, normality, constant variance, and independence;
distinguish a curved relationship from a model nonlinear in its parameters;
use curvature and changing residual spread to motivate a transformation;
distinguish transforming a predictor from transforming the response;
interpret predictions after transforming the response;
carry out forward selection, backward elimination, and stepwise selection;
write the regression models and estimation objectives for ridge, lasso, and elastic net;
tune penalties using cross-validation within training data;
separate model development from final prediction assessment.
NoteQuick Recap: Tools for Model Development
Chapter 2: OLS fits the mean patterns represented by the design columns.
Chapter 3: coefficient tests and intervals rely on assumptions for a specified model.
Chapter 4: compare nested models on the same observations and response; a lower training SSE alone does not establish better prediction.
Chapter 5: interpret coefficients conditionally, code factors as whole terms, and retain the lower-order terms needed for interactions.
Chapter 6: distinguish response outliers, leverage, and influence; investigate flags and recheck sensitivity when the model changes.
7.1 When does a transformation help?
Start with a fitted straight line and ask two questions: does it follow the center of the responses, and is the spread around that center roughly constant? A systematic curve points to a mean-shape problem. A widening spread points to a variance problem. A transformation may help with one or both, but the choice must match the pattern and the meaning of the variables. NIST’s transformation examples illustrate these two goals.
The following datasets are deliberately constructed to make the differences easy to see. Each has 15 observations, three at each of five predictor values; they illustrate model behavior, rather than provide evidence about a population.
Example 7.1 (A Curved Mean with Similar Spread: Transform the Predictor) Suppose we measure a response at input levels \(x=1,2,4,8,16\). In the top-left panel, the response increases quickly at first and then more slowly. The straight line misses the center of several groups. In the top-right panel, doubling \(x\) becomes a one-unit step on the \(\log_2 x\) axis, and the same observations follow a straight line.
Show R code
library(ggplot2)ch7_curve <-data.frame(x =rep(c(1, 2, 4, 8, 16), each =3))ch7_curve$y <-10+4*log2(ch7_curve$x) +rep(c(-0.6, 0, 0.6), 5)ch7_curve_raw <-lm(y ~ x, data = ch7_curve)ch7_curve_logx <-lm(y ~log2(x), data = ch7_curve)
Figure 7.1: The same 15 observations before and after transforming x. Top: data and fitted lines. Bottom: residuals in the same response units, with matching vertical limits. The curved residual pattern disappears after using log2(x); the response units stay unchanged.
Read the residuals. With the original line, low-\(x\) responses lie below the line, middle responses tend to lie above it, and high-\(x\) responses fall below it again. That shared pattern is not an isolated outlier. After transforming \(x\), every group has residuals \(-0.6,0,0.6\); no curved mean pattern remains in this constructed example.
Put numbers on the change. The group means increase by 4 whenever \(x\) doubles. The sample SD within each group is 0.6, so the motivation here is the mean shape, not increasing spread.
At \(x=4\), the original line predicts 15.87 and the transformed model predicts 18.00, matching the observed group mean. The fitted slope 4 means a doubling of \(x\) adds 4 response units, since \(\log_2(2x)-\log_2(x)=1\).
The training RMSE, \(\sqrt{\mathrm{SSE}/15}\), falls from 2.090 to 0.490. This comparison is meaningful because both fits use the same observations and response scale, and both have two coefficients. It describes training fit, not held-out prediction accuracy. Only the predictor changed; the response and its units did not.
Example 7.2 (Increasing Level and Spread: Transform the Response) Now suppose a positive amount \(y\) is measured at times \(x=0,1,2,3,4\). The middle response doubles at each step, and the three observations spread farther apart in the original units. A straight line with constant error variance is missing both features. Taking logs compresses large values: the top-right panel becomes straight, and the bottom-right residuals have the same spread at every time.
Show R code
ch7_growth <-data.frame(x =rep(0:4, each =3))ch7_growth$y <-10*2^ch7_growth$x *rep(c(0.8, 1, 1.25), 5)ch7_growth_raw <-lm(y ~ x, data = ch7_growth)ch7_growth_logy <-lm(log(y) ~ x, data = ch7_growth)
Figure 7.2: The same 15 positive responses on original and natural-log scales. On the original scale, the center curves upward and the within-time spread grows. On the log scale, a straight line describes the center and the residual spread is constant in this constructed example. Residual magnitudes across the two scales have different units.
Put numbers on the spread. At \(x=0\), the observations are \(8,10,12.5\); at \(x=4\), they are \(128,160,200\). Both triples use the same multipliers, \(0.8,1,1.25\), around their middle value. The absolute spread grows, but the proportional spread stays the same.
x
Responses
Mean response
SD of y
SD of log(y)
0
8, 10, 12.5
10.17
2.25
0.223
1
16, 20, 25
20.33
4.51
0.223
2
32, 40, 50
40.67
9.02
0.223
3
64, 80, 100
81.33
18.04
0.223
4
128, 160, 200
162.67
36.07
0.223
Each SD is computed from the three observations at that time, not from all 15 pooled together. The original-scale SD rises from 2.25 to 36.07, while the log-scale SD stays at 0.223. A nonlinear mean alone would follow the upward curve but would not remove this increasing original-scale variation.
The original line even predicts a negative amount at \(x=0\). The log fit has a different interpretation: one time step adds approximately \(0.693=\log 2\) on the log scale, so exponentiating the fitted value gives the curve \(10\times2^x\). At \(x=2\), that curve gives 40, whereas the observed arithmetic mean is 40.67. Exponentiating an estimated log mean is not the same as estimating an arithmetic mean; Section 7.3 explains the required back-transformation and its assumptions.
Here log() means the natural logarithm; log2() in the previous example made the doubling interpretation especially direct. Do not compare the raw SSE or RMSE of \(y\) with that of \(\log y\): the residuals have different units. Compare model adequacy on each fitted scale and, for prediction, evaluate the same target in common original units.
When should we keep the original scale? If the fitted mean is plausible and residuals show roughly constant spread without a systematic pattern, there may be no reason to transform. A skewed histogram of \(Y\) alone is not enough: regression assumptions concern errors conditional on the predictors. An isolated unusual point calls for the investigation in Chapter 6, not an automatic transformation of all observations.
Transformations are candidates, not mandatory fixes. Curvature may instead be represented by polynomial terms or splines; nonconstant variance may be modeled with weighted least squares. If both shape and spread need attention, choose a meaningful response scale and then recheck which predictor functions are needed. NIST discusses transformation and weighting. In real data, always refit and recheck: these tidy constructed examples do not establish normality, independence, or performance on new observations.
7.2 What assumptions are we trying to achieve?
Section 7.1 showed patterns that a model can miss. Which part of the model needs to change? Use the four assumptions to guide the diagnosis.
Definition 7.1 (Assumptions of the Normal Linear Model) Using the notation from Chapter 3, write
There are \(d\) predictor columns and \(p=d+1\) coefficients, including the intercept. A column may be an original predictor, a transformation, or an interaction. Conditional on the design \(\mathbf X\), distinguish:
Correct mean:\(E(\varepsilon_i\mid\mathbf X)=0\). Thus \(E(Y_i\mid\mathbf X)=\beta_0+\sum_j\beta_jx_{ij}\). The chosen columns describe the systematic mean pattern.
Constant variance (homoscedasticity):\(\operatorname{Var}(\varepsilon_i\mid\mathbf X)=\sigma^2\) for every \(i\). Responses have the same conditional spread around their respective means. Their means may differ.
Independent errors: given \(\mathbf X\), one observation’s error supplies no information about another’s. Repeated measurements on one person may violate this assumption. Independence implies \(\operatorname{Cov}(\varepsilon_i,\varepsilon_j\mid\mathbf X)=0\) for \(i\ne j\). The converse need not hold. With constant variance, uncorrelated errors suffice for the usual OLS covariance formula.
Normality: each \(\varepsilon_i\mid\mathbf X\) has a normal distribution. Together with assumptions 1–3, this gives \(\boldsymbol\varepsilon\mid\mathbf X\sim N_n(\mathbf0,\sigma^2\mathbf I_n)\).
A unique full-model OLS estimate requires \(\operatorname{rank}(\mathbf X)=p\). Estimating \(\sigma^2\) from residuals also requires \(n>p\). These are design requirements.
Why distinguish them? A correct mean and full column rank give unbiased OLS estimates. Normality is not needed for that result. Constant variance and uncorrelated errors give the usual OLS covariance formula. Normality adds the exact \(t\) and \(F\) distributions from Chapter 3 and Chapter 4.
Four assumptions, four visual questions. The mean locates the centre. Variance describes the spread. Normality describes the shape. Independence concerns links between observations.
Table 7.1: Match each assumption to its diagnostic. A reassuring plot is evidence, not proof.
Assumption
What to inspect
Reassuring pattern
Pattern to investigate
Correct mean
Residuals versus fitted values or predictors
Local centres stay near zero
A curve or group shift
Constant variance
Residual widths; Scale-Location plot
Similar widths
A fan or changing widths
Independence
Study design; time and lag plots
No clear order pattern
Runs, cycles or linked neighbours
Normality
Normal Q-Q plot of standardized residuals
Points follow a straight line
Systematic bending or extreme tails
NoteRead the plots in a useful order
Check the study design first. Then check mean → spread → shape. Refit and repeat after a model change.
We observe residuals \(e_i=y_i-\hat y_i\), not the true errors. A residual may contain a mean pattern that the model missed. Fix that pattern before interpreting spread or normality.
1. Correct mean: follow the centre of the cloud. In Figure 7.3, read the top row first. The purple line joins the average residual at each predictor value. A flat line near zero is reassuring. A U-shaped line indicates missed curvature.
2. Constant variance: compare the widths of the cloud. Now read the bottom row. Both centres stay near zero. The orange dashed lines show the vertical extent of each group. The widening fan on the right points to changing variance.
Figure 7.3: Centre versus width. These four constructed datasets are each fitted with a straight line and an intercept. All residuals average to zero overall. Top: check the local centre. Bottom: check the local width. The axes are identical across panels.
Table 7.2: Choose the next step from the feature that fails.
What you see
What it suggests
Next step
Curved centre
The mean model misses a pattern
Try a meaningful predictor function or interaction.
Stable centre, changing width
Error variance may change
Consider a response transformation or variance model.
With an intercept, OLS automatically gives \(\sum_i e_i=0\). This does not verify the mean assumption. Look for local patterns, including patterns against individual predictors.
R’s Scale-Location plot uses \(\sqrt{|r_i|}\), where \(r_i\) is a standardized residual. Large values mean large residual magnitudes. Look for a level trend, not a line at zero. Standardization adjusts for leverage under the working model. It does not fix heteroscedasticity. See the Chapter 6 supplement.
3. Independence: keep the observation order. Imagine measurements taken once per minute. In Figure 7.4, the left dataset uses independent normal errors. The right uses positively correlated normal errors. Both generating models have constant error variance.
Read the top row in time order. Long runs suggest that nearby errors may be linked. Then read the bottom row. Each dot pairs two neighbours: \((e_{i-1},e_i)\). An upward pattern suggests persistence. Positive residuals often follow positive residuals. Negative residuals often follow negative residuals.
Figure 7.4: Check order as well as distribution. Top: residuals in collection order. Bottom: each residual against its predecessor. Left: independent normal errors. Right: stationary normal errors with lag-one correlation 0.85. Each dataset is fitted with an intercept.
Use actual collection order. Do not sort by residual size or use arbitrary row numbers. Repeated people, shared sites and spatial neighbours also need attention. Inspect within-group patterns and the study design. A lag plot only checks selected forms of dependence. An unstructured cloud does not prove independence. Lag-plot guide.
If dependence is plausible, model the time or group structure. A log transformation alone will not remove it. The mixed-effects appendix gives one approach for grouped observations.
4. Normality: compare matching percentiles. Q-Q means quantile versus quantile. A normal Q-Q plot asks: are the small, middle and large residuals in the proportions expected from a normal distribution?
Example 7.3 (Build a Q-Q Plot from Five Residuals) Take the sorted toy residuals \(-2,-1,0,1,2\).
Give the \(i\)th value a plotting position \(p_i=(i-0.5)/5\).
Find the matching standard normal quantile \(z_i=\Phi^{-1}(p_i)\).
Plot \((z_i,e_{(i)})\): normal benchmark horizontally, observed residual vertically.
Here \(e_{(i)}\) is the \(i\)th smallest residual. Sorting changes the order, not the values.
Rank
Plotting percentile
Normal quantile (x)
Residual (y)
1
10%
-1.28
-2
2
30%
-0.52
-1
3
50%
0
0
4
70%
0.52
1
5
90%
1.28
2
The last row pairs the 90th percentile with the 90th percentile. Its plotted point is approximately \((1.28,2)\). The vertical coordinate is a residual value, in response units.
Figure 7.5: Each table row becomes one point. Labels show the plotting percentiles. The orange guides locate the 90% pair. The purple reference line passes through the first and third quartiles. Five points explain the construction; they do not establish normality.
NoteWhy should the points form a line?
For a \(N(\mu,\sigma^2)\) distribution, the \(p\)th quantile is \(\mu+\sigma\Phi^{-1}(p)\). Thus matching normal quantiles gives a straight line: vertical value = centre + scale × horizontal value.
The intercept reflects location. The slope reflects scale. Normality concerns the shape around this line; the slope need not equal 1. This is a distribution comparison, not the fitted regression line of \(Y\) on \(X\).
For regression, usually use standardized residuals from rstandard(fit). Then qqnorm() plots them and qqline() adds a quartile reference line. Software may use slightly different plotting positions. The reading is the same. R Q-Q documentation.
Read departures from the line. In Figure 7.6, the horizontal benchmark is always normal. Only the vertical distribution changes. These idealized quantiles make the patterns clear. Real samples add random scatter.
Figure 7.6: Three idealized Q-Q patterns, all with mean zero and variance one. Purple lines match the theoretical quartiles. Compare each tail with its reference line. Real samples add scatter.
Table 7.3: Interpret the tails using the axis directions shown above.
Q-Q pattern
Read the points
Meaning to investigate
Approximately straight
Small, middle and large values match a normal shape
Normality is plausible. Allow sampling scatter.
Upper end bends upward
Large residuals exceed the normal benchmark
A long right tail or right skew
Left end below; right end above
Both extremes exceed the normal benchmark in magnitude
Heavier tails than normal
One isolated departure may be an unusual observation. Use the Chapter 6 checks to investigate it. A whole curved pattern suggests a broader distributional issue. Normal probability-plot guide.
ImportantA straight Q-Q plot does not check every assumption
A Q-Q plot sorts the residuals. It discards time order. It therefore cannot establish independence. It also pools residuals across predictor values, so it cannot establish constant conditional variance.
Check the mean and spread first. Then read the Q-Q plot. Neither the predictors nor the raw response must have a normal histogram.
Now read real diagnostics. The tree-volume model below uses diameter and height. Read its panels in the same order: centre, width, then shape. We revisit this model in Section 7.3.
Figure 7.7: Original-scale tree-volume diagnostics: mean pattern (left), spread (middle), and normality (right). Read the mean pattern first, then reconsider spread and shape after refitting. None of these panels establishes independence between trees.
The residual smooth is curved. Revisit the mean model first. The Q-Q plot cannot settle that issue. These data lack collection times and group identifiers. Independence needs information about how the trees were sampled.
R plot numbers. Use which = 1 for residual mean patterns, which = 3 for spread, and which = 2 for normal shape. The default which = 5 panel concerns leverage and influence. Check independence separately. R diagnostic-plot documentation.
Exercise 7.1 (Read the Tree Diagnostics) Look at the three panels above. Which model component would you revisit first? Which assumption cannot be assessed from these panels?
Solution 7.1. Revisit the mean first: the residual centre curves. Then recheck spread and the Q-Q plot. These panels do not assess independence between trees. We need sampling information and any relevant time or group identifiers.
The pictures now guide the next step. A transformation may improve the mean form, stabilize variance, or improve the error shape. Choose it for a stated reason. Then refit and check again.
7.3 Transform the model, fit, and check again
A curved relationship can still be a linear model. A transformation is useful when it gives a plausible conditional mean and error structure. It need not make the original response a straight line in the original predictor.
Definition 7.2 (Linear in the Parameters) A regression model is linear in its parameters when it can be written as \(Y_i=\beta_0+\sum_{j=1}^d\beta_j h_j(\mathbf x_i)+\varepsilon_i\), where the functions \(h_j\) are specified independently of the unknown coefficients. The design row is \((1\;h_1(\mathbf x_i)\;\cdots\;h_d(\mathbf x_i))\); the response need not be a straight-line function of the original predictors. More generally, a transformed-response model is
It describes \(E\{g(Y_i)\mid\mathbf x_i\}\), not automatically \(g\{E(Y_i\mid\mathbf x_i)\}\). Constant variance and normality, when assumed, now refer to errors on the transformed response scale.
Relationship
Model fitted by least squares
What changes?
Curvature in \(x\)
\(Y=\beta_0+\beta_1x+\beta_2x^2+\varepsilon\)
Add a polynomial column; retain the response scale.
Diminishing change, \(x>0\)
\(Y=\beta_0+\beta_1\log x+\varepsilon\)
Transform a predictor.
Exponential growth, \(Y>0\)
\(\log Y=\beta_0+\beta_1x+\varepsilon\)
Transform the response; specify errors on the log scale.
Power relationship, \(Y,x>0\)
\(\log Y=\beta_0+\beta_1\log x+\varepsilon\)
Transform both variables.
In R, write a quadratic as lm(y ~ x + I(x^2), data = dat); I() makes the square arithmetic. For a log-log model, use lm(log(y) ~ log(x), data = dat). An interaction from Chapter 5 is another constructed column, \(x_1x_2\).
ImportantTransform the statistical model, including its errors
\(Y=a\exp(bx)\exp(\varepsilon)\) becomes \(\log Y=\log a+bx+\varepsilon\). In contrast, taking logs of \(Y=a\exp(bx)+\varepsilon\) does not produce that same additive-error model. Likewise, a saturation model such as \(Y=ax/(b+x)+\varepsilon\) is nonlinear in \(a,b\); reciprocating it changes the errors and may be inappropriate. Some problems need nonlinear least squares or another response model, not forced linearization.
Exercise 7.2 (Linear in Which Sense?) Which can be fitted by lm() using known predictor columns?
\(Y=\beta_0+\beta_1\sqrt{x}+\varepsilon\), with \(x\geq0\).
\(Y=\beta_0+\beta_1x^{\beta_2}+\varepsilon\), with unknown \(\beta_2\).
\(\log Y=\beta_0+\beta_1x+\varepsilon\), with \(Y>0\).
Solution 7.2. Models 1 and 3 are linear in their coefficients. Model 2 has an unknown exponent inside the predictor function; ordinary lm() cannot estimate all three coefficients as written. Model 3 models the conditional mean of \(\log Y\), not directly the conditional mean of \(Y\).
Motivation: measuring tree volume. Measuring a standing tree’s diameter and height is easier than measuring its timber volume. The trees from Chapter 5 suggest a physical starting point: volume is roughly proportional to cross-sectional area times height, so a power relationship such as \(V\approx aD^bH^c\) may be more natural than an additive straight-line model. The exponents are to be estimated, not fixed by the cylinder analogy. Data documentation.
Apply what the two examples showed.
State the goal and inspect the data. Identify the response units, positive/zero values, measurement process, and a scientifically plausible shape. For prediction, reserve test observations before exploring relationships.
Fit a baseline and inspect residuals. Curvature suggests a mean-shape problem; a fan-shaped spread suggests nonconstant variance. A skewed marginal response alone is not a reason to transform it: regression assumptions concern conditional errors.
Choose a small candidate set. A predictor transformation can address shape while keeping response units. A response transformation changes the target and error model; log responses are natural for positive multiplicative variation.
Refit and recheck. Examine residuals versus fitted values and predictors, a normal Q-Q plot when using normal-model inference, and influential observations. A transformation does not fix dependence or a poor study design.
Interpret and validate. Explain the new coefficients, back-transform appropriately, and compare predictions on a common scale. If choices are learned from data, repeat that learning within training folds.
Example 7.4 (From Tree Geometry to a Log-Linear Model) For the 31 trees, consider
Here \(D\) is diameter in inches, \(H\) is height in feet, and \(V\) is volume in cubic feet. We use all 31 trees to illustrate specification and diagnostics; this is not a held-out performance claim.
Show R code
library(ggplot2)ch7_trees <-transform(datasets::trees, Diameter = Girth)ch7_raw <-lm(Volume ~ Diameter + Height, data = ch7_trees)ch7_log <-lm(log(Volume) ~log(Diameter) +log(Height), data = ch7_trees)
Figure 7.8: The same 31 trees on two scales, each with a simple least-squares line using diameter alone. The log-log view motivates a power relationship; the fitted multiple regression above also adjusts for height. This plot does not establish that the transformed model is adequate.
The estimated diameter and height exponents are about 1.983 and 1.117, close to the geometric motivation but not constrained to it. Holding height fixed, multiplying diameter by \(r\) changes the fitted log-volume by \(\hat b\log r\). Under the lognormal model below, the fitted median volume is multiplied by \(r^{\hat b}\). Thus a 1% diameter increase corresponds approximately to a \(\hat b\)% change in that median; the exact percentage is \(100(1.01^{\hat b}-1)\).
Figure 7.9: Standardized residuals from the additive original-scale model and the log-log model, both using diameter and height. The smooth orange line helps reveal remaining mean structure. Inspect each pattern; their fitted-value scales differ, so raw SSE or R-squared values across response scales would not be a fair comparison.
The log model is a candidate to examine, not a guaranteed cure. Both panels still deserve scrutiny. With only 31 observations, a large search over powers and interactions would be hard to justify.
Definition 7.3 (Back-Transformation Under Lognormal Errors) If \(\log Y=\eta(\mathbf x)+\varepsilon\) with \(\varepsilon\mid\mathbf x\sim N(0,\sigma^2)\), then
Exponentiating a fitted log mean therefore estimates a conditional median, not the original-scale mean. A plug-in mean prediction uses an estimate of \(\sigma^2\); it is not an exactly unbiased estimator.
Predicted cubic feet for diameter 15 inches and height 80 feet.
Median volume
Mean volume, lognormal correction
37.82
37.94
With homoscedastic errors independent of the predictors, a less distribution-specific alternative is the smearing estimate \(\exp(\hat\eta)\,n^{-1}\sum_i\exp(e_i)\), where \(e_i\) are training log residuals. A single correction can fail if the error distribution changes with the predictors. Estimate any correction within each training fold when validating.
Other powers. For positive responses, Box-Cox considers \(g_\kappa(y)=(y^\kappa-1)/\kappa\) for \(\kappa\ne0\) and \(g_0(y)=\log y\). Thus \(\kappa=1\) leaves the response shape unchanged, \(\kappa=1/2\) gives a square-root scale, and \(\kappa=0\) gives logs. MASS::boxcox(ch7_raw) profiles the transformation parameter for the specified mean model. Treat it as guidance, then refit and inspect; choosing the power is itself model selection. Box-Cox documentation.
WarningKeep scales and data comparable
Do not rank an lm(y ~ ...) fit against an lm(log(y) ~ ...) fit using their raw SSE, \(R^2\), or unadjusted software AIC. Their responses differ. Compare held-out predictions of the same target in the same units, or use likelihoods expressed on a common response scale with the transformation Jacobian. Logs require positive values; adding an arbitrary constant changes the model. Binary or count responses may instead call for a GLM.
Exercise 7.3 (What Does Exponentiating Predict?) Suppose \(\log Y=2+0.4x+\varepsilon\), with \(\varepsilon\sim N(0,0.25)\). At \(x=0\), give the conditional median and mean of \(Y\). Does increasing \(x\) by one add 0.4 units to the original-scale mean?
Solution 7.3. The median is \(e^2\approx7.39\) and the mean is \(e^{2+0.25/2}\approx8.37\). A one-unit increase in \(x\) multiplies both by \(e^{0.4}\approx1.49\) because the log-error variance is constant. It does not add 0.4 response units.
NoteRecheck the fitted model before selecting variables
A new response scale changes residuals, and new predictor columns can change leverage. Apply the Chapter 6 diagnostics to the transformed fit. For the tree model, the largest Cook’s distance belongs to tree 18 (\(D_i\approx0.224\)). A sensitivity fit gives:
Show R code
ch7_flag <-which.max(cooks.distance(ch7_log))ch7_log_without <-update(ch7_log, data = ch7_trees[-ch7_flag, ])knitr::kable(data.frame(Term =names(coef(ch7_log)),All_trees =unname(coef(ch7_log)),Omit_flagged_tree =unname(coef(ch7_log_without))),digits =3, row.names =FALSE,col.names =c("Term", "All trees", "Omit flagged tree"))
Term
All trees
Omit flagged tree
(Intercept)
-6.632
-7.185
log(Diameter)
1.983
1.958
log(Height)
1.117
1.261
The height exponent changes more than the diameter exponent. This is a reason to investigate and report sensitivity, not to delete an accurate measurement automatically. An influential observation can also change which variables a search retains. Resolve data-quality and model-form questions within training data, and recheck the selected model. Squared-error ridge, lasso, and elastic net are not automatic remedies for outliers.
7.4 From candidate terms to variable selection
Transformation asks how a variable should enter the model; variable selection asks which candidate terms to retain. In Section 7.1, replacing \(x\) by \(\log_2 x\) kept the same predictor but changed its relationship with the response. A selection method choosing among only untransformed straight-line terms could not discover that relationship.
Now suppose several predictors are available. Adding columns cannot increase training SSE, even when they represent noise, but estimating more coefficients can make predictions less stable. Selection balances fit against complexity within a scientifically defensible candidate set. Specify plausible transformations and required terms first; if their choice is data-driven, include it in the validation procedure in Section 7.6.
Definition 7.4 (Variable Selection) Start with the candidate regression model \(Y_i=\beta_0+\sum_{j=1}^d\beta_jx_{ij}+\varepsilon_i\). The columns may already include justified transformations. For a subset \(S\subseteq\{1,\ldots,d\}\), fit
where the estimates minimize \(\mathrm{SSE}_S=\sum_i(y_i-\hat y_{i,S})^2\) on the training observations. Every candidate subset is refitted; we do not simply remove coefficients from the full-model estimate. A subset can miss a mean contribution, so its mean-zero error assumption must be assessed rather than presumed.
Variable selection chooses \(S\) using a stated criterion and search rule. A term may span several columns, such as a factor. Specify the candidate terms, required terms, comparison criterion, and stopping rule before running the search.
Choose the rule before the search. For Gaussian models fitted to the same observations and response, \(\mathrm{AIC}=-2\hat\ell+2K\) and \(\mathrm{BIC}=-2\hat\ell+K\log n\), where \(K=p_S+1\) includes the subset model’s \(p_S\) regression coefficients (including the intercept) and its variance parameter. Smaller values are preferred. Ordinary \(R^2\) alone cannot stop a search because it never decreases when columns are added. These criteria are scores, not significance levels. R documentation.
Method
Start
One step
Stop
Forward selection
Required terms, often just the intercept
Try each admissible addition; take the greatest score improvement.
No addition improves the score.
Backward elimination
Full candidate model
Try each admissible removal; take the greatest improvement.
No removal improves the score.
Stepwise selection
A specified starting model
Reconsider additions and removals as the fit changes.
Neither move improves the score.
These are greedy searches; different starting points or directions can yield different models. Best subset selection evaluates every allowed subset and finds the best score within that finite set, but \(d\) optional terms can require \(2^d\) fits. Backward elimination requires an estimable starting model with positive residual degrees of freedom for the usual Gaussian criterion.
ImportantProtect model structure
Keep treatment assignments, blocking variables, and scientifically required adjustment terms when the question requires them. Retain parent terms with interactions and lower powers with polynomial terms. Search a multi-level factor as a whole term. R’s formula-based step() respects interaction hierarchy, but it will not infer every scientific constraint, such as retaining x whenever I(x^2) is present.
Example 7.5 (Which Sensor Measurements Help Predict an Outcome?) Imagine a manufacturing outcome and eight candidate sensor readings. We simulate data so the useful predictors are known: \(Y=5+3x_1-2x_2+1.5x_3+\varepsilon\), with independent \(N(0,4)\) errors. Predictors \(x_1,x_2\) are correlated; \(x_4,\ldots,x_8\) have no additional mean contribution. All methods receive the same eight candidates. This is a teaching simulation, not an industrial dataset.
We reserve 40 observations now and fit all searches on the other 120. The candidate set contains only main effects, so no polynomial or interaction hierarchy needs to be imposed in this example.
See the first forward step. Start with an intercept-only model. Fit each of the eight one-predictor alternatives on the same 120 training observations; the table shows the starting model and the three best additions. Use AIC() consistently, and choose the lowest score only if it improves on the starting score.
The first move adds x1. AIC falls from 646.33 to 604.57. The next step must compare additions conditional on that retained term; it cannot simply reuse the remaining one-predictor rankings.
Here all three searches retain \(x_1,x_2,x_3,x_4\). The first three generate the mean, but \(x_4\) is noise: selection can retain a chance association.
step() uses AIC by default; k = log(nrow(ch7_train)) changes the penalty to the BIC-style rule. Fix the analysis rows across all candidates, especially when missing values occur. The trace uses extractAIC(), whose additive constant differs from AIC() for lm; compare scores computed with the same function. Selection documentation.
Exercise 7.4 (Take One Forward Step) The current AIC is 210. Adding \(x_2\), \(x_3\), or \(x_4\) gives 207, 209, or 212. Which move is made? At the next step, the best available AIC is 207.4. Continue or stop? Would a different answer prove that one variable has no scientific role?
Solution 7.4. Add \(x_2\), producing the largest AIC decrease. Stop at the next step because 207.4 is worse than 207. This is a conditional search result under the chosen candidates and score, not proof that an excluded variable has no scientific role.
7.5 Shrinkage: ridge, lasso, and elastic net
Subset selection drops a term completely or estimates its coefficient without a penalty. Regularization provides another way to control complexity: keep the candidate mean model, but penalize coefficient size when fitting it. This introduces bias and can reduce sampling variability and prediction error. Lasso and elastic net can also remove columns by assigning zero coefficients.
The regression model stays explicit. Let \(z_{ij}=(x_{ij}-\bar x_j)/s_j\) denote training-standardized predictor columns. For all three methods, the working model is
Here \(\mathbf Z\) contains the standardized columns. One may additionally assume independent \(N(0,\sigma^2)\) errors for a Gaussian model, but normality is not required to compute a squared-error fit. A log response or transformed predictor can be used if the preceding model checks justify it. The methods below change how the coefficients are estimated, not what a linear conditional mean means.
For a training sample of size \(n_{\mathrm{tr}}\), define the common loss
In this section \(\boldsymbol\beta\) contains the slopes; the intercept \(\beta_0\) is written separately and is not penalized. The \(1/(2n_{\mathrm{tr}})\) convention matches the Gaussian objective in the glmnet guide; changing the loss scaling changes the numerical meaning of \(\lambda\).
Definition 7.5 (Ridge Regression) Ridge estimates the linear model by minimizing squared error plus a squared-coefficient penalty:
Its fitted regression function is \(\hat m_R(\mathbf z)=\hat\beta_0^R+\sum_j\hat\beta_j^Rz_j\). Larger \(\lambda\) discourages large slopes more strongly. Ridge generally retains every column; it is primarily a shrinkage method, not a variable-selection rule.
Why use it? If two sensors are strongly correlated, many coefficient combinations can produce similar predictions. OLS coefficients can then change sharply between samples. Ridge discourages extreme compensating coefficients and stabilizes the fit. With centred columns and response, its slope estimate is
For \(\lambda>0\), the inverse exists even with linearly dependent columns or \(d\geq n_{\mathrm{tr}}\). This makes estimation possible; it does not make unsupported scientific effects identifiable from the data.
Definition 7.6 (Lasso Regression) Lasso uses the same linear model with an absolute-value penalty:
Its fitted regression function is \(\hat m_L(\mathbf z)=\hat\beta_0^L+\sum_{j\in\hat S_L}\hat\beta_j^Lz_j\), where \(\hat S_L=\{j:\hat\beta_j^L\ne0\}\). The corner of \(|\beta_j|\) at zero permits exact zero estimates: lasso performs shrinkage and column selection together.
Why use it? A sparse rule can be easier to deploy when collecting many measurements is costly. However, correlated predictors can compete for the same signal: a small sample change may alter which one is retained. A zero coefficient is an outcome of this penalty and dataset, not proof that the variable is scientifically irrelevant.
Definition 7.7 (Elastic Net Regression) Elastic net combines the ridge and lasso penalties in the same linear model:
Its fitted regression function is \(\hat m_E(\mathbf z)=\hat\beta_0^E+\sum_{j\in\hat S_E}\hat\beta_j^Ez_j\), where \(\hat S_E=\{j:\hat\beta_j^E\ne0\}\). The absolute-value part can remove columns, while the squared part stabilizes coefficients of correlated columns.
Two tuning parameters, two roles.\(\lambda\) controls the overall penalty strength; \(\alpha\) controls its mixture. At \(\alpha=0\) the objective is ridge, and at \(\alpha=1\) it is lasso. At \(\lambda=0\) all reduce to OLS, with uniqueness depending on design rank. Here \(\alpha\) is a mixing parameter, not a significance level. Elastic net can retain related predictors together, but it does not guarantee selection of the true variables.
Example 7.6 (See How a Small Coefficient Becomes Zero) Consider centred, mutually orthogonal predictor columns scaled so that \(\mathbf Z^\top\mathbf Z/n_{\mathrm{tr}}=\mathbf I_d\). Let \(b_j\) be a slope from unpenalized OLS. In this special case, each slope is fitted separately:
Here \((u)_+=\max(u,0)\). Thus ridge divides continuously; lasso first subtracts a threshold from the magnitude and sets small values to zero; elastic net combines thresholding with division. These separate-coordinate formulas require the orthogonal design. They are not formulas for fitting the correlated sensor data below.
Figure 7.10: An orthogonal-design illustration with lambda = 0.5 and elastic-net alpha = 0.5. Dashed lines show no shrinkage. The flat portions at zero show how lasso and elastic net remove small coefficients; ridge continuously shrinks them.
Exercise 7.5 (Which Method Removes This Column?) In the orthogonal example, an OLS slope is \(b_j=0.3\). Use \(\lambda=0.5\) for all methods and \(\alpha=0.5\) for elastic net. Find the three penalized slopes. Why are the lasso and elastic-net answers different?
Solution 7.5. Ridge gives \(0.3/1.5=0.2\); lasso gives \((0.3-0.5)_+=0\); elastic net gives \((0.3-0.25)/1.25=0.04\). Elastic net uses threshold \(\lambda\alpha=0.25\), then applies ridge shrinkage. Equal \(\lambda\) does not mean equal shrinkage across methods; tune each method for the intended prediction task.
Scale using training data. Converting metres to millimetres divides the corresponding slope by 1,000 without changing predictions, but it changes either penalty. Standardization makes the penalty comparable across units. With standardize = TRUE, glmnet scales within each fit, including CV training folds, and returns coefficients and predictions on the original input scale. Do not standardize the entire dataset before splitting it. Ordinary lasso/elastic net select columns; they do not automatically retain a whole factor or preserve interaction hierarchy.
Choose the penalty before opening the test set. On the 120 sensor training observations, use the same five folds for each candidate:
For each fixed \(\alpha\), fit a sequence of \(\lambda\) values using four folds and score the fifth; repeat over all folds.
lambda.min minimizes average validation MSE. lambda.1se is the largest \(\lambda\) within one estimated standard error of that minimum, favouring more shrinkage. This is a practical rule, not a hypothesis test.
For elastic net, compare a prespecified grid \(\alpha\in\{0.25,0.5,0.75\}\) using each curve’s minimum CV MSE. Choose the best \(\alpha\), then use its lambda.1se. This is our stated tuning rule, not a unique definition of an optimal elastic net.
A single cv.glmnet() call tunes \(\lambda\) at a fixed\(\alpha\); it does not choose both. Reusing the same folds makes comparisons less sensitive to different random partitions. The selected CV score is a tuning score, not an unbiased final performance estimate. CV documentation.
The selected elastic-net mixture is \(\alpha=\)0.75. Each CV object also stores fits to all 120 training observations; use the fit at the selected penalty. For example, predict(ch7_enet, newx = ch7_xtest, s = "lambda.1se") uses that fitted rule on new sensor readings.
These are coefficients in the original sensor units, so read each column as \(\hat y_M=\hat b_{0,M}+\sum_{j=1}^8\hat b_{j,M}x_j\), for \(M\in\{R,L,E\}\). Zero entries remove columns from that prediction rule. The full-precision coefficients, not the rounded table, are used for prediction. Compare how the three methods handle the known signal \(x_1,x_2,x_3\) and the five noise columns.
Read the path and the validation curve. The coefficient path shows how the fitted rule changes as the penalty increases. The CV curve asks which penalty predicts held-out training folds well; the path alone cannot answer that question.
Figure 7.11: Lasso coefficient paths from the sensor training data. Larger penalties move slopes toward zero. The three true signal predictors are colored; the other five are gray. A path displays fitted coefficients, not confidence intervals.
Figure 7.12: Five-fold CV within the sensor training set. Bars show plus or minus one estimated CV standard error, not confidence intervals for individual responses. Dashed: minimum-error lambda. Solid: the more regularized one-SE choice used below.
7.6 Evaluate the complete procedure
Our choices are now fixed: AIC for the three subset searches, the one-SE rule for ridge and lasso, and the stated \(\alpha\)-grid/one-SE rule for elastic net. Evaluate these six procedures on the same untouched 40 observations. Add full-model OLS and a training-mean baseline for context. For \(m\) test observations, \(\mathrm{RMSE}=\{m^{-1}\sum_i(y_i-\hat y_i)^2\}^{1/2}\), in response units.
Read this table together with the coefficient table: fewer retained columns and lower test error are different goals. A penalized model may retain a noise predictor or shrink a true signal. Equal search results are possible, and no method must win on every split. Report the comparison without revising the methods after looking at test outcomes. If choosing a winning method from this table, its same test error is no longer an independent final assessment of that choice.
ImportantValidate the search, not just its final formula
If CV is used to evaluate a selection procedure, rerun the entire variable search inside each training fold. Selecting on all observations first and then cross-validating only the selected formula leaks information. Learned transformations, imputation, scaling, and tuning follow the same rule. Final performance claims require an untouched test set or appropriately nested resampling.
Prediction and inference answer different questions. A selected coefficient is not automatically a scientific discovery; ordinary fixed-model \(t\) tests and confidence intervals do not automatically remain valid after selection or lasso followed by OLS refitting. Likewise, a good predictive score does not establish a causal effect.
Exercise 7.6 (Find the Leakage) A student selects predictors using all 160 outcomes, then splits the observations and reports test RMSE. Another student reserves 40 observations first and runs selection only on the other 120. Which analysis gives an honest held-out assessment of its fitted procedure? Does that make its selected-coefficient p-values valid?
Solution 7.6. Only the second keeps test outcomes out of model development, provided the test results are not used to revise the procedure. That separation assesses prediction; it does not repair ordinary post-selection coefficient inference.
7.7 Summary
What do you see or need?
What should you try?
What must you check?
Mean, spread, or distribution assumptions are doubtful
Identify which feature fails using the design and targeted diagnostics.
Normality and constant variance are different; transformations do not fix dependence.
Curved mean, roughly constant spread
A predictor transformation or suitable polynomial terms; keep \(Y\) in its units when appropriate.
Does the residual mean pattern disappear? Interpret the new predictor scale.
Positive responses with proportional variation
A log response, possibly together with transformed predictors.
Recheck shape and spread; distinguish the original-scale mean from an exponentiated log prediction.
Many plausible candidate terms
Forward, backward, or stepwise selection with a stated score and required terms.
Preserve factors and hierarchy; selected terms and ordinary p-values are not automatic discoveries.
Unstable estimates or many correlated columns
Ridge shrinks; lasso also selects; elastic net combines shrinkage and selection for correlated columns.
Scale within training data and tune penalties by CV.
A final claim about prediction
Evaluate the entire fixed procedure on untouched observations.
Use the same target and units; learn transformations and selection within training folds.
A transformation is unnecessary when the existing model is adequate for the purpose. A curved relationship can still be linear in its coefficients. Whatever the chosen form, recheck the Chapter 6 diagnostics.
The next design unit asks which terms must be retained because of randomization, blocking, and the scientific question. That provides a useful boundary for automatic selection. Dose-specific comparisons from Chapter 5 are retained in Contrasts and Estimable Comparisons (Optional).
7.8 Practice Problems
Exercise 7.7 (Practice: Specify, Select, and Assess)
A positive outcome grows approximately exponentially with time. Write a multiplicative-error model and its log-linear form. Explain why an additive original-scale error would change the argument.
A quadratic mean fits better than a straight line. Does it cease to be a linear regression model? State the columns of its design matrix.
A backward search considers removing a three-level factor. How many coefficient restrictions are imposed when there is an intercept and no interaction? Why should its indicators be treated together?
Explain why ridge is not generally a variable-selection method and why lasso selections can vary among correlated predictors.
Compare lambda.min with lambda.1se. Where should the folds, transformation choices, and final test observations enter the procedure?
Errors are independent and normal, with mean zero and variance \(\sigma^2x_i^2\) for \(x_i>0\). Which assumption fails? Why is a normal Q-Q plot alone insufficient to diagnose it?
Write the elastic-net regression model and objective for four standardized predictors. Explain which parameters are estimated from the data, which are tuned, and why \(\alpha=0.5\) is not a 50% test level.
Solution 7.7.
\(Y=a\exp(bt)\exp(\varepsilon)\) gives \(\log Y=\log a+bt+\varepsilon\). Logging \(a\exp(bt)+\varepsilon\) does not yield this additive log-error structure.
It remains linear in the coefficients; the columns are \(\mathbf 1\), \(\mathbf x\), and the vector of squared predictor values.
Two restrictions. The two columns represent one factor; selecting them separately makes the search depend on the chosen reference coding.
Ridge usually leaves all slopes nonzero. Correlated predictors offer similar information, so small sample changes can alter which one lasso retains.
lambda.min minimizes CV error; lambda.1se prefers the largest penalty within one estimated SE of that minimum. Choose transformations and tune models within training data/folds, then evaluate the fixed procedure on an untouched test set.
Constant variance fails when the predictor values vary. Normality at each predictor value does not imply equal spread. Inspect residual spread against the predictor or fitted values; a pooled Q-Q plot does not isolate the source of a variance pattern.
Use \(Y_i=\beta_0+\sum_{j=1}^4\beta_jz_{ij}+\varepsilon_i\), with conditional error mean zero. Minimize \(L+\lambda\{(1-\alpha)\sum_{j=1}^4\beta_j^2/2+\alpha\sum_{j=1}^4|\beta_j|\}\). Estimate the intercept and four slopes for each tuning choice; select \(\lambda\) and \(\alpha\) within training data. At \(\alpha=0.5\), the objective mixes squared and absolute-value penalties; it does not define a test.