FRM Part 1 · Quantitative Analysis · Chapter QTA 9
Regression Diagnostics
Learning Objectives
- Explain how to test whether a regression is affected by heteroskedasticity.
- Describe approaches to using heteroskedastic data.
- Explain the concept of multicollinearity and differentiate between multicollinearity and perfect collinearity.
- Describe and illustrate the consequences of excluding a relevant explanatory variable from a model, and contrast those with the consequences of including an irrelevant regressor.
- Explain two model selection procedures and how these relate to the bias-variance tradeoff.
- Describe the various methods of visualizing residuals and their relative strengths.
- Describe methods for identifying outliers and their impact.
- Determine the conditions under which OLS is the best linear unbiased estimator.
The previous two chapters showed how to fit a regression and read its output. This one asks the harder question: can the output be trusted? A regression runs on almost any data and always produces numbers, but those numbers are only meaningful when the model is well specified and its assumptions hold. Regression diagnostics are the checks that catch the failures, a missing variable, a wrong functional form, unequal error variance, near-duplicate regressors, a single wild data point, before they mislead a decision.
Two threads organize the chapter. The first is specification: which variables belong in the model. Leaving out a relevant one biases everything; throwing in irrelevant ones just adds noise, and balancing the two is the bias-variance tradeoff that drives model selection. The second is assumption checking: tests and plots for heteroskedasticity, multicollinearity, and outliers, the three defects that most often quietly corrupt a financial regression. The chapter ends by pinning down exactly when OLS earns its status as the best linear unbiased estimator.
Key Takeaways
- Omitting a relevant, correlated variable biases the included coefficients; including an irrelevant one leaves them unbiased but less precise.
- The bias-variance tradeoff balances a larger model’s lower bias against its higher variance; total error is squared bias plus variance.
- General-to-specific selection prunes insignificant variables; cross-validation picks the model with the best out-of-sample fit.
- Heteroskedasticity leaves OLS unbiased but invalidates the standard errors; White’s test detects it, and robust standard errors, transformations, or weighted least squares handle it.
- Multicollinearity is high but not perfect correlation among regressors; it inflates standard errors and is measured by the variance inflation factor.
- Residual plots and Cook’s distance expose misspecification and influential outliers; under the standard assumptions, OLS is BLUE by the Gauss-Markov theorem.
Omitting a Relevant Variable
The most damaging specification error is leaving out a variable that truly belongs. An omitted variable is one with a non-zero coefficient in the true model that is missing from the estimated one. When it is correlated with a variable that is included, the included variable is forced to do double duty, absorbing not only its own effect but also the part of the missing variable’s effect that travels with it. The result is a biased and inconsistent coefficient: wrong on average, and wrong even in an infinite sample.
where β2 is the true coefficient on the omitted variable and δ (delta) is the slope of a regression of the omitted variable on the included one. The bias is the product of the two, so it vanishes only when the omitted variable is genuinely irrelevant (β2 = 0) or uncorrelated with the included variable (δ = 0).
A model of a stock’s return uses only the market factor, omitting a size factor. The stock has a genuine positive size exposure (βsize > 0), and in the data the size factor is positively correlated with the market (δ > 0). Which way is the market coefficient biased?
Answer: upward. The bias is the product βsize × δ, and both terms are positive, so the estimated market beta is pulled above its true value. The missing size exposure, riding on the market’s correlation with size, gets misattributed to the market. Add the size factor back and the market beta falls to its true level, which is exactly why relevant controls matter.
Including an Irrelevant Variable
The opposite error is milder. An irrelevant, or extraneous, variable is one whose true coefficient is zero. Including it does not bias the other coefficients: in a large sample its own coefficient simply converges to zero, and the rest are estimated correctly on average. The cost is precision. An extra variable competes with the relevant ones to explain the data and consumes a degree of freedom, which inflates the standard errors of all the estimates, and the penalty is worst when the extraneous variable is correlated with the useful ones.
| Omit a relevant variable | Include an irrelevant variable | |
|---|---|---|
| Coefficients | Biased and inconsistent | Unbiased |
| Standard errors | Understated (residuals mislabeled) | Inflated; estimates less precise |
| Severity | Serious: wrong answers | Milder: noisier answers |
| Adjusted R² | Lower than it should be | Usually falls |
The two errors are not symmetric in danger. Omitting a relevant variable biases the estimates, which corrupts the conclusion itself. Including an irrelevant one only adds noise, which widens the confidence intervals but leaves the estimates centered on the truth. Faced with genuine uncertainty about whether a variable belongs, and when the sample is large enough to absorb the precision cost, erring toward inclusion is often the safer mistake.
The Bias-Variance Tradeoff
Those two errors are the poles of the bias-variance tradeoff, the central challenge of choosing a model. A larger model, with more variables, is less likely to omit something relevant, so it has lower bias; but each of its many coefficients is estimated from the same data, so each is less precise, giving it higher variance. A smaller model reverses both. Since total expected squared error is the sum of squared bias and variance, the best model minimizes their combination, and that best model is usually neither the largest nor the smallest.
where reducing bias by adding variables tends to raise variance, and simplifying the model to cut variance tends to raise bias. The minimum of the sum, not of either piece alone, marks the best model.
A small model has a bias of 0.2 and a variance of 0.01. A larger model removes the bias entirely but has a variance of 0.08. Which has the lower expected squared error?
Step 1. Expected squared error is squared bias plus variance.
Answer: the small model wins, with 0.05 against 0.08, despite being biased. A little bias bought a large reduction in variance, and total error is what counts, not bias alone. This is why the unbiased model is not automatically the best: an estimator can be systematically slightly off yet more accurate overall than one that is right on average but wildly variable.
Model Selection Procedures
Two standard procedures navigate the tradeoff. General-to-specific selection starts from a large model containing every candidate variable, then repeatedly drops the least significant variable, the one with the smallest absolute t-statistic, re-estimating after each removal, until every remaining variable is significant at a strict level such as 1 percent. It prunes the model down from the top. Cross-validation takes a different tack: it splits the data into blocks, estimates the model on all but one block, measures its error on the held-out block, rotates through all blocks, and selects the model with the smallest out-of-sample error.
| Procedure | How it works | Selects on |
|---|---|---|
| General-to-specific | Start large, drop the least significant variable, repeat until all are significant | In-sample significance (t-statistics) |
| Cross-validation | Train on part of the data, test on the held-out part, rotate, and compare | Out-of-sample prediction error |
Both embody the bias-variance tradeoff, from opposite directions. General-to-specific guards against variance by refusing to keep variables that do not clearly earn their place. Cross-validation guards against both at once by rewarding whatever model actually predicts new data best, a criterion that automatically punishes an overfit model whose extra variables help in-sample but hurt out-of-sample. Cross-validation is the workhorse of modern data science precisely because out-of-sample accuracy is the honest test of a model.
Heteroskedasticity and White’s Test
Heteroskedasticity means the variance of the regression error is not constant but changes systematically with the explanatory variables, the opposite of the constant-variance homoskedasticity that OLS assumes. It is pervasive in financial data, where volatility clusters and larger values often carry larger shocks. Critically, heteroskedasticity does not bias the OLS coefficients, they remain unbiased and consistent, but it makes the usual standard-error formula wrong, so t-tests and confidence intervals built on it are invalid.
The standard detector is White’s test. It runs the regression, then regresses the squared residuals on the explanatory variables, their squares, and their cross-products. If the squared residuals can be explained by the regressors, the error variance depends on them, which is heteroskedasticity. The test statistic is the sample size times the R-squared of that auxiliary regression, and it follows a chi-squared distribution; a large value rejects the null of homoskedasticity.
where R²aux is the R-squared from regressing the squared residuals on the explanatory variables, their squares, and cross-products. A large statistic means the squared residuals are predictable from the regressors, so the null of constant variance is rejected.
A regression on 200 observations is followed by White’s auxiliary regression of the squared residuals, which returns an R² of 0.08. The 5% chi-squared critical value for the relevant degrees of freedom is about 11. Is there heteroskedasticity?
Step 1. Compute the test statistic, n times the auxiliary R².
Step 2. Compare with the critical value of about 11.
Answer: the statistic of 16 exceeds the critical value, so the null of constant variance is rejected: the data are heteroskedastic. The coefficients are still fine, but the ordinary standard errors cannot be trusted, and robust standard errors should be used for any test.
Working With Heteroskedastic Data
Three approaches handle heteroskedasticity, in increasing order of effort. The simplest is to keep the OLS estimates and swap in heteroskedasticity-robust standard errors, also called White or Eicker-White standard errors, which correct the variance formula without changing the coefficients. This is the everyday fix and is a single option in any statistics package. The second is to transform the data, for instance by taking logarithms or dividing by a scaling variable, which can flatten the error variance; converting a firm’s dividend into a dividend yield, by dividing by price, is a classic example. The third and most involved is weighted least squares, which reweights each observation by the inverse of its error variance so that the transformed errors are homoskedastic before OLS is applied.
Believing heteroskedasticity biases the coefficients. It does not. The estimated slopes and intercept stay unbiased and consistent; only their standard errors are wrong. The correct response is therefore to fix the standard errors, most simply with robust standard errors, not to distrust the coefficients themselves. Confusing a standard-error problem with a bias problem leads people to throw away perfectly good estimates.
Multicollinearity
Multicollinearity is high, but not perfect, correlation among the explanatory variables, so one regressor can be largely explained by the others. It must be distinguished sharply from perfect collinearity. Perfect collinearity, where a variable is an exact linear combination of the others, makes estimation impossible: the coefficients cannot be computed at all. Multicollinearity is milder, the variable is nearly but not exactly explained, so estimation still works; it just becomes imprecise. It is not a violation of any OLS assumption, merely a data limitation.
Its signature is a regression whose coefficients are jointly significant but individually insignificant: the F-test says the variables matter as a group, yet the t-tests cannot pin the effect on any single one, because the correlated variables keep stealing significance from each other. The standard diagnostic is the variance inflation factor (VIF), which measures how much a coefficient’s variance is inflated by its correlation with the other regressors.
where R²j is the R-squared from regressing variable j on all the other explanatory variables. A VIF above 10, corresponding to an R²j above 0.9, is the common warning threshold for serious multicollinearity.
Regressing one explanatory variable on the others in the model gives an R² of 0.90. What is its variance inflation factor, and what does it signal?
Step 1. Apply the VIF formula.
Answer: a VIF of 10, right at the warning threshold. It says the other variables explain 90% of this one, so its coefficient’s variance is inflated tenfold relative to an uncorrelated regressor. The estimate is poorly determined, and one option is to drop or combine the offending variable, though multicollinearity is a precision problem, not a bias problem, so it can also simply be tolerated if prediction is the goal.
Residual Plots
The fastest specification check is to look at the residuals. A residual plot graphs the residuals against an explanatory variable, or against the fitted values, and a well-specified model shows a shapeless, random band of small residuals centered on zero, with no pattern. Any structure is a symptom. A funnel that widens across the plot is heteroskedasticity. A curved, systematic sweep is a missing nonlinear term, the model needs a squared variable or a transformation. A few points stranded far from the band are outliers.
Plotting the standardized residuals, each divided by the estimated error standard deviation, sharpens the reading, because it puts every deviation on a common scale where a value beyond roughly three or four is clearly extreme. Figure 2 shows the three patterns to recognize at a glance.
Outliers and Influence
An outlier is an observation that, if removed, would change the estimated coefficients substantially. Because OLS minimizes squared errors, a single extreme point can drag the whole fitted line toward itself, so one bad data point, a fat-finger price, a stale quote, a one-off event, can distort an entire model. The danger is greatest when the outlier sits far out along the explanatory variable, giving it leverage over the slope.
The standard measure of an observation’s influence is Cook’s distance, which quantifies how much the fitted values shift when that one observation is dropped. A large Cook’s distance flags a point with an outsized effect on the model; the common rule of thumb is that a value above 1 marks an influential outlier worth investigating.
where Dj measures the change in the fitted values when observation j is removed. Values well below 1 indicate ordinary points; values above 1 indicate a single observation that materially moves the estimates.
Detecting an outlier is not the same as deleting it. An influential point may be a genuine, informative extreme, a real crash, not an error, and discarding it can throw away exactly the tail behavior a risk model needs to capture. The right response is to investigate: confirm whether the point is a data error or a true event, and decide accordingly, rather than reflexively removing anything inconvenient.
Exhibit 3 gathers the whole diagnostic toolkit into one place: for each common defect, the symptom that reveals it, the tool that confirms it, and the standard fix.
| Problem | Symptom | Tool | Fix |
|---|---|---|---|
| Omitted variable | Biased coefficients; curved residuals | Theory; residual plot | Add the missing variable |
| Irrelevant variable | Inflated standard errors | t-test; adjusted R² | Drop it if it does not earn its place |
| Heteroskedasticity | Funnel-shaped residuals; invalid tests | White’s test | Robust standard errors, transform, or WLS |
| Multicollinearity | Joint significance, individual insignificance | Variance inflation factor | Drop or combine; or tolerate |
| Outlier | One point shifts the whole fit | Cook’s distance | Investigate, then keep or correct |
When OLS Is BLUE
The chapter closes by pinning down what makes OLS the estimator of choice. Under the standard regression assumptions, mean-zero errors, constant error variance, no correlation across observations, and no perfect collinearity, OLS is the Best Linear Unbiased Estimator, or BLUE. “Best” means smallest variance; “linear” means a weighted sum of the data; “unbiased” means correct on average. So among every estimator that is both a linear function of the data and unbiased, none beats OLS on precision. This result is the Gauss-Markov theorem.
BLUE is a claim within a class, not an absolute crown. OLS is the best only among linear, unbiased estimators. A biased or nonlinear estimator can sometimes beat it, which is exactly the door the bias-variance tradeoff opens. But add one more assumption, that the errors are normally distributed, and OLS strengthens to the best among all unbiased estimators, linear or not. The efficiency of OLS is therefore only as strong as the assumptions behind it, which is the whole reason the diagnostics in this chapter matter.
A researcher is unsure whether to include a variable. If it is truly irrelevant, what is the cost of including it, and if it is truly relevant, what is the cost of leaving it out?
Show answer
Including an irrelevant variable costs precision only: the coefficients stay unbiased, but the standard errors grow, so estimates are noisier. Leaving out a relevant, correlated variable costs correctness: the included coefficients become biased and inconsistent, so the conclusions themselves are wrong. Because bias is the more serious harm, the two are not symmetric; when in doubt and the sample is large, including the variable is usually the safer error.
A residual plot of a fitted model shows the residuals fanning out into a widening cone as the fitted values increase. What problem is this, does it bias the coefficients, and what is the simplest fix?
Show answer
The funnel shape is heteroskedasticity: the error variance grows with the fitted value. It does not bias the coefficients, which remain unbiased and consistent; it only invalidates the ordinary standard errors. The simplest fix is to keep the OLS coefficients and report heteroskedasticity-robust (White) standard errors, so that the t-tests and confidence intervals are valid again.
In a regression, the overall F-test strongly rejects that all slopes are zero, yet none of the individual t-statistics is significant. What is likely going on, and how would you confirm it?
Show answer
This is the classic signature of multicollinearity. The variables jointly explain the dependent variable, which the F-test detects, but they are so correlated that no single coefficient can be pinned down, so the t-tests all fall short. Confirm it with variance inflation factors: regress each variable on the others and compute one over one minus that R-squared. A VIF above 10 for one or more variables points to serious multicollinearity as the cause.
An observation has a Cook’s distance of 1.4. What does that mean, and should the observation simply be deleted?
Show answer
A Cook’s distance above 1 marks the observation as influential: dropping it would move the fitted model substantially, so this single point has an outsized effect on the coefficients. But a high Cook’s distance is a flag to investigate, not a license to delete. The point may be a genuine extreme event that the model should reflect, especially in risk work, where the tail is the whole point. Check whether it is a data error or a real observation, then decide.
Chapter Summary
- Omitting a relevant, correlated variable biases the included coefficients (bias = β2 × δ); including an irrelevant variable leaves them unbiased but inflates their standard errors.
- The bias-variance tradeoff balances a bigger model’s lower bias against its higher variance; total expected error is squared bias plus variance, minimized in between.
- General-to-specific selection prunes the least significant variables; cross-validation selects the model with the lowest out-of-sample error. Both manage the tradeoff.
- Heteroskedasticity does not bias OLS but invalidates its standard errors. White’s test (n times the auxiliary R², chi-squared) detects it.
- Heteroskedastic data are handled with robust standard errors, a data transformation, or weighted least squares.
- Multicollinearity is high but not perfect correlation among regressors; it inflates standard errors (jointly significant, individually not) and is measured by the VIF (above 10 is serious). Perfect collinearity, by contrast, prevents estimation entirely.
- Residual plots reveal heteroskedasticity (funnel), missing nonlinearity (curve), and outliers; standardized residuals sharpen the reading. Cook’s distance above 1 flags an influential outlier to investigate, not automatically delete.
- Under the standard assumptions, OLS is BLUE (Gauss-Markov); adding normal errors makes it best among all unbiased estimators, not just linear ones.
Frequently Asked Questions
What happens if you omit a relevant variable versus include an irrelevant one?
Omitting a relevant variable that is correlated with the included variables biases the coefficients on those included variables, because they absorb the effect of the missing variable, and the estimates become inconsistent. Including an irrelevant variable, by contrast, does not bias the coefficients, since its own coefficient simply converges to zero, but it inflates the standard errors and makes the other estimates less precise. So the two errors trade off: omission causes bias, inclusion causes imprecision, which is the essence of the bias-variance tradeoff.
What is the bias-variance tradeoff in model selection?
The bias-variance tradeoff is the tension between two sources of error in an estimate. A larger model with more variables tends to have less bias, because it is less likely to omit something relevant, but higher variance, because each coefficient is estimated less precisely. A smaller model has more bias but lower variance. Total expected squared error is the sum of squared bias and variance, so the best model is not the biggest or the smallest but the one that balances the two, minimizing their combined contribution to error.
What are general-to-specific and cross-validation model selection?
General-to-specific selection starts from a large model with all candidate variables and repeatedly removes the least significant variable, the one with the smallest absolute t-statistic, re-estimating each time until every remaining variable is significant at a strict level such as 1 percent. Cross-validation splits the data into blocks, estimates the model on all but one block and measures its error on the held-out block, rotating through the blocks, and picks the model with the smallest out-of-sample error. Both are ways of navigating the bias-variance tradeoff: the first by pruning insignificant variables, the second by rewarding out-of-sample predictive accuracy.
What is heteroskedasticity, how do you test for it, and how do you handle it?
Heteroskedasticity means the variance of the regression error is not constant but changes with the explanatory variables, a common feature of financial data. It does not bias the OLS coefficients, but it makes the usual standard errors wrong, so hypothesis tests become invalid. It is tested with White’s test, which regresses the squared residuals on the explanatory variables, their squares, and cross-products, and uses the resulting fit as a chi-squared statistic. It is handled by using heteroskedasticity-robust (White) standard errors, by transforming the data, or by weighted least squares.
What is multicollinearity, and how does it differ from perfect collinearity?
Multicollinearity is high but not perfect correlation among the explanatory variables, so one variable is largely, but not exactly, explained by the others. It does not stop estimation, but it inflates the standard errors, so coefficients can be jointly significant on an F-test yet individually insignificant on t-tests. It is measured with the variance inflation factor; a value above 10 is a common warning threshold. Perfect collinearity is the extreme case where a variable is an exact linear combination of the others, which makes estimation impossible rather than merely imprecise.
How do residual plots and outlier measures help diagnose a regression?
A residual plot graphs the residuals against an explanatory variable or the fitted values, and a good model shows a random, patternless band of small residuals. A funnel shape signals heteroskedasticity, a curved pattern signals a missing nonlinear term, and isolated large residuals flag outliers. Standardized residuals make the size of each deviation easier to judge. Outliers are points that change the coefficients substantially when removed, and their influence is measured by Cook’s distance, where a value above 1 marks an observation with an outsized effect on the fitted model.
When is OLS the best linear unbiased estimator?
When the standard regression assumptions hold, mean-zero errors, constant error variance, no correlation across observations, and no perfect collinearity, OLS is the best linear unbiased estimator, meaning it has the smallest variance among all estimators that are both linear in the data and unbiased. This is the Gauss-Markov theorem. If, in addition, the errors are normally distributed, OLS becomes the best among all unbiased estimators, not just the linear ones, so its efficiency claim strengthens further.
Loading comments...
Add your Thoughts: