FRM Part 1 · Quantitative Analysis · Chapter QTA 8
Regression with Multiple Explanatory Variables
Learning Objectives
- Differentiate between the relative assumptions of single and multiple regression.
- Interpret regression coefficients in a multiple regression.
- Interpret goodness-of-fit measures for single and multiple regressions, including R² and adjusted-R².
- Construct, apply, and interpret joint hypothesis tests and confidence intervals for multiple coefficients in a regression.
- Calculate the regression R² using the three components of the decomposed variation of the dependent variable data: the explained sum of squares, the total sum of squares, and the residual sum of squares.
The single-variable regression of the previous chapter answers “how does Y respond to X”. Real problems are rarely that clean. A stock’s return responds to the market, but also to its size and its value characteristics; a firm’s default risk depends on leverage, but also on profitability and size. When several drivers act at once, and those drivers are themselves correlated, a single regression on any one of them mixes up their effects. Multiple regression is the tool that untangles them, letting each coefficient measure the distinct contribution of its own variable while holding the others fixed.
That “holding the others fixed” is the heart of the chapter, and it is what makes multiple regression so much more useful than a bundle of separate single regressions. The rest follows from it: one new assumption, no perfect collinearity; a richer way to measure fit, splitting variation into explained and unexplained pieces and correcting R-squared for the number of variables; and a new kind of test, the joint F-test, for hypotheses about several coefficients at once. The running theme in finance is the multi-factor model, the workhorse for measuring risk exposures and judging manager skill.
Key Takeaways
- A multiple regression, Y = α + β₁X₁ + … + βₖXₖ + ε, explains one variable with several, each coefficient measuring a distinct effect.
- Each coefficient is the effect of its variable holding all others constant, the ceteris paribus interpretation that a single regression cannot give.
- Multiple regression adds one assumption to the single-variable list: no perfect collinearity among the explanatory variables.
- Variation decomposes as TSS = ESS + RSS, and R² = ESS/TSS = 1 − RSS/TSS.
- R² never falls when a variable is added; adjusted R² penalizes extra variables, so it can fall and is the fairer measure across models.
- A t-test checks one coefficient; an F-test checks several jointly, because the estimates are correlated and cannot be tested one at a time.
The Multiple Regression Model
The multiple regression model simply adds more explanatory variables to the line, each with its own coefficient, and keeps a single intercept and a single shock.
where there are k explanatory variables, each with its own slope coefficient βj, a single intercept α, and a mean-zero shock ε. The model is still estimated by ordinary least squares, minimizing the sum of squared residuals, and it remains linear in the coefficients.
Everything about the fitting is the same in spirit as before, though the closed-form solution now needs matrix algebra rather than a single ratio. What changes is not the mechanics but the meaning of the coefficients, and that shift in meaning is the reason to reach for multiple regression in the first place.
Interpreting Coefficients: Holding Others Constant
Here is the single most important idea in the chapter. In a multiple regression, a coefficient measures the effect of its own variable holding all the other explanatory variables constant. This is called the ceteris paribus interpretation, Latin for “all else equal”. The coefficient on the size factor, for instance, is the effect of size on the return after the market and value effects have been stripped out, its partial or distinct contribution.
This is precisely what a single regression cannot deliver. Regress a return on the size factor alone, and the coefficient absorbs not just size but whatever else moves with size, including any market exposure that size happens to be correlated with. Put both in the same regression, and each coefficient is purged of the other’s influence. The value of multiple regression is exactly this ability to isolate one effect from a tangle of correlated drivers.
There is a clean intuition for how the model achieves this. To find the coefficient on one variable while holding the others fixed, imagine first stripping out of both that variable and the dependent variable everything the other variables can explain, keeping only the residual, purified parts, and then relating those two residuals. The coefficient is the slope of that final relationship, built only from the piece of the variable that the others cannot account for. This is why a variable with no independent variation, the perfectly collinear case, has no coefficient to estimate: once the others are partialled out, nothing of it is left.
A fund’s monthly excess return is regressed on three factors, the market, a size factor, and a value factor, giving: return = 0.1% + 1.1 × market + 0.4 × size − 0.3 × value. Interpret each coefficient.
- The intercept, 0.1%, is the fund’s alpha: the monthly return earned when all three factor returns are zero, the part attributable to skill rather than factor exposure.
- The market coefficient, 1.1, says the fund rises 1.1% for each 1% market move, holding size and value fixed. It is slightly more sensitive to the market than average.
- The size coefficient, 0.4, is a positive tilt toward small stocks, holding market and value fixed.
- The value coefficient, −0.3, is a tilt toward growth (the opposite of value), holding market and size fixed.
Answer: the fund is a slightly high-beta, small-cap, growth-leaning fund with a small positive alpha. Each tilt is measured net of the others, so the size tilt is a genuine small-cap exposure, not an artifact of the fund’s market or value positioning.
Assumptions: What Changes With More Regressors
Multiple regression keeps every assumption from the single-variable model, mean-zero shocks, iid data, homoskedasticity, no large outliers, and adds exactly one: no perfect collinearity. Perfect collinearity means one explanatory variable is an exact linear combination of the others, so it carries no independent variation for the model to work with. When that happens, there is no unique way to divide the explained variation among the collinear variables, and the coefficients cannot be estimated at all.
| Assumption | Single regression | Multiple regression |
|---|---|---|
| Variance in each regressor | Var(X) > 0 | Each variable varies |
| Mean-zero shocks | Required | Required (given all X) |
| iid data | Required | Required |
| Homoskedasticity | Required | Required |
| No large outliers | Required | Required |
| No perfect collinearity | — (only one variable) | New: no variable is an exact combination of the others |
The classic traps are mechanical: including a variable and a simple multiple of it, or including a complete set of dummy variables alongside an intercept, which is the dummy variable trap. Perfect collinearity is easy to spot and easy to fix, drop the redundant variable, and statistical software flags it automatically. The subtler cousin, high but not perfect correlation among regressors, does not stop estimation but inflates the standard errors, making individual coefficients hard to pin down even when the model as a whole fits well.
Control Variables and Omitted Variable Bias
A major reason to add variables is to serve as controls. A control is an explanatory variable included not because it is the focus, but because it is known to affect the dependent variable and would otherwise contaminate the coefficient of interest. Leaving out a relevant, correlated variable causes omitted variable bias: the missing variable’s influence gets loaded onto the coefficients of the included variables that move with it, pulling them away from their true values.
To trust that a new variable is finding something real, control for what is already known to matter. When testing whether a novel signal predicts returns, researchers routinely include the market, size, and value factors as controls, so that any explanatory power the signal shows is genuinely new and not just a repackaging of a known factor it happens to correlate with. A coefficient is only as clean as the set of controls surrounding it.
Decomposing Variation: TSS, ESS, RSS
To measure how well a model fits, its variation is split into a part the model explains and a part it does not. The total variation in the dependent variable is the total sum of squares (TSS), the summed squared deviations of the observations from their mean. Least squares splits this exactly into the explained sum of squares (ESS), the variation of the fitted values around the mean, and the residual sum of squares (RSS), the leftover variation in the residuals.
where TSS = Σ(Yi − Ȳ)² is the total variation, ESS = Σ(Ŷi − Ȳ)² the explained variation of the fitted values, and RSS = Σε̂i² the residual variation. The clean split holds because the fitted values and the residuals are uncorrelated by construction.
| Component | Definition | Captures |
|---|---|---|
| Total (TSS) | Σ(Yi − Ȳ)² | All variation in the dependent variable |
| Explained (ESS) | Σ(Ŷi − Ȳ)² | Variation the model accounts for |
| Residual (RSS) | Σε̂i² | Variation left unexplained |
R-Squared and Its Limitations
From the decomposition, R-squared is the explained share of the total variation, equivalently one minus the residual share. It runs from 0, the model explains nothing, to 1, the model explains everything. Computing it from the three sums of squares is a required exam skill.
where a higher R² means the model captures more of the movement in the dependent variable. Because OLS minimizes RSS, it simultaneously maximizes R² for a given set of variables.
A model reports a total sum of squares of 200 and a residual sum of squares of 50. Find the explained sum of squares and the R².
Step 1. The explained sum of squares is the total minus the residual.
Step 2. R² is the explained share, or one minus the residual share.
Answer: an R² of 0.75, so the model explains 75% of the variation in the dependent variable and leaves 25% in the residuals. Both routes, ESS over TSS and one minus RSS over TSS, give the same number, which is a quick internal check.
R-squared has three limitations worth naming, because each is a favorite exam trap. First, it never falls when a variable is added, even a completely irrelevant one, since any extra variable can only shrink the residuals. Second, it cannot be compared across models with different dependent variables, for example a model of returns against a model of log returns. Third, there is no universally good value: an R-squared of 5% might be impressive for predicting daily returns but dismal for a diversified equity portfolio’s relationship to the market.
Adjusted R-Squared
The first limitation, that R-squared always rises with more variables, is fixed by the adjusted R-squared. It penalizes each added variable for the degree of freedom it consumes, so it rewards a variable only if the variable improves the fit by more than the penalty. Unlike plain R-squared, it can fall when a useless variable is added, and it can even go negative for a truly poor model.
where n is the number of observations and k the number of explanatory variables (not counting the intercept). The factor (n − 1)/(n − k − 1) grows as k grows, which is the penalty for adding variables.
The model from Worked Example 2 (R² = 0.75) uses 3 explanatory variables and 100 observations. Compute the adjusted R².
Step 1. Apply the penalty factor with n = 100 and k = 3.
Answer: an adjusted R² of about 0.742, a touch below the plain R² of 0.75. With a large sample the penalty is mild. If a fourth variable were added that barely moved the RSS, the plain R² would still tick up while the adjusted R² would fall, correctly flagging the variable as not worth its place.
Testing a Single Coefficient
Testing one coefficient in a multiple regression works exactly as in the single-variable case. The t-statistic is the estimated coefficient minus its null value, divided by its standard error, and it is compared with the usual critical values. Software reports the standard error, the t-statistic against a null of zero, the p-value, and the confidence interval for every coefficient, and the three agree on significance just as before.
where β̂j is the estimated coefficient on variable j and βj0 its null value, usually 0. The one subtlety is that the standard errors now account for the correlations among the regressors, which is why highly correlated variables have large standard errors.
Joint Tests: the F-Test
The t-test has a blind spot: it checks coefficients one at a time, but many questions are about several coefficients together. Do the size and value factors jointly add anything? Are all the slopes zero, meaning the model explains nothing? These are joint hypotheses, and they cannot be answered by a string of separate t-tests, because the coefficient estimates are correlated and testing them one by one distorts the overall error rate. The F-test handles them.
The F-test works by comparing two models: the full, unrestricted model, and a restricted model that imposes the null by dropping the coefficients being tested. If dropping them barely worsens the fit, the residual sum of squares hardly rises, and the null that they add nothing is not rejected. If the fit deteriorates sharply, the restricted variables clearly mattered, and the null is rejected. The statistic scales the increase in RSS by the number of restrictions and the remaining error.
where q is the number of coefficients being tested jointly (the number of restrictions). A large F means the restricted variables improved the fit enough to matter, so the null that they are all zero is rejected.
One special case is reported by every statistics package: the overall F-test, whose null is that all the slope coefficients are zero together, meaning the whole model has no explanatory power beyond the intercept. Rejecting it says the model as a whole explains something, even if no individual coefficient stands out. It is possible, and instructive, for the overall F-test to reject while several individual t-tests do not, a signature of correlated regressors sharing the credit among themselves; the model clearly works, yet the work cannot be cleanly assigned to any single variable.
In the three-factor fund regression, the size and value coefficients each have a t-statistic of about 1.5, so neither is significant on its own at 5%. A joint F-test of the null that both are zero, however, is rejected. What does this mean?
Answer: individually, neither factor clears the bar, but together they explain a meaningful share of the return that the market alone cannot. This pattern is common when two variables are correlated: each steals significance from the other in the separate t-tests, yet as a pair they are clearly doing work. It is exactly why a joint F-test is needed, dropping both factors would worsen the fit more than chance allows, so the tilt toward size and value is real even though no single t-test proves it.
Application: Multi-Factor Models
The signature application is the multi-factor model, which regresses a portfolio’s return on several risk factors at once. The best-known example adds a size factor and a value factor to the market, extending the single-factor CAPM into a three-factor description of returns. Each factor coefficient is the portfolio’s distinct exposure to that source of risk, held apart from the others, and the intercept is the alpha that survives all of them, a much sterner test of skill than the CAPM’s single-factor alpha.
Exhibit 3 shows a typical output. Read it exactly as any regression: each slope is an exposure held constant against the rest, the intercept is the unexplained excess, and the R-squared says how much of the portfolio the factors together capture, which is how the appropriateness of the factor set is judged.
| Term | Estimate | t-stat | Reads as |
|---|---|---|---|
| Alpha (α) | 0.10% | 0.9 | Excess not explained by the factors (skill) |
| Market (βm) | 1.10 | 18.5 | Sensitivity to the market, all else equal |
| Size (βs) | 0.40 | 4.2 | Tilt toward small stocks |
| Value (βv) | −0.30 | −3.1 | Tilt toward growth stocks |
| R² | 0.88 | — | Factors explain 88% of the fund’s variation |
Figure 3 makes the point visually, splitting the fund’s return variation into the share each factor explains and the part left unexplained, which is one minus the R-squared.
The move from one factor to three is the move from CAPM to a real risk model. A fund can post a handsome CAPM alpha simply by tilting toward small or value stocks; add those factors as controls and the tilt is reclassified as factor exposure, not skill. Whatever alpha survives the full factor set is the genuinely unexplained excess. This is why performance is judged against multi-factor benchmarks, and why omitting a relevant factor flatters a manager who was merely harvesting a known risk premium.
In a regression of a firm’s default probability on leverage and profitability, the coefficient on leverage is 0.02. What exactly does that coefficient mean?
Show answer
It means that a one-unit increase in leverage is associated with a 0.02 increase in the default probability, holding profitability constant. The multiple-regression coefficient isolates the effect of leverage from profitability, so it is the distinct contribution of leverage among firms with the same profitability. A single regression of default on leverage alone would blend in any effect of profitability that happens to move with leverage.
A model has a total sum of squares of 500 and an explained sum of squares of 300. What are the residual sum of squares and the R²?
Show answer
The residual sum of squares is the total minus the explained: RSS = 500 − 300 = 200. The R² is the explained share: 300 / 500 = 0.60, equivalently 1 − 200 / 500 = 0.60. So the model explains 60% of the variation in the dependent variable, leaving 40% unexplained.
Adding a fourth explanatory variable raises R² from 0.75 to 0.752 but lowers the adjusted R² from 0.742 to 0.738. Should the variable be kept?
Show answer
Probably not. The plain R² rose only because R² can never fall when a variable is added, so its tiny increase is uninformative. The adjusted R² fell, which means the new variable did not improve the fit by enough to justify the degree of freedom it consumed. When the adjusted R² declines on adding a variable, that variable is not earning its place and is best dropped.
You want to test whether three seasonal dummy variables, taken together, have any effect on a return. Why not just run three t-tests?
Show answer
Because the question is joint, and the three coefficient estimates are correlated. Three separate t-tests each at 5% inflate the overall chance of a false positive and, more importantly, can miss a joint effect that no single coefficient shows significantly on its own. The F-test is designed for exactly this: it compares the model with all three dummies against the model without them, and rejects the null that all three are zero if dropping them worsens the fit by more than chance would allow.
Chapter Summary
- Multiple regression, Y = α + β₁X₁ + … + βₖXₖ + ε, explains one variable with several and is still fit by OLS.
- Each coefficient is the effect of its variable holding all others constant, the ceteris paribus interpretation a single regression cannot provide.
- The only new assumption is no perfect collinearity: no explanatory variable may be an exact linear combination of the others (the dummy variable trap is the classic case).
- Controls guard against omitted variable bias, in which a missing correlated factor distorts the coefficients of the included variables.
- Variation decomposes as TSS = ESS + RSS, and R² = ESS/TSS = 1 − RSS/TSS.
- R² never falls when a variable is added, cannot be compared across different dependent variables, and has no universal “good” value; adjusted R² penalizes extra variables and can fall or go negative.
- A single coefficient is tested with a t-test; several coefficients jointly are tested with an F-test, which compares the fit of the full and restricted models.
- Multi-factor models regress a return on several risk factors at once; each slope is a distinct exposure and the surviving intercept is a sterner measure of alpha than the CAPM’s.
Frequently Asked Questions
How do the assumptions of multiple regression differ from single regression?
Multiple regression carries over all the assumptions of single regression, mean-zero errors, iid data, constant error variance, and no large outliers, and adds one more: no perfect collinearity. Perfect collinearity means one explanatory variable is an exact linear combination of the others, which leaves no distinct variation for the model to attribute to it, so its coefficient cannot be estimated. In practice this requires each explanatory variable to have some movement that the other variables cannot explain.
How do you interpret a coefficient in a multiple regression?
A coefficient in a multiple regression is the estimated change in the dependent variable for a one-unit increase in that explanatory variable, holding all the other explanatory variables constant. This holding-others-constant, or ceteris paribus, interpretation is the whole point of multiple regression: it isolates the distinct contribution of each variable, stripping out the influence of the others. It differs from a single regression coefficient, which mixes in the effects of any correlated variables that were left out of the model.
What is perfect collinearity and why is it a problem?
Perfect collinearity occurs when one explanatory variable can be written exactly as a linear combination of the others, for example including a variable and twice that same variable, or a full set of dummy variables plus an intercept. When this happens there is no unique way to split the explained variation among the collinear variables, so the individual coefficients are not identified and standard software returns an error. It is easy to detect and fix by dropping the redundant variable.
What is the difference between R-squared and adjusted R-squared?
R-squared is the fraction of the variation in the dependent variable explained by the model, and it never falls when a new explanatory variable is added, even a useless one, because adding a variable can only reduce the residual variation. Adjusted R-squared corrects for this by penalizing each added variable for the degree of freedom it consumes. It rises only when a new variable improves the fit by more than the penalty, so it can fall when an irrelevant variable is added, which makes it the better measure for comparing models with different numbers of variables. Adjusted R-squared can even be negative for a very poor fit.
What are the total, explained, and residual sums of squares?
The total sum of squares, TSS, is the total variation in the dependent variable, the sum of squared deviations of each observation from the sample mean. The explained sum of squares, ESS, is the variation the model captures, the sum of squared deviations of the fitted values from the mean. The residual sum of squares, RSS, is the variation left over, the sum of squared residuals. They satisfy TSS equals ESS plus RSS, so the total variation splits cleanly into an explained part and an unexplained part, and R-squared is the explained part over the total, or equivalently one minus the residual part over the total.
When do you use an F-test instead of a t-test in a regression?
A t-test checks one coefficient at a time. An F-test checks several coefficients jointly, such as whether all the slopes are zero together, or whether two particular factors add explanatory power as a pair. The joint test is needed because the coefficient estimates are correlated, so testing them one by one with separate t-tests does not correctly control the overall error rate. The F-test compares the fit of the full model against a restricted model that imposes the null, and rejects when dropping the restricted variables worsens the fit by more than chance would explain.
What is a multi-factor model, and what is omitted variable bias?
A multi-factor model regresses a portfolio’s return on several risk factors at once, such as the market, a size factor, and a value factor, so each factor’s coefficient measures the portfolio’s distinct exposure to that source of risk while holding the others constant. Omitted variable bias arises when a relevant factor is left out of the model: its influence gets absorbed into the coefficients of the included variables that are correlated with it, distorting them. Including the right control variables, such as the standard risk factors, is how that bias is avoided.
Loading comments...
Add your Thoughts: