CFA Level 1 · Module 02 Quantitative Methods · Chapter 7
An analyst almost never has all the data. You cannot observe every return an asset will ever produce, only a sample of the returns it has produced so far. Estimation is the art of saying something trustworthy about the whole from a part, and hypothesis testing is the discipline of deciding whether a pattern in the sample is real or just noise. Together they turn a limited set of observations into a defensible conclusion, which is the entire point of statistics in investing.
The reading moves in two halves. The first builds estimation: the central limit theorem, which makes sample means well behaved, the standard error that measures their precision, and the confidence interval that expresses the uncertainty honestly. The second builds hypothesis testing: the null and alternative, the test statistic, the two kinds of error you can make, and the p-value that has become the standard way to report a result. A short final section contrasts parametric tests with their non-parametric cousins.
None of the calculations are heavy, but the concepts are precise, and the exam is unforgiving about the vocabulary. Knowing exactly what a confidence interval does and does not claim, and exactly what rejecting a null hypothesis means, is worth more here than raw computation.
A population is the entire set of items you care about, such as every monthly return a strategy could generate. A sample is the subset you actually observe. A number computed from the population, like its true mean, is a parameter; the same number computed from a sample is a statistic, and it serves as an estimate of the parameter. The whole enterprise rests on one uncomfortable fact: the sample statistic will almost never equal the population parameter exactly, so every estimate carries uncertainty that must be quantified rather than ignored.
A single number offered as an estimate is a point estimate: the sample mean is a point estimate of the population mean. It is the best single guess, but on its own it is silent about how far off it might be. That silence is dangerous, because a point estimate from ten observations and one from ten thousand look identical on the page while deserving very different confidence. The rest of estimation exists to attach a measure of precision to the point estimate, so that the reader knows how much to trust it.
A good estimator has three properties worth naming. It is unbiased if its average across all possible samples equals the true parameter, so it does not systematically overshoot or undershoot. It is efficient if it has the smallest variance among unbiased estimators, so it wastes the least information. And it is consistent if it converges on the true parameter as the sample grows without bound. The sample mean is unbiased, efficient, and consistent for the population mean, which is a large part of why it is the estimator the whole reading is built around.
The central limit theorem is the result that makes estimation work. It says that when you draw a large enough sample, the distribution of the sample mean is approximately normal, regardless of the shape of the underlying population. Returns can be skewed, fat-tailed, or lumpy, and it does not matter: the average of a large sample of them still follows a normal distribution centered on the true population mean. As a rule of thumb, a sample of thirty or more is usually enough for the approximation to hold.
This is a remarkable piece of good fortune. It means an analyst can use the clean, well-understood normal distribution to reason about a sample mean even when the individual returns are anything but normal. Every confidence interval and every hypothesis test about a mean in this reading leans on the central limit theorem, because it is what guarantees the sample mean behaves predictably. Without it, each messy population would need its own bespoke analysis; with it, one toolkit serves them all.
The distribution of the sample mean, its behavior across all the samples you could have drawn, is called the sampling distribution. Its center is the population mean, which is why the sample mean is an unbiased estimator, and its spread is measured by the standard error, the subject of the next section. Understanding that a statistic has its own distribution, distinct from the distribution of the raw data, is the conceptual leap the whole reading turns on.
The standard error of the mean measures how much a sample mean would vary from sample to sample. It is the standard deviation of the sampling distribution, and it shrinks as the sample grows, because a larger sample pins down the mean more tightly. This is the precise sense in which more data is better: precision improves with the square root of the sample size, so quadrupling the sample halves the standard error.
where SE is the standard error of the mean, s is the sample standard deviation, and n is the sample size. When the population standard deviation is known it replaces s, but in practice the sample value is almost always used.
Setup. An analyst has 36 monthly returns for a strategy, with a sample standard deviation of 6%. Find the standard error of the mean.
Answer: a standard error of 1%. Notice the difference between this and the 6% standard deviation of the individual returns. The returns are volatile, but their average over 36 months is estimated far more precisely, and the standard error is the number that captures that precision.
Do not confuse the standard deviation with the standard error. The standard deviation measures the spread of individual observations; the standard error measures the spread of the sample mean, and it is smaller by a factor of the square root of the sample size. Confidence intervals and test statistics use the standard error, not the raw standard deviation.
A confidence interval turns a point estimate into a range that is likely to contain the true parameter. It is built by taking the point estimate and adding and subtracting a margin equal to a reliability factor times the standard error. The reliability factor comes from the sampling distribution and depends on how much confidence you want: a 95% interval uses a larger factor than a 90% interval, because covering the truth more often requires casting a wider net.
where x̄ is the sample mean, SE is the standard error, and the reliability factor is drawn from the normal or t-distribution for the chosen confidence level. For a 95% interval using the normal distribution, the factor is 1.96; for 90% it is 1.65; for 99% it is 2.58.
Setup. The 36 monthly returns average 1.2% with a standard error of 1%. Construct a 95% confidence interval for the true mean monthly return, using the normal distribution.
Answer: the 95% confidence interval runs from about negative 0.76% to positive 3.16%. Because the interval includes zero, the data cannot rule out that the true mean return is zero, an observation that connects directly to hypothesis testing later. A wider confidence level would widen the interval further.
Reading a 95% confidence interval as “a 95% chance the true mean is in this range.” The parameter is fixed, not random; it is the interval that varies from sample to sample. The correct reading is that if the sampling were repeated many times, about 95% of the intervals so constructed would contain the true mean. The distinction is subtle but the exam tests it.
| Confidence level | Significance in each tail | Reliability factor (z) |
|---|---|---|
| 90% | 5% | 1.65 |
| 95% | 2.5% | 1.96 |
| 99% | 0.5% | 2.58 |
A confidence interval trades width for confidence. Demanding more confidence, say 99% instead of 95%, forces a larger reliability factor and a wider interval. There is no way to be both more certain and more precise from the same data; the only lever that improves both at once is a larger sample, which shrinks the standard error.
The reliability factor comes from one of two distributions, and choosing correctly is a common exam point. Use the normal distribution, giving z-factors like 1.96, when the population standard deviation is known or the sample is large. Use the t-distribution, met in the previous reading, when the population standard deviation is unknown and the sample is small. The t-distribution has fatter tails, so its reliability factors are larger, producing a wider interval that honestly reflects the extra uncertainty of estimating the spread from a small sample.
The t-distribution’s factor depends on the degrees of freedom, which for a single mean is the sample size minus one. As the sample grows, the t-factors shrink toward the z-factors, so for large samples the choice barely matters and the normal is used for convenience. For small samples the difference is real, and defaulting to the z-factor understates the interval.
Setup. A different strategy has only 16 monthly returns, averaging 5% with a sample standard deviation of 8%. The population standard deviation is unknown. Build a 95% confidence interval.
Answer: a 95% interval from about 0.74% to 9.26%. Had the analyst wrongly used the z-factor of 1.96, the interval would have been too narrow at 5% ± 3.92%, overstating the precision. With a small sample and an unknown population spread, the t-distribution is the honest choice.
An estimate is only as good as the sample behind it, and how the sample is drawn matters enormously. In simple random sampling every member of the population has an equal chance of selection, which keeps the sample representative. Stratified sampling first divides the population into groups, such as sectors, then samples within each in proportion, which can improve precision when the groups differ. The goal in every method is a sample that mirrors the population rather than a distorted slice of it.
Several biases corrupt this goal, and recognizing them is heavily tested. Sampling bias arises when the sample is not representative of the population. Survivorship bias is a common special case: studying only the funds or firms that still exist ignores those that failed, which flatters the measured returns because the losers have quietly dropped out. Look-ahead bias uses information in a test that would not have been available at the time, and time-period bias draws conclusions from a window too short or too unusual to generalize. Each of these can make a perfectly correct calculation produce a badly wrong conclusion.
| Bias | What goes wrong |
|---|---|
| Sampling bias | The sample is not representative of the population |
| Survivorship bias | Failed funds or firms are excluded, inflating measured returns |
| Look-ahead bias | The test uses data not available at the time |
| Time-period bias | The window is too short or too unusual to generalize |
Trusting a backtest of “the funds available today.” That set has already excluded every fund that closed after poor performance, so its average return is inflated by survivorship bias. A sound study must include the funds that no longer exist, or its estimate of the typical outcome is systematically too high.
Hypothesis testing is a structured way to decide whether the data support a claim. It begins with two competing statements. The null hypothesis is the default position, usually one of no effect or no difference, such as a mean return equal to zero. The alternative hypothesis is what you would conclude if the data were convincing enough to overturn the default. The test never proves the alternative; it only asks whether the evidence against the null is strong enough to reject it.
A test is one-tailed if the alternative points in a specific direction, such as a mean greater than zero, and two-tailed if it merely claims a difference in either direction, such as a mean not equal to zero. The choice is set by the question before any data is examined, never adjusted afterward to make a result look better. A one-tailed test concentrates all its skepticism on one side, while a two-tailed test splits it between both, which affects the critical value the test statistic must clear.
A hypothesis test can reject the null or fail to reject it, but it never accepts the null and never proves the alternative. Failing to reject means only that the evidence was not strong enough to overturn the default, not that the default is true. This asymmetry is deliberate: the burden of proof sits entirely on the alternative.
The test statistic measures how far the sample result sits from what the null hypothesis predicts, expressed in standard errors. For a test of a single mean with an unknown population standard deviation, it is a t-statistic: the difference between the sample mean and the hypothesized mean, divided by the standard error. A large test statistic means the sample is far from the null’s prediction, which is evidence against the null.
where x̄ is the sample mean, μ0 is the mean claimed by the null hypothesis, s is the sample standard deviation, and n is the sample size, so that the denominator is the standard error.
The decision rule compares the test statistic against a critical value set by the chosen significance level, written as alpha, the probability of wrongly rejecting a true null. A 5% significance level is the common default. If the test statistic exceeds the critical value, the result is statistically significant and the null is rejected; if not, the null stands. Significance is a statement about the strength of the evidence, not about the economic size or importance of the effect, a distinction worth keeping firmly in mind.
The critical value carves the distribution into a rejection region and a non-rejection region. In a two-tailed test at the 5% level the rejection region sits in both tails, each holding 2.5%, so the test statistic must clear roughly 1.96 in absolute value. In a one-tailed test the whole 5% sits in a single tail, so the hurdle is lower, about 1.65 for the normal, and the direction of the alternative decides which tail. This is why a one-tailed test is easier to pass in its chosen direction, and why the direction must be fixed by the question rather than chosen after seeing which way the data went.
Setup. An analyst tests whether a strategy’s mean monthly return exceeds zero. From 36 months, the mean is 2.0% and the standard error is 1%. Test at the 5% significance level with a one-tailed alternative that the mean is greater than zero.
Answer: reject the null; the mean return is significantly greater than zero at the 5% level. Note how much rode on the standard error: with the same 2% mean but a larger standard error, the statistic would have fallen short. Significance depends on the size of the effect relative to its precision, not on the effect alone.
Because a test decides from limited data, it can be wrong in two distinct ways. A Type I error is rejecting a null hypothesis that is actually true, a false alarm, and its probability is exactly the significance level alpha. A Type II error is failing to reject a null hypothesis that is actually false, a missed detection, and its probability is written as beta. The two errors trade off: lowering the significance level to guard against false alarms makes the test more likely to miss a real effect.
The power of a test is its ability to detect a real effect, equal to one minus beta, the probability of correctly rejecting a false null. Power rises with the sample size, which is the cleanest way to reduce both error types at once: more data lets a test be both cautious about false alarms and sensitive to real effects. When a study fails to find significance, the right question is often whether it had enough power to find the effect it was looking for.
| Reality | Test rejects the null | Test fails to reject |
|---|---|---|
| Null is true | Type I error (probability alpha) | Correct decision |
| Null is false | Correct decision (power, 1 minus beta) | Type II error (probability beta) |
Fix the definitions cold. The significance level alpha is the probability of a Type I error, chosen by the analyst before the test. Power is one minus the probability of a Type II error. A frequent trap swaps the two errors or claims that lowering alpha has no cost, when in fact it raises the chance of a Type II error unless the sample size also grows.
Modern practice reports a p-value rather than a bare reject-or-not verdict. The p-value is the probability, if the null hypothesis were true, of observing a test statistic at least as extreme as the one obtained. A small p-value means the observed data would be very surprising under the null, which is evidence against it. The decision rule becomes simple: reject the null when the p-value is below the chosen significance level, and fail to reject otherwise.
The p-value is more informative than a yes-or-no result because it reports how strong the evidence is, not merely whether it cleared a threshold. A p-value of 0.001 and a p-value of 0.049 both reject at the 5% level, but the first is far stronger evidence. What the p-value does not do is measure the size or importance of an effect, nor the probability that the null is true; it is purely a statement about how compatible the data are with the null.
Setup. A one-tailed test produces a t-statistic of 2.0, corresponding to a p-value of about 0.027. Interpret the result at the 5% and 1% significance levels.
Answer: significant at 5% but not at 1%. The same p-value can lead to different conclusions depending on the significance level chosen, which is why the level must be set in advance and reported alongside the result.
Do not confuse statistical significance with economic significance. A strategy can beat zero by a statistically significant but tiny margin that, after trading costs and taxes, is not worth pursuing. The p-value answers whether an effect is real; whether it is large enough to matter is a separate judgment the test cannot make for you.
A parametric test assumes the data follow a particular distribution, usually the normal, and tests statements about its parameters, such as a mean or a variance. The t-test used above is parametric. A non-parametric test makes far weaker assumptions about the distribution and is used when those assumptions fail, when the data are ranks or categories rather than measured quantities, or when the sample is too small to trust the normal approximation. The trade-off is that non-parametric tests, by using less structure, are generally less powerful when the parametric assumptions actually do hold.
The practical rule is to use a parametric test when its assumptions are reasonable, because it extracts more information from the data, and to switch to a non-parametric test when the data are not numeric, are badly non-normal, or come in ranks. Tests based on the ranks of observations, and tests of association in categorical data, are the common non-parametric tools, and the next section shows one of them.
A short checklist decides the choice in practice. Ask first what the data look like: measured quantities that are roughly normal point to a parametric test, while ranks, categories, or badly skewed small samples point to a non-parametric one. Ask next what is being tested: a mean or a variance has a natural parametric test, while a question of association between categories, or of whether one group tends to rank above another, is naturally non-parametric. The parametric test is the default when it is defensible, precisely because its extra assumptions buy extra power, and the non-parametric test is the safe fallback when those assumptions cannot be trusted.
A frequent non-parametric question is whether two categorical variables are independent, such as whether a strategy’s up or down days are related to the market’s up or down days. The data are arranged in a contingency table of observed counts, and a chi-square test compares those observed counts with the counts that would be expected if the two variables were truly independent. Large gaps between observed and expected counts produce a large test statistic and evidence of a relationship.
where O is each observed count, E is the count expected under independence, computed as the row total times the column total divided by the grand total, and the sum runs over every cell of the table.
Setup. Over 100 days, a strategy and the market are each classified up or down. Suppose one cell has an observed count of 40 while independence would predict an expected count of 30. What is that cell’s contribution to the chi-square statistic?
Answer: this cell contributes 3.33 to the chi-square total. Summing such contributions across all cells gives the test statistic, which is then compared with a chi-square critical value. A large total signals that the strategy’s direction is not independent of the market’s.
A sample of 64 returns has a standard deviation of 8%. What is the standard error of the mean?
SE = 8% ÷ √64 = 8% ÷ 8 = 1%. The larger sample gives the same standard error as a smaller, less volatile one would, showing how sample size and volatility trade off.
A sample mean is 4% with a standard error of 1.5%. What is the 95% confidence interval using the normal distribution?
4% ± 1.96 × 1.5% = 4% ± 2.94%, giving an interval from about 1.06% to 6.94%.
An analyst rejects a null hypothesis that is in fact true. Which type of error is this, and what is its probability?
A Type I error, a false alarm. Its probability is the significance level alpha chosen for the test, commonly 5%.
A test produces a p-value of 0.08. Do you reject the null at the 5% level, and what does the value say about the evidence?
No. Since 0.08 exceeds 0.05, the null is not rejected at the 5% level. The evidence against the null is weak, though it would be significant at a more lenient 10% level.
Two studies test the same effect, one with 30 observations and one with 300. Which is more likely to detect the effect if it is real, and why?
The study with 300 observations. A larger sample has a smaller standard error and therefore higher power, one minus the probability of a Type II error, so it is more likely to correctly reject a false null and detect a real effect.
Because of the central limit theorem, which says the sample mean of a large enough sample is approximately normal no matter how the underlying data are distributed. It is the mean that becomes normal, not the individual returns, and a sample of about thirty or more is usually sufficient.
Standard deviation measures the spread of individual observations. Standard error measures the spread of the sample mean across samples, and it equals the standard deviation divided by the square root of the sample size, so it is smaller and shrinks as the sample grows.
No. The true mean is fixed; the interval is what varies. The correct reading is that if you repeated the sampling many times, about 95% of the intervals you built would contain the true mean. Any single interval either contains it or does not.
When the population standard deviation is unknown and the sample is small. The t-distribution has fatter tails and larger reliability factors, giving a wider, more honest interval. For large samples the two converge and the normal is used for convenience.
It means the evidence against the null hypothesis is strong enough to reject it at the chosen significance level. It says nothing about how large or economically important the effect is; a tiny effect can be significant with enough data, and a large one can be insignificant without it.
A Type I error is a false alarm with probability alpha; a Type II error is a missed detection with probability beta. Power is one minus beta, the chance of catching a real effect. Lowering alpha raises beta unless the sample grows, which is the cleanest way to improve both.
Because the p-value shows how strong the evidence is, not merely whether it crossed a line. A p-value of 0.001 and one of 0.049 both reject at 5%, but the first is far more convincing. It lets a reader apply their own significance level to your result.
Through calculation of standard errors, confidence intervals, and test statistics, and through precise-definition questions on the two error types, power, significance, and the p-value. Expect to choose between z and t, and to identify a sampling bias from a described study.
Loading comments...
Add your Thoughts: