FRM Part 1 · Quantitative Analysis · Chapter QTA 6
Hypothesis Testing
Learning Objectives
- Construct an appropriate null hypothesis and alternative hypothesis and differentiate between the two.
- Differentiate between a one-sided and a two-sided test and identify when to use each test.
- Explain the difference between Type I and Type II errors and how these relate to the size and power of a test.
- Explain how a hypothesis test and a confidence interval are related.
- Explain what the p-value of a hypothesis test measures.
- Construct and apply confidence intervals for one-sided and two-sided hypothesis tests, and interpret the results of hypothesis tests with a specific confidence level.
- Identify the steps to test a hypothesis about the difference between two population means.
- Explain the problem of multiple testing and how it can lead to biased results.
A risk manager is forever asking questions that the data can only answer with a probability. Is this strategy’s average return really positive, or did it just get lucky? Is this desk riskier than that one? Does this value-at-risk model break too often to trust? Hypothesis testing is the disciplined procedure for turning a noisy sample into a yes-or-no decision about the world, while keeping the chance of being fooled under explicit control.
The whole framework reduces to one question: how likely is the observed data if a particular assumption were true? If the data would be very unlikely under that assumption, the assumption is rejected. The chapter builds the machinery that makes this precise, the hypotheses, the test statistic, the two kinds of error, and shows that a test, a confidence interval, and a p-value are three views of the same decision. It closes with a warning that matters enormously in practice: run enough tests and something will look significant by pure chance.
Key Takeaways
- A test pits a null hypothesis, the assumption of nothing unusual, against an alternative, and asks whether the data are too unlikely under the null to keep believing it.
- A two-sided test looks for a departure in either direction; a one-sided test looks in only one, placing the whole rejection region in a single tail.
- A Type I error rejects a true null (probability alpha, the size); a Type II error keeps a false null (probability beta); power is one minus beta.
- A confidence interval is the set of null values a test would not reject, so the interval and the test always agree.
- A p-value is the probability of data at least as extreme as observed if the null is true; reject when it falls below the size.
- Running many tests inflates the chance of a false positive, which is why data mining produces significant-looking results that do not hold up.
The Six Elements of a Test
A hypothesis test is assembled from six parts. Naming them upfront makes every later section a matter of filling in the blanks. Exhibit 1 lists them.
| Component | What it is |
|---|---|
| Null hypothesis (H₀) | The parameter value being tested, assumed true for the test |
| Alternative hypothesis (H₁) | The values that would count as evidence against the null |
| Test statistic (T) | A summary of the data with a known distribution when the null is true |
| Size (α) | The significance level, the accepted chance of rejecting a true null |
| Critical value | The cutoff the statistic is compared against |
| Decision rule | Reject the null if the statistic is more extreme than the critical value |
Null and Alternative Hypotheses
The null hypothesis, written H-zero, is the claim under examination, and it is usually the dull one: nothing unusual is going on. A fund earns no more than its benchmark; a mean return is zero; a model is correct. It is called the maintained hypothesis because it is assumed true throughout the test, and the entire distribution of the test statistic is built on that assumption. The alternative hypothesis, written H-one, is everything that would count as evidence against the null, the interesting outcome the analyst hopes to detect.
The asymmetry is deliberate and worth internalizing. A test can reject the null when the data are sufficiently at odds with it, or it can fail to reject the null when they are not. It never accepts or proves the null. Failing to reject means only that the evidence was not strong enough, not that the null is true, in the same way that a “not guilty” verdict is not a declaration of innocence.
where μ is the true population mean and μ₀ is the specific value being tested (often 0). This pairing, an equality null against a “not equal” alternative, is the standard two-sided setup.
One-Sided and Two-Sided Tests
The shape of the alternative decides where the evidence has to land. A two-sided test uses a “not equal” alternative, so a departure in either direction, above or below the null value, counts against the null. Its rejection region is split between the two tails. A one-sided test uses a “greater than” or “less than” alternative, so only a departure in one direction counts, and the whole rejection region sits in a single tail.
Choosing between them is about which departures matter. Testing whether a fund’s return simply differs from its benchmark is two-sided. Testing whether it beats the benchmark, or whether a risk model breaks too often, is one-sided. The one-sided test concentrates all of its power in the direction of interest, so it detects a true departure there more readily, but it is deliberately blind to a departure the other way.
Picking a one-sided test after seeing which way the data point, to make a result significant. The direction of the test must be fixed by the question before looking at the data. Choosing one-sided because the sample happened to come out positive quietly doubles the true chance of a false positive, and it is a form of the data-snooping the last section warns about.
The Test Statistic and the Decision Rule
The test statistic converts the estimate into a standardized distance from the null value, measured in standard errors. For a test about a mean, it is the estimated mean minus the null value, divided by the standard error. When the null is true, this statistic follows a known distribution, approximately a standard normal by the Central Limit Theorem, which is what lets a probability be attached to any observed value.
where μ̂ is the sample mean, μ₀ the null value, and the denominator is the standard error. When the variance is estimated and the sample is small (below about 30), the statistic follows a Student’s t with n − 1 degrees of freedom instead, which has slightly fatter tails.
Which distribution to use is a small but testable point. When the variance is known, or the sample is large, the statistic is standard normal and the critical values come from the normal table. When the variance is estimated from a small sample, the extra uncertainty in that estimate fattens the tails, and the correct reference is the Student’s t with n − 1 degrees of freedom, whose critical values are a touch larger. In the large samples typical of financial data the two are almost identical, and practitioners usually default to the normal; the distinction matters mainly for samples of a few dozen observations or fewer.
The critical value is the cutoff that marks off the rejection region, set by the size and the shape of the alternative. The decision rule then compares the statistic with the critical value: reject the null if the statistic is more extreme. Exhibit 2 lists the standard normal critical values, and Figure 2 shows the rejection regions they cut off.
| Size α | Two-sided (each tail α/2) | One-sided (α in one tail) |
|---|---|---|
| 10% | ±1.645 | 1.28 |
| 5% | ±1.96 | 1.645 |
| 1% | ±2.57 | 2.33 |
A quant strategy delivers an average monthly return of 1.2% over 64 months, with a sample standard deviation of 4%. Test, at the 5% two-sided level, whether its true mean return differs from zero.
Step 1. Set the hypotheses. H₀: μ = 0 versus H₁: μ ≠ 0.
Step 2. Compute the standard error, then the test statistic.
Step 3. Compare with the two-sided 5% critical value, 1.96.
Answer: the test statistic 2.4 exceeds 1.96, so the null of a zero mean is rejected at the 5% level. The strategy’s average return is statistically significant, meaning a mean this far from zero would be unlikely if the true mean were really zero.
Type I and Type II Errors, Size and Power
Because a test decides from a sample, it can be wrong in two distinct ways. A Type I error is rejecting a null that is actually true, a false alarm. Its probability is the size of the test, denoted alpha, and it is chosen by the analyst, usually at 5 percent. A Type II error is failing to reject a null that is actually false, a missed detection. Its probability is denoted beta, and one minus beta, the chance of correctly catching a false null, is the power of the test. Exhibit 3 is the decision matrix that every candidate should be able to reproduce.
| Decision | Null is true | Null is false |
|---|---|---|
| Fail to reject | Correct (1 − α) | Type II error (β) |
| Reject | Type I error (α) | Correct: power (1 − β) |
The two errors trade off. At a fixed sample size, shrinking the size to guard against false alarms pushes the critical value further out, which makes it harder to reject anything, so the chance of a missed detection rises. Lowering alpha raises beta. The only way to reduce both at once is to gather more data. Power is not chosen directly; it rises with a larger sample, a larger true effect, and a larger test size, as Figure 3 illustrates with the null and true distributions side by side.
One consequence of the power story deserves emphasis, because it trips up practitioners as often as students. With a large enough sample, the standard error becomes tiny and even a trivially small effect can be statistically significant. A strategy that outperforms by a hundredth of a percent a year could reject the null of zero on a long enough dataset, yet that edge might not cover trading costs. Statistical significance is not the same as economic significance. A test answers whether an effect is distinguishable from zero; it does not answer whether the effect is large enough to matter. Both questions have to be asked, and only the first is settled by a p-value.
Confidence Intervals and the Test
A confidence interval is the same decision seen from a different angle. Rather than testing one null value with a yes-or-no verdict, it reports the whole range of values that the data are consistent with. A 95 percent confidence interval contains exactly the set of null values that a two-sided 5 percent test would fail to reject. So the two always agree: a null value is rejected precisely when it falls outside the interval.
where Cα is the critical value for the chosen size (1.96 for a two-sided 95 percent interval) and SE is the standard error. A one-sided test corresponds to a one-sided interval, open on one end.
Build the 95% confidence interval for the strategy’s mean return from Worked Example 1 (estimate 1.2%, standard error 0.5%), and confirm it agrees with the test.
Step 1. Estimate plus and minus 1.96 standard errors.
Answer: the interval runs from 0.22% to 2.18% and does not contain zero. Because the null value of zero sits outside the interval, the null is rejected, exactly the verdict the test statistic gave. The interval adds information: it says the true mean is plausibly anywhere from about a fifth of a percent to a bit over two percent a month.
The p-Value
The p-value is a third view of the same evidence, and the one most often misread. It is the probability of observing a test statistic at least as extreme as the one actually computed, assuming the null hypothesis is true. A small p-value means the observed data would be surprising under the null, which is evidence against it. The decision rule is simple: reject the null when the p-value is below the size.
where Φ is the standard normal CDF and |T| the absolute test statistic. The factor of 2 accounts for both tails in a two-sided test; a one-sided p-value drops it.
Find the two-sided p-value for the strategy’s test statistic of 2.4, and interpret it.
Step 1. The right-tail probability beyond 2.4 for a standard normal is about 0.0082.
Step 2. Double it for a two-sided test.
Answer: a p-value of about 0.016, well below the 5% size, so the null is rejected, agreeing with the test statistic and the confidence interval. Read it correctly: if the true mean were zero, a result this extreme would occur about 1.6% of the time. It is not the probability that the null is true, and not the probability the result was luck.
Reading a p-value of 0.016 as “there is a 1.6% chance the null is true” or “a 98.4% chance the strategy works.” Both are wrong. The p-value is computed assuming the null is true; it cannot then also be the probability that the null is true. It measures only how surprising the data are under the null, nothing about the probability of the hypotheses themselves.
Testing the Difference Between Two Means
A common question compares two groups: are the mean returns of two desks, two strategies, or two periods different? The setup mirrors the single-mean test, with the null that the two population means are equal, equivalently that their difference is zero. The test statistic is the difference in sample means divided by the standard error of that difference, and the standard error of a difference of two independent estimates combines the two individual standard errors by adding their squares and taking the root.
where μ̂1 and μ̂2 are the two sample means and SE1, SE2 their standard errors. The statistic is then compared with a critical value exactly as before.
Desk A averages 1.5% with a standard error of 0.4%; Desk B averages 0.8% with a standard error of 0.3%. Test, at 5% two-sided, whether their true mean returns differ.
Step 1. Standard error of the difference.
Step 2. Test statistic on the difference of 0.7%.
Answer: the statistic 1.4 is below the critical value of 1.96, so the null of equal means is not rejected. Desk A looks better in the sample, but the 0.7% gap is within the noise: this data cannot establish that the two desks truly differ. A larger sample would shrink the standard errors and might change the verdict.
The Multiple Testing Problem
Everything so far assumed a single test. Run many, and a subtle bias creeps in. A test at a 5 percent size has, by design, a 5 percent chance of rejecting a true null, a false positive. That is acceptable for one test. But the false positives accumulate: across many independent tests, the probability that at least one rejects a true null grows quickly, and with enough tests a spurious “significant” result becomes almost certain.
The arithmetic is stark. If twenty unrelated strategies, none of which truly works, are each tested at 5 percent, the chance that at least one looks significant is one minus the chance that none do, or 1 minus 0.95 to the twentieth power, which is about 64 percent. This is the engine behind data mining and backtest overfitting: search across enough signals and something will always pass, not because it works but because chance guarantees a winner. The defence is to account for the number of tests run, to demand stronger evidence when many hypotheses are screened, and to be deeply skeptical of a lone significant result plucked from a large search.
A significant result is only as trustworthy as the search that produced it. One pre-specified test at 5 percent means what it says. The same 5 percent threshold applied to the best of a hundred backtested strategies means almost nothing, because roughly five of a hundred worthless strategies will clear the bar by chance. Always ask how many hypotheses were tried before the winning one was reported.
The standard remedy is to raise the bar in proportion to the number of tests. The simplest version divides the overall size by the number of tests, so screening twenty strategies at an overall 5 percent means each one must clear a far stricter 0.25 percent threshold to count as significant. This deliberately makes each individual test harder to pass, trading away some power to hold the total false-positive rate back down to the level intended. The precise correction matters less than the habit behind it: the evidence demanded from any one result should scale with the size of the search that surfaced it.
A risk team wants to know whether a value-at-risk model breaks too often, meaning more exceedances than the model allows. Should the test be one-sided or two-sided, and why?
Show answer
One-sided. The concern is only excess breaks, a departure in one direction; a model that breaks less often than expected is conservative, not a problem to flag. So the alternative is “more exceedances than allowed”, and the whole rejection region sits in that single tail. A two-sided test would waste power guarding against the harmless direction. The direction must be set by the risk concern before the exceedances are counted.
An analyst lowers the test size from 5% to 1% to be more careful. Holding the sample fixed, what happens to the chance of a Type II error, and to the power?
Show answer
Lowering the size to 1% pushes the critical value further out, so it becomes harder to reject any null. That reduces the chance of a Type I error but raises the chance of a Type II error, so beta goes up and the power, one minus beta, goes down. The two errors trade off at a fixed sample size; the only way to lower both is to collect more data. Being “more careful” about false alarms costs you missed detections.
A 95% confidence interval for a mean excess return runs from −0.5% to 3.5%. Without any further calculation, what is the verdict of a two-sided 5% test of the null that the mean is zero?
Show answer
Fail to reject. The interval contains zero, and a confidence interval is exactly the set of null values a matching test would not reject. Since zero lies inside the 95% interval, a two-sided 5% test of a zero mean does not reject the null. The interval and the test are the same calculation, so no separate test statistic is needed.
A researcher screens 40 factors for a link to returns, finds two significant at the 5% level, and reports only those two. Why should a reader be cautious?
Show answer
Because of multiple testing. At a 5% size, about 2 of 40 worthless factors are expected to look significant by chance alone, which is roughly what was found. Reporting only the winners, after screening many, hides how large the search was and makes chance results look like discoveries. The two “significant” factors may be pure noise; the finding would need to be confirmed on fresh data before it could be trusted.
Chapter Summary
- A test has six parts: null, alternative, test statistic, size, critical value, and decision rule. It asks how likely the data are if the null is true.
- The null is the maintained assumption; a test rejects it or fails to reject it, but never proves it. The alternative is what the analyst hopes to detect.
- A two-sided test rejects a departure in either direction (critical value 1.96 at 5%); a one-sided test uses a single tail (1.645 at 5%) and is chosen by the direction of interest.
- The test statistic is the estimate minus the null value over the standard error, standard normal under the null (Student’s t for small samples).
- A Type I error rejects a true null (probability α, the size); a Type II error keeps a false null (probability β); power is 1 − β. The two errors trade off at a fixed sample size.
- A confidence interval is the set of null values a test would not reject, so a value is rejected exactly when it falls outside the interval.
- A p-value is the probability of data at least as extreme as observed if the null is true; reject when it is below the size. It is not the probability the null is true.
- Two means are compared by dividing their difference by the standard error of the difference, which combines the two standard errors in quadrature.
- Running many tests inflates the chance of a false positive, so a lone significant result from a large search is weak evidence.
Frequently Asked Questions
What is the difference between the null and the alternative hypothesis?
The null hypothesis is the statement being tested, usually the assumption that nothing unusual is happening, such as a mean return equal to zero. It is assumed true throughout the test. The alternative hypothesis is the range of values that would count as evidence against the null, the outcome the analyst is really interested in detecting. A test never proves the null true; it either finds enough evidence to reject the null in favor of the alternative, or it fails to reject the null, which only means the evidence was not strong enough.
When should I use a one-sided versus a two-sided test?
Use a two-sided test when a departure in either direction matters, for example when checking whether a fund’s return differs from a benchmark, above or below. Use a one-sided test when only one direction is of interest, for example when checking whether a return is greater than zero, or whether a risk model produces too many exceedances. A one-sided test puts the whole rejection region in one tail, which makes it more powerful at detecting a departure in that direction, but blind to a departure in the other.
What is the difference between a Type I and a Type II error?
A Type I error is rejecting a null hypothesis that is actually true, a false positive. Its probability is the size of the test, denoted alpha, and it is chosen by the analyst, commonly at 5 percent. A Type II error is failing to reject a null that is actually false, a false negative. Its probability is denoted beta, and one minus beta is the power of the test. Lowering the chance of one error, at a fixed sample size, tends to raise the chance of the other, which is the central tradeoff of testing.
What are the size and the power of a test?
The size of a test is the probability of a Type I error, the chance of rejecting a true null hypothesis, and it is set by the analyst as the significance level, alpha. The power of a test is the probability of correctly rejecting a false null hypothesis, equal to one minus the probability of a Type II error. Power rises with a larger sample, a larger true effect, and a larger test size. Size is chosen; power is a consequence of the sample and the true state of the world.
How are a hypothesis test and a confidence interval related?
They are two views of the same calculation. A confidence interval at a given confidence level contains exactly the set of null values that would not be rejected by a test at the matching significance level. So a two-sided test at 5 percent rejects a null value precisely when that value falls outside the 95 percent confidence interval. Building the interval and running the test always give the same conclusion; the interval simply shows the whole range of values consistent with the data rather than a yes-or-no verdict on one value.
What does a p-value actually measure?
A p-value is the probability of observing a test statistic at least as extreme as the one actually seen, assuming the null hypothesis is true. A small p-value means the observed data would be surprising if the null were true, which is evidence against the null; the null is rejected when the p-value is below the chosen significance level. A p-value is not the probability that the null hypothesis is true, and it is not the probability that the result occurred by chance; it is a statement about the data under the assumption that the null holds.
What is the multiple testing problem?
Every test at a 5 percent size carries a 5 percent chance of a false positive even when the null is true. Run many independent tests and those small chances accumulate, so the probability that at least one test rejects a true null becomes large. Testing twenty unrelated strategies at 5 percent, for instance, makes a spurious significant result almost expected. This is why data mining across many signals produces findings that fail to hold up, and why the number of tests run must be accounted for before trusting any single significant result.
Loading comments...
Add your Thoughts: