FRM Part 1 · Quantitative Analysis · Chapter QTA 14
Machine-Learning Methods
Learning Objectives
- Discuss the philosophical and practical differences between machine learning techniques and classical econometrics.
- Compare and apply the two methods utilized for rescaling variables in data preparation.
- Explain the differences among the training, validation, and test data sub-samples, and how each is used.
- Examine the differences between and consequences of underfitting and overfitting, and describe potential remedies for each.
- Explain how principal components analysis is used to reduce the dimensionality of a set of features.
- Describe how the K-means algorithm separates a sample into clusters.
- Explain the mechanics behind natural language processing and how it is used.
- Differentiate among unsupervised, supervised, and reinforcement learning models.
- Explain how reinforcement learning operates, and calculate Q-values utilized in the decision-making process.
Machine learning is an umbrella term for techniques that train a model to recognize patterns in data, and it has moved from the fringe to the center of risk management. Credit scoring, fraud detection, algorithmic trading, and stress modeling increasingly lean on it, driven by an explosion in both data and computing power. This chapter is a conceptual survey: it does not turn a candidate into a data scientist, but it builds the vocabulary and the core ideas needed to understand what these methods do, where they help, and where they go wrong.
The chapter has two halves. The first is about how machine learning thinks differently from the econometrics of earlier chapters, its data-driven philosophy, the way data are prepared and split, and the central danger of overfitting. The second is a tour of the key methods: principal components analysis and K-means for finding structure, natural language processing for text, and reinforcement learning for sequential decisions. Throughout, the organizing map is the split into supervised, unsupervised, and reinforcement learning.
Key Takeaways
- Machine learning is data-driven and prediction-focused, letting the algorithm choose features and functional forms, unlike theory-driven econometrics that tests hypotheses.
- The three families are supervised (labeled prediction), unsupervised (finding structure), and reinforcement (sequential decisions for reward).
- Features are rescaled by standardization (mean 0, unit variance) or normalization (min-max to a 0-to-1 range); data are split into training, validation, and test sets.
- Overfitting fits noise (low bias, high variance); underfitting misses the pattern (high bias, low variance); the bias-variance tradeoff sets the right complexity.
- PCA reduces many correlated features to a few uncorrelated components; K-means groups data into K clusters by nearest centroid.
- Reinforcement learning learns a policy over states and actions; a Q-value is the value of an action in a state, updated toward observed rewards.
Machine Learning Versus Classical Econometrics
The deepest difference is philosophical. In classical econometrics, the analyst brings a theory, specifies the model and the variables, and uses the computer mainly to estimate parameters and test hypotheses; the goal is explanation and statistical significance. In machine learning, the emphasis flips to prediction: the algorithm is given wide latitude to decide which features matter and what functional form fits best, and success is judged by how accurately it predicts new, out-of-sample data, not by t-statistics or R-squared.
The practical consequences follow from that. Machine learning shines when there is little guiding theory, when the relationships are nonlinear, and when there are a great many candidate features, situations where classical hypothesis testing struggles, in part because standard errors shrink toward zero in very large samples, so almost any predictor looks significant. Even the vocabulary differs: what econometrics calls independent variables, machine learning calls features or inputs; the dependent variable is the target or output; and its observed values are labels. Exhibit 1 sets the two approaches side by side.
| Classical econometrics | Machine learning | |
|---|---|---|
| Model comes from | Theory, chosen by the analyst | The data, chosen by the algorithm |
| Main goal | Explanation, hypothesis testing | Out-of-sample prediction accuracy |
| Functional form | Usually linear, pre-specified | Flexible, captures nonlinear interactions |
| Evaluation | Significance, R², diagnostics | Predictive accuracy on held-out data |
| Terminology | Independent / dependent variables | Features / targets, values are labels |
Supervised, Unsupervised, and Reinforcement Learning
Machine-learning methods fall into three families, and this split is the map for the whole chapter. Supervised learning uses labeled data to make predictions: either forecasting a number (regression, such as a stock’s next-year return) or assigning a category (classification, such as labeling a loan as likely to repay or default). It needs a target with known answers to learn from. Unsupervised learning has no target; it hunts for structure in the data, grouping observations into clusters or compressing many features into a few, and is used for tasks like anomaly and fraud detection. Reinforcement learning is different again: it makes a sequence of decisions in a changing environment to maximize a reward, learning the best strategy by trial and error.
| Family | Has labels? | Task | Example |
|---|---|---|---|
| Supervised | Yes | Predict a value or a category | Credit scoring; return forecasting |
| Unsupervised | No | Find structure or reduce dimensions | Clustering; fraud detection; PCA |
| Reinforcement | Reward signal | Sequential decisions to maximize reward | Optimal trade execution; hedging |
Data Preparation: Standardization and Normalization
Most machine-learning methods need all features on a comparable scale, or a variable measured in large numbers will dominate one measured in small numbers for no good reason. Two rescaling methods do the job. Standardization subtracts the mean and divides by the standard deviation, giving a feature with mean 0 and unit variance. Normalization, the min-max transformation, subtracts the minimum and divides by the range, squeezing the feature into the interval from 0 to 1.
where μ̂ and σ̂ are the sample mean and standard deviation, and xmin, xmax the sample minimum and maximum. Standardization has no bounds; normalization is bounded in [0, 1] but usually has neither mean 0 nor unit variance.
Which to use depends on the data. Standardization is preferred when the data have a wide scope or contain outliers, because normalization would push the bulk of the observations into a narrow band while a single extreme value stretched the 0-to-1 range, distorting the picture. Only the features are rescaled; the target being predicted usually is not. Good data cleaning, handling inconsistent recording, duplicates, outliers, and above all missing data (dropped or imputed with a mean or median), often consumes most of a project’s effort and matters as much as the choice of algorithm.
A feature has a value of 40, a sample mean of 30, a standard deviation of 8, a minimum of 10, and a maximum of 60. Compute its standardized and normalized values.
Step 1. Standardize: subtract the mean, divide by the standard deviation.
Step 2. Normalize: subtract the minimum, divide by the range.
Answer: a standardized value of 1.25 (1.25 standard deviations above the mean) and a normalized value of 0.60 (60% of the way from the minimum to the maximum). The two rescalings describe the same observation differently: one relative to the spread, the other relative to the range.
Overfitting, Underfitting, and the Bias-Variance Tradeoff
The central danger in machine learning is overfitting: a model so flexible that it fits the random noise in the training data as if it were signal. An overfit model looks brilliant in-sample, its training error near zero, but performs poorly on new data, because the noise it memorized does not repeat. In bias-variance terms it has low bias but high variance. Because machine-learning models can have thousands of parameters, overfitting is a far bigger threat here than in a small linear regression.
The opposite failure is underfitting: a model too simple to capture the real pattern, such as fitting a straight line to a curved relationship. It performs poorly everywhere, in-sample and out, because it is systematically wrong; it has high bias but low variance. The right model complexity sits between these poles, at the minimum of total error, which is the bias-variance tradeoff: adding complexity cuts bias but raises variance, and the best model balances the two. The remedies mirror the diseases, as Figure 2 shows.
Training, Validation, and Test Data
Because overfitting is invisible in-sample, machine learning defends against it by splitting the data into three parts, each with a job. The training set is what the algorithm learns from; it estimates the model’s parameters. The validation set is used to compare competing models and choose between them, for example to tune how complex the model should be. The test set is locked away and touched only at the very end, to give an unbiased estimate of the chosen model’s performance on data it has never seen.
The reason for three, not two, is subtle but important. Once the validation set has been used to select a model, it has quietly influenced the choice, so it can no longer give an honest read on true out-of-sample performance. The test set stays pristine precisely so that final number is trustworthy. A common split is roughly two-thirds for training and the rest for validation and testing. When data are scarce, cross-validation (repeatedly rotating which slice is held out) uses the data more efficiently. For time series, the split is chronological, train on the earliest data, validate on the middle, and test on the most recent, so the model is judged on the freshest period.
| Sub-sample | Used to | Touched during |
|---|---|---|
| Training | Estimate the model’s parameters | Model fitting |
| Validation | Compare and select among models | Model selection / tuning |
| Test | Give an unbiased final performance estimate | Once, at the very end |
Principal Components Analysis
Principal components analysis (PCA) is the workhorse of unsupervised dimensionality reduction. When many features carry overlapping information, because they are highly correlated, PCA replaces them with a handful of new, uncorrelated variables called components, each a linear combination of the originals, that together capture almost all of the information. A large, tangled feature set becomes a small, clean one that is easier to model and to understand.
The classic finance example is the yield curve. The daily movements of interest rates across many maturities are highly correlated, so instead of tracking every rate, PCA compresses them into three components with clear meaning: a level (a parallel shift, where all rates move together, typically the largest component), a slope (a twist, where short and long rates move oppositely), and a curvature. These three alone usually explain nearly all of the variation in the entire curve, so an analyst can model interest-rate risk with three factors instead of dozens of rates.
PCA trades a large number of correlated features for a small number of independent ones with almost no loss of information. That is valuable in two ways: it makes models faster and less prone to overfitting, since there are fewer inputs, and it makes the underlying sources of risk interpretable, level, slope, and curvature are far easier to reason about than the raw movements of twenty individual rates. The components are ordered by how much variation they explain, so keeping the first few is usually enough.
The K-Means Clustering Algorithm
K-means is a simple, widely used unsupervised algorithm that separates data into a chosen number of clusters, K, decided in advance. It works by iterating two steps until things stop moving: place K cluster centers (centroids) at random, then repeat, first assign every data point to its nearest centroid, then recompute each centroid as the average of the points assigned to it, until the centroids no longer change.
“Nearest” needs a distance measure. The two common choices are Euclidean distance (the straight-line, as-the-crow-flies distance) and Manhattan distance (the sum of the absolute differences along each dimension, like driving on a grid of streets). The quality of a clustering is measured by its inertia, the total squared distance of points from their assigned centroids; lower is better. The number of clusters K is chosen by trying several values and looking for an elbow in the plot of inertia against K, the point past which adding clusters barely helps. A drawback is that K must be set in advance, and the algorithm tends to find roughly spherical clusters.
Two data points sit at P = (1, 2) and Q = (4, 6) in a two-feature space. Compute the Euclidean and Manhattan distances between them.
Step 1. Euclidean: the square root of the summed squared differences.
Step 2. Manhattan: the sum of the absolute differences.
Answer: a Euclidean distance of 5 and a Manhattan distance of 7. The Euclidean measure is the direct diagonal route; the Manhattan measure is the longer grid route, summing the moves along each axis. K-means uses whichever measure is chosen to assign each point to its closest centroid.
Natural Language Processing
Natural language processing (NLP) is the branch of machine learning that turns text into something a model can use. Financial text is everywhere, news articles, earnings-call transcripts, regulatory filings, analyst notes, social media, and it carries information that numbers alone do not. The obstacle is that models work on numbers, not words, so the core of NLP is converting language into numerical features.
The mechanics, in outline, are to break text into pieces (words or tokens), then represent it numerically, by counting word frequencies, scoring words for sentiment, or mapping words to vectors that capture meaning, and finally feed those features into a supervised model. The applications in finance are direct: sentiment analysis gauges whether news about a company is positive or negative, classification sorts documents or flags disclosures, and text-derived signals feed trading and risk models. NLP is what lets the vast, unstructured world of written information become an input to quantitative decisions.
Reinforcement Learning and Q-Values
Reinforcement learning is the third family, and the one built for sequential decisions. An agent moves through an environment described by states; in each state it takes an action, receives a reward, and lands in a new state. Over many trials it learns, by trial and error, the policy, the choice of action in each state, that maximizes its total reward. In finance it is used for problems like executing a large order over time, or dynamically hedging a book, where each decision affects the next.
The learned knowledge is stored as Q-values. The Q-value Q(S, A) is the estimated value of taking action A in state S, and the best action in a state is simply the one with the highest Q-value. As new rewards are observed, Q-values are updated to reflect experience. The Monte Carlo update nudges the old Q-value a fraction of the way toward the total reward actually received.
where R is the total reward received and α (alpha) is the learning rate, a small number such as 0.05 that controls how fast the estimate adjusts. The term in brackets is the surprise, the gap between what was received and what was expected, and the Q-value moves a fraction α of the way to close it.
An action in a given state currently has Q = 0.50. The action is taken, the total subsequent reward turns out to be R = 2.0, and the learning rate is α = 0.10. What is the updated Q-value?
Step 1. Apply the Monte Carlo update.
Answer: the Q-value rises from 0.50 to 0.65. The reward of 2.0 was higher than the expected 0.50, so the estimate moves upward, but only by a fraction (the learning rate) of the surprise, so a single lucky outcome does not swing the estimate wildly. Over many trials these small updates converge on the true value of the action.
A refinement, temporal difference learning, updates using only the next step: the immediate reward plus the estimated value of the state the action leads to, rather than waiting for the full sequence of rewards to play out.
where V(next state) is the value of the best action available in the state reached, and the immediate reward is just the payoff from this one step. This lets learning happen after every step, without waiting for a full episode to finish.
The same action has Q = 0.50. Taking it earns an immediate reward of 0.3 and leads to a state whose value is V = 1.2. With α = 0.10, update the Q-value by the temporal difference method.
Step 1. Apply the temporal difference update with reward 0.3 and next-state value 1.2.
Answer: the Q-value updates to 0.60. Temporal difference learning did not wait for the whole reward sequence; it used the one-step reward plus the value of where the action led. This is why it can learn faster than the Monte Carlo method, which must wait for the full outcome before updating.
Reinforcement learning also balances exploration against exploitation. Exploiting means taking the best action known so far; exploring means trying a new one to learn more. A pure exploiter can get stuck on a mediocre choice it never questions, so the algorithm explores with some probability, and relies on exploitation more as it learns. Expect the exam to test the Q-value update arithmetic and the meaning of exploration versus exploitation.
A bank wants to (a) group its customers into segments with no pre-set categories, and (b) predict whether each loan applicant will default. Which type of learning fits each task?
Show answer
Grouping customers with no pre-set categories is unsupervised learning, specifically clustering, since there is no target label, only a search for structure. Predicting default is supervised learning, specifically classification, because it uses labeled historical data (loans known to have repaid or defaulted) to assign each new applicant to a category. The tell is whether a labeled target exists: no target means unsupervised, a known target means supervised.
A model scores 99% accuracy on the training data but only 60% on new data. What has happened, and name two remedies.
Show answer
This is overfitting: the model has memorized the noise in the training data, so it excels in-sample but generalizes poorly. It has low bias but high variance. Remedies include simplifying the model (fewer parameters or features), gathering more training data, and using a validation set or cross-validation to control complexity. The huge gap between in-sample and out-of-sample performance is the signature of an overfit model.
Why is a separate test set kept aside instead of just using the validation set to report final performance?
Show answer
Because the validation set has already been used to select the model, so it has influenced the choice and can no longer give an unbiased measure of out-of-sample performance, it would flatter the model that was tuned to it. The test set is held completely aside, untouched during fitting and selection, so its performance number is an honest estimate of how the chosen model will do on genuinely new data. Reusing the validation set for the final score would overstate the model’s quality.
An action has a Q-value of 1.0. It is taken, the total reward is 0.4, and the learning rate is 0.2. Using the Monte Carlo update, what is the new Q-value?
Show answer
Apply Qnew = Qold + α(R − Qold) = 1.0 + 0.2(0.4 − 1.0) = 1.0 + 0.2(−0.6) = 1.0 − 0.12 = 0.88. The reward of 0.4 was lower than the expected 1.0, so the Q-value falls, but only by a fraction (the learning rate) of the disappointment, from 1.0 to 0.88. Repeated updates like this let the estimate settle on the action’s true value.
Chapter Summary
- Machine learning is data-driven and prediction-focused: the algorithm chooses features and functional form, and success is out-of-sample accuracy, not significance. Its terminology (features, targets, labels) differs from econometrics.
- The three families are supervised (labeled prediction: regression or classification), unsupervised (structure-finding: clustering, PCA), and reinforcement (sequential decisions for reward).
- Features are rescaled by standardization (mean 0, unit variance) or normalization (min-max to [0, 1]); standardization is preferred with outliers or wide-scope data.
- Data are split into training (fit), validation (select), and test (final unbiased evaluation); the test set stays untouched until the end.
- Overfitting fits noise (low bias, high variance, poor out-of-sample); underfitting is too simple (high bias); the best complexity balances the bias-variance tradeoff.
- PCA reduces many correlated features to a few uncorrelated components (yield curve: level, slope, curvature).
- K-means clusters data into K groups by nearest centroid (Euclidean or Manhattan distance), with K chosen by the elbow in the inertia plot.
- NLP converts text into numerical features for sentiment and classification; reinforcement learning stores value in Q-values, updated toward observed rewards by the Monte Carlo or temporal difference method.
Frequently Asked Questions
How does machine learning differ from classical econometrics?
In classical econometrics the analyst specifies the model from theory, chooses the variables, and uses the computer mainly to estimate the parameters and test hypotheses, so the emphasis is on statistical significance and explanation. Machine learning is far more data-driven: the algorithm decides which features and functional forms best fit the data, it can capture complex nonlinear interactions that a linear model would miss, and it is judged on how accurately it predicts new, out-of-sample data rather than on hypothesis tests. Machine learning also handles very large numbers of features gracefully, where classical hypothesis testing struggles because standard errors shrink toward zero in huge samples.
What is the difference between standardization and normalization?
Both rescale variables so they are on a comparable scale, which many machine-learning methods require. Standardization subtracts the mean and divides by the standard deviation, producing a variable with mean zero and unit variance. Normalization, also called the min-max transformation, subtracts the minimum and divides by the range, producing a variable bounded between zero and one. Standardization is usually preferred when the data have a wide scope or contain outliers, because normalization would squash the bulk of the observations into a narrow band, whereas standardization keeps the relative spacing of the data.
What are training, validation, and test data?
A machine-learning dataset is usually split into three parts, each with a distinct role. The training set is used to estimate the model’s parameters, the part the algorithm actually learns from. The validation set is used to compare competing models and choose between them, for example to tune the model’s complexity. The test set is held back untouched and used only at the very end to give an unbiased estimate of the chosen model’s performance on data it has never seen. Keeping the test set completely separate is what makes its performance estimate honest, since the validation data have already been used in model selection.
What are overfitting and underfitting, and how are they remedied?
Overfitting is when a model is too complex and fits the noise in the training data as if it were signal; it performs excellently in-sample but poorly out-of-sample, and it has low bias but high variance. Underfitting is the opposite: the model is too simple to capture the real pattern, so it is biased and performs poorly everywhere, with high bias but low variance. Overfitting is remedied by simplifying the model, adding more data, or using a validation set to control complexity; underfitting is remedied by using a richer model or adding relevant features and interactions. Balancing the two is the bias-variance tradeoff.
What is principal components analysis?
Principal components analysis, or PCA, is an unsupervised technique for dimensionality reduction. It replaces a large set of correlated features with a small number of new, uncorrelated variables called components, each a linear combination of the originals, that together capture almost all of the information in the data. The classic example is the yield curve: the movements of many interest rates of different maturities can be summarized by just three components, a level (parallel shift), a slope (twist), and a curvature, which together explain nearly all of the variation. PCA makes the underlying sources of risk easier to see and to model.
How does the K-means clustering algorithm work?
K-means is an unsupervised algorithm that separates data into a number of clusters, K, chosen in advance. It starts by placing K cluster centers, or centroids, at random. It then repeats two steps until the centroids stop moving: assign each data point to its nearest centroid, using a distance measure such as Euclidean or Manhattan distance, and then recompute each centroid as the average of the points assigned to it. The quality of a clustering is measured by the inertia, the total squared distance of points from their centroids, and the number of clusters K is often chosen using an elbow in the plot of inertia against K.
What is the difference between supervised, unsupervised, and reinforcement learning?
Supervised learning uses labeled data to make predictions, either forecasting a numerical value, called regression, or assigning a category, called classification, such as labeling a loan as likely to repay or default. Unsupervised learning has no target or labels; it finds structure in the data, such as grouping observations into clusters or reducing many features to a few components. Reinforcement learning is about making a sequence of decisions in a changing environment to maximize a reward, learning the best policy by trial and error. The three differ in whether a target exists and whether the task is prediction, pattern-finding, or sequential decision-making.
How does reinforcement learning work, and what is a Q-value?
Reinforcement learning develops a policy for making a sequence of decisions that maximizes a reward. It works in terms of states, actions, and rewards: in each state the agent chooses an action, receives a reward, and moves to a new state, learning by trial and error. A Q-value, written Q of S and A, is the estimated value of taking action A in state S, and the best action in a state is the one with the highest Q-value. Q-values are updated as new rewards are observed, for example by the Monte Carlo method, which nudges the old Q-value toward the total reward actually received, scaled by a learning rate.
Loading comments...
Add your Thoughts: