FRM Part 1 · Quantitative Analysis · Chapter QTA 15
Machine Learning and Prediction
Learning Objectives
- Explain the role of linear regression and logistic regression in prediction.
- Evaluate the predictive performance of logistic regression models.
- Describe and apply methods used to encode categorical variables.
- Discuss why regularization is useful, and compare the ridge regression and LASSO approaches.
- Illustrate how a decision tree is constructed and interpreted.
- Describe how ensembles of learners are built.
- Explain the intuition and processes behind the K nearest neighbors and support vector machine methods for classification.
- Explain how neural networks are constructed and how their weights are determined.
- Compare the logistic regression and neural network classification approaches using a confusion matrix.
The previous chapter surveyed the ideas of machine learning: the data-driven philosophy, the split into supervised, unsupervised, and reinforcement learning, and the ever-present danger of overfitting. This chapter puts those ideas to work on the task machine learning is used for most in risk management, prediction. The question throughout is a practical one: given a set of features about a borrower, a security, or a transaction, how does a model produce a forecast, and how do we tell a good model from a bad one?
The chapter moves from the familiar to the new. It starts with regression, linear regression for predicting a number and logistic regression for predicting a category, then shows how a classifier is scored with a confusion matrix. It then covers the practical machinery that makes prediction work: encoding categorical features into numbers, and regularization to keep models from overfitting. The second half is a tour of the main prediction algorithms beyond regression, decision trees and the ensembles built from them, K-nearest neighbors and support vector machines, and neural networks, closing with how any two of these models are compared head to head.
Key Takeaways
- Linear regression predicts a continuous target; logistic regression predicts a probability through the sigmoid function and is estimated by maximum likelihood, then thresholded into a class.
- A binary classifier is scored with a confusion matrix, from which accuracy, precision, and recall are computed; the ROC curve and its area (AUC) summarize performance across thresholds.
- Categorical features are turned into numbers by one-hot (dummy) encoding, or by ordinal encoding when the categories have a natural order.
- Regularization adds a penalty on coefficient size to fight overfitting: ridge (L2) shrinks all coefficients toward zero; LASSO (L1) sets some exactly to zero, performing feature selection.
- Decision trees split data by information gain (entropy or Gini) into readable rules; ensembles (bagging, random forests, boosting) combine many trees into a stronger predictor.
- KNN classifies by majority vote of nearest neighbors; SVMs find the maximum-margin boundary; neural networks learn weights by gradient descent and backpropagation.
Linear and Logistic Regression in Prediction
Regression is the natural starting point for prediction because it is already familiar from earlier chapters, but the emphasis now shifts. In classical econometrics a regression is fitted to explain a relationship and to test whether coefficients are significant. In machine learning the same regression is used to predict, and it is judged on how accurately it forecasts new, out-of-sample data rather than on t-statistics.
Linear regression is used when the target is a continuous number: the next quarter’s revenue, a bond’s spread, a portfolio’s return. It fits a straight-line combination of the features and predicts the target directly. Its accuracy on new data is summarized by the mean squared forecast error, the average of the squared gaps between predicted and actual values on the held-out test set.
where yi is the actual value, ŷi the model’s prediction, and n the number of test observations. A lower MSFE means more accurate prediction. The square root of the MSFE, in the units of the target, is often quoted as the root mean squared error.
Many important risk questions, though, are not about a number but about a category: will this loan default or not, is this transaction fraudulent or genuine, will this counterparty be downgraded. Here the target is a label, usually a binary one coded as 1 or 0. Linear regression handles this badly, because a straight line is unbounded and will happily predict a “probability” of 1.4 or minus 0.2, which is meaningless. The fix is logistic regression.
Logistic regression takes the same linear combination of the features, call it y, but then squeezes it through the logistic (or sigmoid) function, an S-shaped curve that maps any number, however large or small, into the interval from 0 to 1. The output can therefore be read as a probability.
where F(y) is the predicted probability that the label is 1. As y grows large and positive, F(y) approaches 1; as y grows large and negative, it approaches 0; at y = 0 it equals exactly 0.5. The curve is the smooth S shown in Figure 1.
The coefficients cannot be found by ordinary least squares, because the outcome is a 0-or-1 label passed through a nonlinear curve, not a continuous value scattered around a line. Instead they are estimated by maximum likelihood: the algorithm searches for the coefficient values that make the observed sequence of ones and zeros most probable under the model. To turn the probability into a decision, it is compared with a threshold Z. If the predicted probability exceeds Z, the model predicts the positive class; otherwise the negative class. The threshold is a choice, not a fixed number, and moving it trades one kind of error for another, a point that matters greatly when the two errors have different costs.
A lender fits a logistic model for the probability that a small-business borrower defaults within a year. The fitted score is y = −3.0 + 1.5 D + 0.8 L, where D is a debt-service stress indicator (1 if stressed, 0 if not) and L is leverage measured in turns. For a borrower who is stressed (D = 1) with leverage of L = 2.0, find the predicted default probability. If the bank approves only applicants with a default probability below the threshold Z = 0.40, is this borrower approved?
Step 1. Compute the linear score.
Step 2. Pass the score through the sigmoid.
Answer: the predicted default probability is about 0.525, or 52.5%. That is above the 0.40 threshold, so the borrower is not approved. So what does the number mean: the model reads this stressed, moderately leveraged applicant as slightly more likely to default than not, and because that estimate sits above the bank’s risk tolerance, the application is declined.
Evaluating a Logistic Regression: the Confusion Matrix
Because a classifier’s output is a category, its accuracy cannot be judged by a squared error. It is judged instead by counting how often the predicted class matches the actual class, laid out in a confusion matrix, a two-by-two table cross-tabulating prediction against reality for a binary problem. The positive class is the event of interest, here a default. The four cells have standard names.
From these four counts come the headline scores. Accuracy is the share of all cases classified correctly. Precision asks, of the cases the model flagged as positive, how many really were: it measures the cost of false alarms. Recall (also called sensitivity) asks, of the cases that really were positive, how many the model caught: it measures the cost of misses. The error rate is simply one minus accuracy.
where TP, FP, FN, and TN are the counts of true positives, false positives, false negatives, and true negatives. Precision and recall trade off against each other: raising the threshold Z catches fewer positives but with fewer false alarms.
A default model is run on a test set of 1,000 loans, of which 200 actually defaulted. The confusion matrix comes out as: true positives 150, false negatives 50, false positives 90, true negatives 710. Compute the accuracy, precision, and recall, and interpret them.
Step 1. Accuracy: the correct predictions over all predictions.
Step 2. Precision: true positives over all predicted positives (150 + 90).
Step 3. Recall: true positives over all actual positives (150 + 50).
Answer: accuracy 86%, precision 62.5%, recall 75%. So what do these mean together: the model is right on most loans, but when it flags a loan as a default only about five in eight actually default (modest precision, many false alarms), while it catches three of every four real defaults (recall). For a lender who fears missing a bad loan more than annoying a good one, the 75% recall matters more than the precision, and the threshold could be lowered to catch more defaults at the cost of more false alarms.
No single threshold is right for every purpose, so a classifier is often summarized across all thresholds at once by the receiver operating characteristic (ROC) curve, which plots the true positive rate against the false positive rate as the threshold sweeps from high to low. The area under the curve (AUC) condenses that whole curve into one number: an AUC of 0.5 is no better than a coin toss, and 1.0 is perfect. AUC is a favorite for comparing classifiers because it does not depend on the choice of a single threshold.
Accuracy alone can be badly misleading when the classes are imbalanced. If only 2% of transactions are fraudulent, a lazy model that predicts “not fraud” for everything scores 98% accuracy while catching zero fraud, its recall is zero. This is why precision and recall are reported alongside accuracy: they expose exactly the errors that a single accuracy figure hides. On a rare-event problem, recall on the rare class is usually what the business cares about.
Encoding Categorical Variables
Machine-learning algorithms operate on numbers, but many useful features are categorical: an industry sector, a credit rating bucket, a country, a product type. These must be converted into numbers before a model can use them, and the way it is done matters.
The default method is one-hot encoding, also called dummy encoding. A categorical feature with m categories becomes m separate columns, each a 0-or-1 indicator that is 1 when the observation falls in that category and 0 otherwise. A “sector” feature taking the values Technology, Energy, and Utilities becomes three columns, and a technology firm is coded (1, 0, 0). The virtue of one-hot encoding is that it imposes no false ordering: the model is not tricked into thinking Utilities is “greater than” Energy, which it would be if the three sectors were simply numbered 1, 2, 3.
When the categories do have a natural order, ordinal encoding is appropriate: low, medium, and high risk can sensibly become 1, 2, and 3, because the numbers preserve a real ranking. The rule is simple: use ordinal encoding only when the order is genuine, and one-hot encoding otherwise, so that unordered categories are never given a spurious numeric scale.
| Firm | Sector (raw) | is_Tech | is_Energy | is_Utilities |
|---|---|---|---|---|
| Firm A | Technology | 1 | 0 | 0 |
| Firm B | Utilities | 0 | 0 | 1 |
| Firm C | Energy | 0 | 1 | 0 |
| Firm D | Technology | 1 | 0 | 0 |
One caution: with many categories, one-hot encoding creates many columns, which can inflate the feature count and feed overfitting, exactly the problem regularization, next, is built to control.
Regularization: Ridge and LASSO
A model with many features can fit the training data almost perfectly and still predict new data poorly, the overfitting problem. Regularization fights this by adding a penalty on the size of the coefficients to the quantity the model minimizes. Ordinary regression minimizes just the sum of squared errors; a regularized regression minimizes the sum of squared errors plus a penalty that grows as the coefficients grow. The effect is to shrink the coefficients toward zero, deliberately accepting a little bias in exchange for a large fall in variance. That is why regularization is useful: it directly targets the high-variance side of the bias-variance tradeoff, and it tends to improve out-of-sample accuracy even though it worsens the in-sample fit.
The two standard forms differ only in the shape of the penalty. Ridge regression adds an L2 penalty, proportional to the sum of the squared coefficients. LASSO (least absolute shrinkage and selection operator) adds an L1 penalty, proportional to the sum of the absolute coefficients.
where bj are the coefficients and λ (lambda) is the penalty strength. When λ = 0 both reduce to ordinary regression; as λ rises the coefficients are pulled harder toward zero.
The difference in the penalty produces one decisive difference in behavior. Ridge shrinks every coefficient toward zero but essentially never sets one exactly to zero, so all features are retained, just dampened. LASSO, because of the geometry of the absolute-value penalty, drives some coefficients exactly to zero, effectively deleting those features from the model. LASSO therefore performs automatic feature selection and yields a sparser, more interpretable model, which is prized when there are many candidate features and only a few are expected to matter. A hybrid called the elastic net blends the two penalties to get some of each.
| Ridge | LASSO | |
|---|---|---|
| Penalty | L2: sum of squared coefficients | L1: sum of absolute coefficients |
| Effect on coefficients | Shrinks all toward zero | Sets some exactly to zero |
| Feature selection? | No, keeps all features | Yes, drops features |
| Best when | Many features all somewhat useful | Only a few features truly matter |
The penalty strength λ is a hyperparameter: not learned from the training data like an ordinary coefficient, but set from outside and tuned on the validation set. Too small a λ barely regularizes and leaves the overfitting in place; too large a λ over-shrinks and pushes the model toward underfitting. The validation set is used to find the value that predicts held-out data best.
Two candidate models fit the same data equally well in terms of squared error, but differ in one coefficient: Model A has a coefficient of 4, Model B has a coefficient of 2 on that feature (all other coefficients equal). Compare the penalty each pays under a ridge (L2) rule and under a LASSO (L1) rule, for the change from 4 to 2, with λ = 1.
Step 1. Ridge penalty on this coefficient: the square.
Step 2. LASSO penalty on this coefficient: the absolute value.
Answer: the ridge penalty falls from 16 to 4 (a drop of 12) when the coefficient halves, while the LASSO penalty falls from 4 to 2 (a drop of 2). So what does this reveal: the squared L2 penalty punishes large coefficients very hard, so it aggressively shrinks the biggest ones but eases off near zero, never quite reaching it. The absolute L1 penalty keeps pulling with the same constant force all the way down, which is what lets it push small coefficients the last step to exactly zero. That constant pull is the mechanical reason LASSO produces feature selection and ridge does not.
Decision Trees
A decision tree predicts by asking a sequence of simple yes-or-no questions about the features, following the branches down to a leaf that gives the answer. It looks exactly like a flowchart, and that is its great appeal: the path from the top (the root) to any leaf is a plain-language rule, so a tree is a white-box model whose reasoning can be read off and explained to a regulator or a credit committee. This transparency contrasts sharply with the black-box neural networks later in the chapter.
The tree is built greedily from the top down. At each node the algorithm examines every feature and every possible cutoff and picks the split that best separates the classes, then repeats the process on each resulting branch, and so on until the leaves are sufficiently pure or a stopping rule halts growth. “Best separation” is made precise by a measure of impurity. Two are standard. Entropy measures disorder; Gini impurity measures the chance of mislabeling a randomly picked item. Both are zero when a node is perfectly pure (all one class) and largest when the classes are evenly mixed.
where pc is the proportion of the node belonging to class c. The algorithm chooses the split with the largest information gain, the impurity of the parent node minus the weighted average impurity of the child nodes it produces.
The split with the greatest information gain, the biggest drop in impurity from parent to children, is the one chosen at each step. A worked calculation makes the mechanics concrete.
A node holds 40 loans, 16 of which defaulted (a 40% default rate). A candidate split on “collateral posted” divides them into a left branch of 24 loans with 4 defaults, and a right branch of 16 loans with 12 defaults. Using the Gini index, compute the information gain from this split.
Step 1. Parent Gini, with default proportion 16/40 = 0.40.
Step 2. Left branch (4/24 = 0.167 default) and right branch (12/16 = 0.75 default).
Step 3. Weighted child impurity, weighting by branch size (24/40 and 16/40).
Step 4. Information gain: parent impurity minus weighted child impurity.
Answer: the split lowers Gini impurity from 0.48 to 0.317, an information gain of 0.163. So what does that gain represent: the collateral question does real work, it herds most of the low-risk loans into the left branch and concentrates the defaults in the right, making both children purer than the parent. The tree-building algorithm computes exactly this gain for every candidate split and keeps the one with the largest value.
A single tree has one serious weakness: grown deep enough, it will keep splitting until every leaf is pure, memorizing the training data and overfitting badly. Growth is therefore controlled by stopping rules or by pruning back an overgrown tree, and, more powerfully, by combining many trees into an ensemble.
Ensembles of Learners
An ensemble combines many models into one, on the principle that a diverse committee of weak predictors outvotes the mistakes of any single member. Ensembles are among the most accurate methods in practice, and they are usually built from decision trees, whose individual instability is exactly what makes their combination powerful. Two broad recipes are built into the syllabus.
Bagging, short for bootstrap aggregation, trains many models in parallel, each on a different bootstrap resample of the data (a sample drawn with replacement), and then averages their predictions for a number or takes a majority vote for a class. Because each tree sees a slightly different sample, their errors are partly independent and tend to cancel when averaged, which mainly reduces variance. A random forest is bagging applied to trees with one extra twist: at each split only a random subset of the features is considered, which decorrelates the trees so their errors cancel even more thoroughly. The result is one of the most reliable off-the-shelf predictors available.
Boosting takes the opposite approach: it builds models in sequence, each new model trained to fix the mistakes the previous ones made, by giving more weight to the observations they got wrong. The models are then combined into a weighted sum. Boosting chips away mainly at bias and can reach very high accuracy, but because each model chases the previous errors it is more prone to overfitting and must be tuned with care. Exhibit 3 sets the two recipes side by side.
| Bagging (and random forests) | Boosting | |
|---|---|---|
| How models are built | In parallel, independently | In sequence, each fixing the last |
| Data each model sees | A bootstrap resample | Reweighted toward prior errors |
| Mainly reduces | Variance | Bias |
| Overfitting risk | Lower, robust | Higher, needs careful tuning |
The price of an ensemble’s accuracy is interpretability. A single decision tree is a readable rule; a random forest of five hundred trees is not, even though each tree in it is. This is the recurring tradeoff of predictive modeling: the most accurate models are often the least transparent. In regulated finance, where a lending decision may have to be explained to a customer or a supervisor, that opacity is a real cost, and it sometimes tips the choice toward a simpler, more explainable model even at a small loss of accuracy.
K-Nearest Neighbors and Support Vector Machines
Two more supervised classifiers round out the toolkit, each with a distinct intuition. K-nearest neighbors (KNN) is the simplest idea in machine learning: to classify a new point, look at the K training points closest to it in feature space and let them vote, assigning the new point to the majority class among its neighbors. There is no model to fit in advance; KNN just stores the data and does its work at prediction time. That makes it easy to understand but slow to predict on large datasets, and highly sensitive to the scale of the features, which is why the standardization of the previous chapter matters so much here, an unscaled large-magnitude feature would dominate the distance and swamp the others. The choice of K is a hyperparameter: a small K follows the data closely and risks overfitting to noise, while a large K smooths the boundary and risks underfitting.
A new firm is to be classified as investment grade or high yield. Its five nearest neighbors in the (leverage, coverage) feature space have known labels: investment grade, high yield, investment grade, investment grade, high yield. With K = 5, what does KNN predict?
Step 1. Count the labels among the five neighbors.
Answer: the majority (3 of 5) are investment grade, so KNN classifies the new firm as investment grade. So what does the method assume: that firms close together in the feature space tend to share a label, so the new firm most likely resembles the class of the crowd nearest to it. Note that an odd K avoids ties in a two-class vote.
A support vector machine (SVM) takes a very different view. Rather than store points, it looks for the boundary that best separates the two classes, and specifically the one with the widest margin, the largest gap between the boundary and the nearest training points of each class. Those nearest points, which sit on the edge of the margin and pin the boundary in place, are the support vectors. A wide margin tends to generalize well because it leaves the most room for new points to fall on the correct side. When the classes cannot be split by a straight line, the SVM uses the kernel trick to draw a curved, nonlinear boundary, in effect separating the classes in a higher-dimensional space. KNN and SVM are both supervised classifiers, but one is memory-based and local while the other learns a single explicit separating boundary.
Neural Networks
A neural network is the most flexible predictor in the FRM toolkit, and the archetypal black box. It is built from layers of simple units called neurons. The input layer receives the features. One or more hidden layers sit in the middle, and each of their neurons forms a weighted sum of the values feeding into it and passes that sum through a nonlinear activation function (such as the sigmoid met earlier, or the popular rectified linear unit). The output layer produces the final prediction, a number for regression or a probability for classification. A network with several hidden layers is what “deep learning” refers to. The nonlinear activations, stacked across layers, are what let the network approximate almost any relationship, including complex interactions no analyst specified.
The weights on all those connections are the parameters that must be learned, and there can be thousands or millions of them. They are chosen to minimize a loss function that measures how far the network’s predictions fall from the true labels. The minimization is done by gradient descent: start from random weights, compute the gradient of the loss with respect to every weight, and nudge each weight a small step in the direction that lowers the loss, repeating until the loss stops falling. Computing all those gradients efficiently is the job of backpropagation, which applies the chain rule to push the output error backward through the layers, one layer at a time, so every weight learns how it contributed to the error. Training is a repeated forward pass (features in, prediction out, loss measured) and backward pass (gradients back, weights updated), run over the data many times until the weights settle. The number of layers, the number of neurons, and the learning rate are hyperparameters tuned on the validation set.
Keep the vocabulary straight: weights are learned by the network from the data, while hyperparameters (number of layers, number of neurons, learning rate, and the penalty λ in a regularized model) are set from outside and tuned on validation data. A frequent trap is to call the number of hidden layers a weight; it is not, it is a hyperparameter. And remember the mechanism: weights are found by gradient descent, and the gradients are computed by backpropagation. The two are not the same thing, backpropagation supplies the gradients that gradient descent then uses.
Comparing Models with a Confusion Matrix
The chapter closes by pulling the threads together, because the exam asks candidates to compare two classifiers, typically a logistic regression against a neural network, and the confusion matrix is the tool that does it. The procedure is even-handed: fit each model on the training data, then run both on the same held-out test set, build a confusion matrix for each, and read off accuracy, precision, and recall. Because both models face identical test data, the comparison is fair.
There is rarely a single winner on every metric, so the choice depends on which error matters more for the problem. A neural network might achieve higher overall accuracy by capturing nonlinear interactions a logistic regression misses, yet a logistic regression might offer better recall on the rare positive class, or, decisively in a regulated setting, the transparency to explain each decision. Exhibit 4 shows a head-to-head comparison of the kind the exam expects.
| Metric | Logistic regression | Neural network |
|---|---|---|
| Accuracy | 0.86 | 0.89 |
| Precision | 0.63 | 0.71 |
| Recall | 0.75 | 0.68 |
| Interpretability | High (readable coefficients) | Low (black box) |
Read the exhibit the way a risk manager would. The neural network wins on accuracy and precision, so it makes fewer errors overall and raises fewer false alarms. But the logistic regression has higher recall, it catches more of the actual defaults, which is often the error a lender fears most, and it is fully interpretable. Which model to deploy is therefore not settled by accuracy alone; it turns on the relative cost of a missed default versus a false alarm, and on whether the decision must be explainable. The confusion matrix does not make that judgment, but it lays out exactly the numbers the judgment needs.
A logistic model outputs a default probability of 0.32 for a borrower. The bank declines any applicant whose probability is at or above a threshold of 0.30. Is the borrower declined, and what happens to the count of false alarms if the bank raises the threshold to 0.45?
Show answer
At a threshold of 0.30 the borrower’s 0.32 probability is above the cutoff, so the applicant is declined (predicted positive for default). Raising the threshold to 0.45 makes the model more demanding before it flags a default, so fewer applicants are predicted positive. That reduces the number of good borrowers wrongly flagged, the false positives (false alarms) fall, but it also lets through more true defaults, so false negatives (missed defaults) rise. Raising the threshold improves precision at the expense of recall.
An analyst has 30 candidate features and believes only a handful truly drive the target. Which regularization method should be preferred, and why?
Show answer
LASSO (the L1 penalty). Because LASSO can drive coefficients exactly to zero, it performs automatic feature selection, keeping the handful of features that matter and discarding the rest, which produces a simpler, more interpretable model that matches the analyst’s belief. Ridge would shrink all 30 coefficients toward zero but keep every feature in the model, so it would not deliver the sparse solution wanted here. If the analyst were unsure and wanted a blend, the elastic net combines both penalties.
A node of 50 observations is perfectly pure, all 50 belong to the same class. What are its entropy and its Gini impurity?
Show answer
Both are zero. With one class holding a proportion of 1 and the other 0, entropy = −(1)log2(1) = 0 (since log2 of 1 is 0), and Gini = 1 − (1² + 0²) = 1 − 1 = 0. A pure node has no disorder and no chance of mislabeling, so both impurity measures hit their floor of zero. A tree stops finding useful splits once its leaves are pure, and a split is valuable precisely when it moves child nodes closer to this state.
In training a neural network, which items are the “weights” learned from the data, and which are hyperparameters set from outside: (a) the connection strengths between neurons, (b) the number of hidden layers, (c) the learning rate?
Show answer
Only (a), the connection strengths between neurons, are the weights the network learns, by gradient descent using gradients from backpropagation. Both (b) the number of hidden layers and (c) the learning rate are hyperparameters: they are chosen from outside the training loop and tuned on the validation set, not learned from the training data. The distinction is a common exam point: the network learns its weights, but a human (or a search over the validation set) sets its architecture and learning rate.
Chapter Summary
- Linear regression predicts a continuous target (scored by mean squared forecast error); logistic regression predicts a probability via the sigmoid, is estimated by maximum likelihood, and is thresholded into a class.
- A binary classifier is scored with a confusion matrix (TP, FP, FN, TN): accuracy = (TP+TN)/all, precision = TP/(TP+FP), recall = TP/(TP+FN). Accuracy misleads under class imbalance; the ROC curve and AUC summarize all thresholds.
- Categorical features are converted by one-hot (dummy) encoding, which imposes no false order, or by ordinal encoding when the categories are genuinely ordered.
- Regularization adds a coefficient-size penalty to fight overfitting: ridge (L2) shrinks all coefficients toward zero; LASSO (L1) sets some to zero, giving feature selection; elastic net blends them; λ is tuned on validation data.
- Decision trees split by information gain (entropy or Gini) into readable white-box rules but overfit if grown too deep; ensembles combine trees, bagging and random forests cut variance in parallel, boosting cuts bias in sequence.
- KNN classifies by majority vote of the K nearest neighbors (scale-sensitive, memory-based); SVMs find the maximum-margin boundary and use kernels for nonlinear separation.
- Neural networks stack layers of weighted sums and nonlinear activations; weights are learned by gradient descent with gradients from backpropagation, while layers, neurons, and learning rate are hyperparameters.
- Two classifiers are compared by building a confusion matrix for each on the same test set; the better choice depends on the relative cost of each error and on the need for interpretability, not on accuracy alone.
Frequently Asked Questions
How are linear and logistic regression used in prediction?
Linear regression predicts a continuous target, such as a return or a spread, by fitting a straight-line relationship between the features and the target. Logistic regression predicts a category, most often a binary yes-or-no outcome such as default or no default, by first computing a linear combination of the features and then passing it through the logistic, or sigmoid, function so the output is a probability between zero and one. That probability is turned into a class by comparing it with a threshold. In machine learning both are usually judged on out-of-sample predictive accuracy rather than on the significance of individual coefficients.
Why is logistic regression estimated by maximum likelihood rather than least squares?
The target in logistic regression is a zero-or-one label, not a continuous value, and the model produces a probability through the nonlinear sigmoid function. Ordinary least squares, which minimizes squared errors around a straight line, is not appropriate for a bounded probability and a binary outcome. Instead the coefficients are chosen by maximum likelihood: the algorithm searches for the coefficient values that make the observed pattern of ones and zeros most probable under the model. This handles the nonlinearity and the bounded output correctly, which least squares cannot.
How are categorical variables encoded for a machine-learning model?
Models work on numbers, so a categorical feature such as sector or rating must be converted. The most common method is one-hot encoding, also called dummy encoding, which creates a separate zero-or-one indicator column for each category, set to one when the observation belongs to that category and zero otherwise. When the categories have a natural order, such as low, medium, and high, ordinal encoding can instead map them to ordered integers. One-hot encoding avoids imposing a false numerical order on categories that have none, which is why it is the default choice for unordered categories.
What is the difference between ridge and LASSO regularization?
Both add a penalty on the size of the coefficients to a regression to fight overfitting, shrinking the coefficients toward zero and trading a little bias for a large fall in variance. Ridge uses an L2 penalty, the sum of the squared coefficients, which shrinks every coefficient toward zero but never exactly to zero, so all features are kept. LASSO uses an L1 penalty, the sum of the absolute coefficients, which can drive some coefficients exactly to zero, so it performs automatic feature selection and yields a simpler model. Elastic net combines the two. In every case a hyperparameter lambda controls the strength of the penalty and is tuned on the validation data.
How is a decision tree built and interpreted?
A decision tree splits the data into branches by asking a sequence of yes-or-no questions about the features, ending in leaves that give a prediction. At each node the algorithm chooses the feature and cutoff that best separate the classes, measured by the reduction in impurity, using entropy or the Gini index; the split with the largest information gain is chosen, and the process repeats down each branch. Trees are called white-box models because the path from root to leaf is a readable set of rules, which makes them easy to interpret, but a single deep tree overfits easily, which is why trees are often combined into ensembles.
What is an ensemble of learners?
An ensemble combines many models into one stronger predictor, on the principle that a crowd of diverse weak learners beats a single one. Bagging, short for bootstrap aggregation, trains many models on bootstrap resamples of the data and averages or votes their predictions, which reduces variance; a random forest is bagging applied to decision trees with an extra random selection of features at each split, which decorrelates the trees. Boosting instead builds models in sequence, each new one focusing on the observations the previous ones got wrong, which mainly reduces bias. Ensembles usually predict more accurately than any single member, at the cost of being harder to interpret.
What are K-nearest neighbors and support vector machines?
K-nearest neighbors, or KNN, classifies a new observation by looking at the K training points closest to it in feature space and taking the majority class among them; it stores the data rather than learning an explicit model, so it is simple but slow to predict on large datasets and sensitive to the scale of the features. A support vector machine, or SVM, instead finds the boundary that separates the classes with the widest possible margin, the gap between the boundary and the nearest points of each class; using what is called the kernel trick it can also draw curved, nonlinear boundaries. Both are supervised classifiers, but KNN is memory-based while an SVM learns an explicit separating boundary.
How does a neural network determine its weights?
A neural network is a stack of layers of simple units, or neurons, connected by weights: the input layer takes the features, one or more hidden layers transform them through weighted sums passed into nonlinear activation functions, and the output layer produces the prediction. The weights are the parameters that must be learned. They are found by minimizing a loss function that measures prediction error, using gradient descent, and the gradients are computed efficiently by an algorithm called backpropagation, which works backward from the output error through the layers. Training repeats this forward-and-backward pass over the data until the weights settle.
What is a confusion matrix and how is it used to compare models?
A confusion matrix is a two-by-two table for a binary classifier that cross-tabulates predicted against actual outcomes into true positives, false positives, false negatives, and true negatives. From it come the standard scores: accuracy is the share of all cases classified correctly, precision is the share of predicted positives that were truly positive, and recall is the share of actual positives that were caught. To compare a logistic regression with a neural network, each is applied to the same test data, its confusion matrix built, and the scores read off; the model with the better mix of accuracy, precision, and recall for the problem at hand, or the larger area under its ROC curve, is preferred.
Loading comments...
Add your Thoughts: