Definition And History

• Models are widely used in the financial industry for decision-making, but with increased use comes increased model risk. Model risk has always been a concern due to the potential for large losses, and regulators have raised expectations for proper management. However, the tail nature of model risk often leads to underestimation and misplaced confidence in institutions’ ability to manage it. The rarity of catastrophic events also makes the cost of managing model risk seem disproportionate. Despite these challenges, effective model risk management is critical for mitigating potential losses and avoiding harm to stakeholders.
• Defining what counts as a model is crucial for managing model risk. Initially, models were seen as statistical tools used for forecasting, but the definition has since expanded to include other techniques, like non-statistical or qualitative models. This change was necessary as model risk management became more sophisticated and better understood.
• According to the US Federal Reserve:
  “The term model refers to a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates.
  The definition of model also covers quantitative approaches whose inputs are partially or wholly qualitative or based on expert judgment, provided that the output is quantitative in nature.”
• In the industry, it is now generally accepted that a model refers to any estimation method that utilizes data and is based on a set of assumptions to produce a forecast that has a degree of uncertainty. The focus is shifting towards the uncertain nature of the forecast rather than the specific method or technique used for estimation.

Model Risks

• Institutions face two primary risks from model risk.
  1. The first is execution risk, which arises when a model fails to perform its intended function.
  2. The second is conceptual errors, which occur when incorrect assumptions or modeling techniques are used in developing the model.
• Although execution errors may appear minor and unlikely to cause significant losses, they can lead to catastrophic consequences when combined with unfavorable circumstances and unfortunate events. The Barclays’ purchase of Lehman Brothers and the NASA Mars orbiter are notable examples of this phenomenon. Execution errors can stem from coding mistakes, improper implementation, using incorrect data, and other factors. Additionally, similar errors can occur in tools that are not classified as models.
• Identifying conceptual errors in a model can be challenging as it is often a matter of differing opinions and assumptions. While a model may be performing adequately in a specific context, it could be entirely wrong if the circumstances change. Modelers may document these limitations, but it is essential to communicate them transparently to non-experts who use the model. Model risk management should ensure that users understand when the model can be safely used and when it should not. The example of the CDO case study from the 2007-2009 financial crisis shows how the Gaussian copula assumption was valid under normal market conditions but failed when market conditions deteriorated, resulting in pricing errors.

Model Risk Management

• The model risk management (MRM) function is responsible for managing all aspects of a model’s lifecycle. It sets standards for model documentation, data quality, and versioning, and reviews and challenges models to minimize risks. MRM is staffed with independent experts in the second line of defense.
• To balance the cost of model validation with addressing model risk, MRM functions categorize models into tiers based on their risk level. Factors that are considered when assessing model tiering include the materiality of the model output, complexity, client-facing use, and regulatory use. The model tier determines the level of validation required.
• High-tier models are thoroughly reviewed with comprehensive back-testing and frequent full-scope validations every 2 to 3 years. Mid-tier and low-tier models undergo a similar process with depth and frequency tailored to the tier. All models are subject to an annual review of the environment, data, and other important elements to ensure no material changes have occurred since the last full-scope validation. If no changes are observed, the last validation results are still valid, and the model can continue to be used.
• In addition to the routine validations and reviews, MRM groups regularly monitor model performance through monitoring reports produced by model owners. Monitoring intervals are determined based on the frequency of model use. Capital models may only require annual or quarterly monitoring, while other models may require monthly or even weekly monitoring, particularly in dynamic spaces like fraud monitoring.
• MRM should be a continuous effort rather than just “point-in-time” periodic validations and reviews, which is still a challenge for many institutions. Initially, institutions adopted a “point-in-time” model due to the long development and deployment cycles of regulatory models. This approach allowed banks to operate smaller teams of validators, making staffing needs predictable and allowing validators to move on to the next scheduled task once a validation or review was completed. However, as the reliance on models grows and the environment in which models are deployed becomes more dynamic, MRM functions must move towards a more continuous risk management approach.
• Model risk management is part of the three lines of defense model, where the validation teams should not make the modeling teams complacent about their risk management practices. The first line, which includes model developers and model owners, should own and mitigate the risk while the second line, MRM, conducts an independent assessment of the risk and management practices. At mature institutions, QA/QC teams form the first line’s risk team and play a critical role in mitigating model risk, especially execution risk.

---

Case Study – Gaussian Copula and CDO Pricing

• In a situation like this, the risk management function’s primary role is to increase transparency by illuminating the so-called black box. While MRM needs to replicate the model to ensure the intended prices and no coding errors, its most critical responsibility is to challenge model assumptions and ensure that users comprehend its limitations. Effective communication is the most crucial skill required for successful MRM. Most financial institution model users lack a quantitative background and can deflect responsibility by claiming they cannot understand the model, leaving the modeler responsible for its efficacy. A good MRM function will help users understand the limitations in non-technical language to minimize model misuse.

Case Study – Barclays Acquisition of Lehman Brothers and Spreadsheet Error

• Lehman Brothers’ collapse in September 2008 triggered the 2008 global financial crisis. However, one lesser-known incident is when Barclays Capital nearly acquired Lehman Brothers’ 179 trading contracts by mistake. Barclays offered to purchase some of Lehman’s assets, including trading positions, three days after the bankruptcy. As part of the deal, the law firm representing Barclays had to submit the purchase offer to the US Bankruptcy Court’s website by midnight on September 18. Barclays sent an Excel file with assets they wanted to acquire to their law firm just a few hours before the deadline on September 18, 2008. The spreadsheet included hidden (instead of deleted) rows with the contracts they did not want to buy. A junior law associate reformatted the Excel file to a PDF document, but the hidden rows became visible again in the PDF file. The mistake was discovered on October 1 after the deal had been approved, and the law firm had to file a legal motion to exclude those contracts.
• The Excel error is a simple implementation mistake. Although the spreadsheet contains information about assets, it is not a model as there is no uncertainty surrounding the data. The error of forgetting to delete hidden rows nearly cost Barclays millions, highlighting the importance of proper review, challenge, and controls for even seemingly basic tools or models.

Case Study – NASA Mars Orbiter

• NASA’s loss of a $125 million Mars orbiter occurred due to a misalignment in the unit of measurement used by the Lockheed Martin engineering team (English units) and the agency’s team (metric system) during a crucial spacecraft operation. This oversight may seem too obvious to have gone unnoticed, and yet it led to the destruction of a multimillion-dollar satellite.
• This error highlights the importance of considering “assumptions” in the model, such as the choice of unit (e.g., metric vs. imperial) or other simple assumptions that are often taken for granted, such as the currency or discount factor in financial modeling. Despite the perceived benign nature of some of the mistakes caught by a solid model risk management function, executives may still find the associated costs to be an unnecessary burden. While most benign mistakes result in insignificant losses, a small subset of these mistakes can result in catastrophic losses, such as the loss of the Mars orbiter. It is impossible to predict which mistakes will lead to catastrophic losses and which ones will not.