Model Risk Management (MRM) is a systematic process used by financial institutions (FIs) to govern, evaluate, and mitigate risks related to the utilization of mathematical models and quantitative techniques in their operations, particularly in the context of internal controls and governance for financial operations. These procedures are designed to reduce risks stemming from models and ensure their effective management. Initially considered necessary mainly for large banks employing data-driven forecasting models, this expectation now extends to smaller and midsize banks as well. Errors in spreadsheets and the data supporting them have become commonplace, especially given the advanced data modeling required for portfolio management and asset-liability management, which are both heavily reliant on data and models.
The banking landscape has become significantly more intricate in the last decade, with an increased reliance on data, analytics, automation, processing vast amounts of information for customer acquisition, and dependence on third-party providers. This proliferation and diversity of data and analytics underscore the critical need for a robust MRM program within banks.
Financial institutions employ models in various capacities, including liquidity assessment, profit and loss evaluation, investment portfolio performance, fair lending practices, and stress testing scenarios, among others. MRM also encompasses models used by vendors in Know Your Customer (KYC) processes, where it is the financial institution's responsibility to ensure that these providers have appropriate MRM frameworks in place.
Determining what qualifies as a model can be more challenging than it appears. The complexity of the analytics involved, the use of automation, and even the incorporation of AI techniques may not alone define an artifact or process as a model.
According to the SR 11-7 Supervisory Guidance on Model Risk Management (“SR 11-7”), the term model refers to a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates. This definition also encompasses quantitative approaches with partially or wholly qualitative inputs or expert judgment, provided that the output is quantitative in nature. A model can range from a simple spreadsheet using a formula to a complex combination of multiple predictive algorithms.
To determine if something qualifies as a model, the financial institution reviews all relevant artifacts, presentations, and processes. An entity is considered a model if it:
1. Provides predictive estimates, such as liquidity positions and capital ratios.
2. Drives ongoing business decisions, such as loan extensions or security sales.
3. Produces a range of outcomes based on different assumptions and input variables.
4. Comprises identifiable components: an information input component, a processing component that transforms inputs into estimates, and a reporting component that translates estimates into actionable business information.
A robust MRM framework necessitates each department to maintain an inventory of such artifacts and the results of the model classification exercise.
1. The 2008 Financial Crisis: One of the most significant financial crises in recent history was partly caused by the misuse of complex financial models. Many banks relied on mathematical models to assess the risk of mortgage-backed securities, but these models failed to account for the possibility of a housing market collapse, resulting in substantial financial losses. An effective MRM program could have scrutinized the assumptions and limitations of these models, leading to more conservative risk assessments.
2. Long-Term Capital Management (LTCM): In the late 1990s, LTCM, a hedge fund led by Nobel laureates and renowned financial experts, used complex mathematical models to manage its portfolio. However, these models did not consider extreme market events. When the Russian financial crisis occurred in 1998, LTCM faced massive losses and required a bailout. Robust model validation and risk assessment procedures could have identified the limitations of LTCM's models and prompted risk mitigation measures.
3. Credit Rating Agencies and the Subprime Mortgage Crisis: Credit rating agencies used models to assess the creditworthiness of mortgage-backed securities. These models underestimated the risk of subprime mortgages, contributing to the 2008 financial crisis. A more rigorous MRM framework within rating agencies could have revealed the model deficiencies and prompted adjustments to their risk assessment processes.
4. ProMedica Health System's Bond Default (2016): ProMedica, a non-profit healthcare system, issued bonds with variable-rate debt tied to the London Interbank Offered Rate (LIBOR). They used a financial model that projected lower interest costs. However, when LIBOR rates unexpectedly spiked, it strained ProMedica's finances, leading to a bond default. A more comprehensive MRM framework could have accounted for unexpected interest rate fluctuations and assessed the model's sensitivity to rate changes.
Examples Relevant to Regional and Community Banks
1. Interest Rate Risk Management (Late 1990s - Early 2000s): Some community banks during this period relied on interest rate risk models that underestimated the potential impact of significant interest rate changes. When rates rose unexpectedly, these banks experienced losses on their portfolios. Improved MRM practices, including more rigorous model validation and sensitivity analysis, could have helped identify these vulnerabilities and prompted risk mitigation measures.
2. Commercial Real Estate Lending (Mid-2000s): In the mid-2000s, several community banks expanded their commercial real estate lending portfolios based on optimistic real estate valuation models. When the real estate market declined, these banks faced substantial losses. Enhanced MRM would have involved stress testing these models under adverse scenarios to gauge their robustness and assess the potential risks.
3. Small Business Loan Defaults: Some community banks have faced issues related to small business loan default predictions. Overreliance on models that failed to incorporate local economic factors or industry-specific risks led to higher-than-expected loan defaults. Effective MRM would have involved continuous validation and recalibration of these models to ensure accuracy and relevance.
4. Vendor Risk Assessment: Community banks often engage third-party vendors for services such as data processing or cybersecurity. When assessing the risk associated with these vendors, some banks relied on models that did not adequately capture the operational and security risks. Improved MRM would have included a more comprehensive evaluation of vendor risk models and their assumptions.
5. Branch Expansion Models: Community banks sometimes use models to assess the feasibility of opening new branches. However, these models may not always account for changing customer behavior, local demographics, or competitive factors accurately. Effective MRM would involve regular model validation and adjustment to reflect changing market dynamics.
6. Loan Portfolio Diversification: To diversify loan portfolios, some community banks relied on models that recommended new types of loans. However, these models occasionally failed to consider local economic conditions and credit risk adequately. Robust MRM would involve ongoing monitoring of loan performance and model recalibration to ensure prudent lending practices.
The utilization of incorrect or misused models can lead to adverse consequences, including financial losses, giving rise to model risks. Each identified model must have clearly defined users and use cases, input data, and expected outcomes. FIs should manage model risk at both the individual model and aggregate levels. They must be vigilant for design flaws, the expertise of those developing the model, limitations, and assumptions. The use of open-source libraries like Python and R, as well as off-the-shelf components, can contribute to fundamental errors if developers misunderstand these open-source functions or lack the capability to test the model at each development step. Inappropriate usage of a model can involve using an intermediate result from the model or employing it in a decision process that was not the intended and thoroughly vetted use case. When it comes to "model failure," typically meaning an inaccurate result from a model causing a loss or other impact to the FI, developers often point to inappropriate usage. The saying "all models are wrong, some are useful" emphasizes this issue and highlights that it is the usage, not just the design, that is the primary source of model risk.
An effective MRM framework encompasses Model Governance, Identification, Inventory, Risk Assessment, Development, Business Continuity, Validation, Monitoring, Risk Appetite, and Risk Reporting. The MRM Framework within a financial institution should align closely with Enterprise Risk Management and Data Management Programs.
For an MRM Framework to be effective, the financial institution must foster a culture of constructive criticism, effective challenges, and competence within the three lines of defense. Smaller financial institutions may consider outsourcing MRM support to help them achieve their risk identification and mitigation goals.
The focus of the MRM Framework should not solely be on the model itself but also on its intended usage. For example, using a simple spreadsheet-based model for liquidity stress testing carries higher risk than employing a complex AI-based algorithm that guides risk managers in selecting potential accounts, transactions, and complaints for compliance reviews.
If a financial institution relies on a third-party service using a model, it is expected to leverage its Third-Party Risk Management program to ensure that the third-party adopts a sound MRM Framework.
FIs must determine mitigating actions based on a model risk assessment that considers all relevant factors.
One of the initial and most valuable risk mitigation activities a financial institution can perform is verifying that models are performing as expected, aligning with their design objectives and business uses. A reasonable degree of independence between validator and developer is expected; testing during the model's development and implementation does not constitute validation. For higher-risk models, such as those used in liquidity stress testing, an independent third-party validation is highly advisable. The use of tools that enable the checking of intermediate outputs and stress testing the model itself enhances the validation process. Models should be validated before deployment into production, after any modifications, and on a recurring basis to ensure their continued effectiveness, especially in the face of significant changes in the macroeconomic environment.
For heavily used models, such as loan-level decision models or KYC models, appropriate monitoring is highly advisable to detect changes in macroeconomic conditions, rising interest rates, and default rates.
FIs should judiciously use risk appetite statements where appropriate and feasible for model risks. Such statements should specify actions triggered if the risk appetite is exceeded. Is suspending the use of a model a viable alternative? If not, what actions would the financial institution take if a critical model becomes nonviable, as in the case of a breach in a KYC model? Is redundancy possible in such cases? Are manual workarounds available? These questions must be addressed in advance.
An effective data management program is essential for the success of the MRM Framework. The use of inconsistently populated or deficient data can lead to unreliable results. For instance, many KYC models source data from multiple third-party providers, often unverifiable. Another example is the data used by several providers in verifying Cannabis-Related Businesses, which relies on data harvested from publicly available sources, often lacking critical elements. Even in cases of a "hit" or a positive identification, these data sources may not provide high confidence in the results.
While models are expected to reduce inconsistency in decision-making, most models tend to replicate the majority of decision makers' views or "expert opinions," potentially amplifying deficiencies from historical data. Supervised Learning-based AI methods are particularly susceptible to this risk. In the financial services industry, unforeseen scenarios and new regulations may mean that there is sometimes no historical data to base models on. The CARES Act of 2021 is a recent example of such a case. FIs should consider relying on more modern techniques like Unsupervised Learning to develop models whenever feasible and appropriate.
Effective governance, aligned with the FI's Enterprise Risk Management Framework, is essential to ensure that all departments consistently and regularly employ the MRMR Framework. The financial institution must also ensure that the status and results of MRM activities are regularly presented to executives and the Board of Directors. When presenting critical results from models, such as liquidity stress testing, it is appropriate to include evidence of MRM activities pertaining to the stress testing model.
In summary, MRM is a comprehensive framework that aims to identify, assess, validate, and monitor the risks associated with the use of quantitative models in financial institutions. Its purpose is to prevent financial losses, protect reputation, and ensure accurate decision-making.
This presentation is being furnished on a confidential basis to provide preliminary summary information. The information, tools and material (collectively, information) contained herein is not directed to or intended for distribution or use by any person or entity who is a citizen or resident of or located in any jurisdiction where such distribution, publication, availability or use would be contrary to law or regulation or which would subject Endurance Advisory Partners, LLC, to any registration or licensing requirement within such jurisdiction.
The information presented herein is provided for informational purposes only and is not to be used or considered as an offer to sell, or buy securities or other financial instruments, or any advice or recommendation with respect to such securities or other financial instruments. The information may not be reproduced in whole or in part or otherwise made available without the prior written consent of Endurance Advisory Partners, LLC. Information and opinions presented have been obtained or derived from sources believed to be reliable, but Endurance Advisory Partners, LLC makes no representation as to their accuracy or completeness. Endurance Advisory Partners, LLC, accepts no liability for any loss arising from the use of the information contained herein.
This information is subject to periodic update and revision. Materials should only be considered current as of the date of the initial publication, without regard to the date on which you may access the information. Endurance Advisory Partners, LLC, maintains the right to delete or modify the information without prior notice.
Under no circumstances and under no theory of law, tort, contract, strict liability or otherwise, shall Endurance Advisory Partners, LLC be liable to anyone for any damages resulting from access or use of, or inability to access or use, this information regardless of whether they are dire, indirect, special, incidental, or consequential damages of any character, including damages for trading losses or lost profits, or for any claim or demand by any third party, even if Endurance Advisory Partners, LLC knew or had reason to know of the possibility of such damages, claim or demand.
5420 LBJ Freeway, Suite 1940
Dallas, TX 75240
1350 S Boulder Ave 3rd floor
Tulsa, OK 74119
