Machine Learning Models: Benefits and Challenges
Having good Prepayment and Credit Models is critical in the analysis of Residential Mortgage-Backed Securities. Prepays and Defaults are the two biggest risk factors that traders, portfolio managers and originators have to deal with. Traditionally, regression-based Behavioral Models have been used to accurately predict human behavior. Since prepayments and defaults are not just complex human decisions but also competing risks, accurately modeling them has been challenging. With the exponential growth in computing power (GPUs, parallel processing), storage (Cloud), “Big Data” (tremendous amount of detailed historical data) and connectivity (high speed internet), Artificial Intelligence (AI) has gained significant importance over the last few years. Machine Learning (ML) is a subset of AI and Deep Learning (DL) is a further subset of ML. The diagram below illustrates this relationship:
Due to the technological advancements mentioned above, ML based prepayment and credit models are now a reality. They can achieve better predictive power than traditional models and can deal effectively with high-dimensionality (more input variables) and non-linear relationships. The major drawback which has kept them from being universally adopted is their “black box” nature which leads to validation and interpretation issues. Let’s do a quick comparison between traditional and ML models:
Within ML Models are two ways to train them:
- Supervised Learning (used for ML Prepay and Credit Models)
- Regression based
- Classification based
- Unsupervised Learning
Let’s compare the major differences between Supervised and Unsupervised Learning:
The large amounts of loan level time series data available for RMBS (agency and non-agency) lends itself well for the construction of ML models and early adopters have reported higher accuracy. Besides the obvious objections mentioned above (black box, lack of control, interpretation) ML models are also susceptible to overfitting (like all other models). Overfitting is when a model does very well on the training data but less well on unseen data (validation set). The model ends up “memorizing” the noise and outliers in the input data and is not able to generalize accurately. The non-parametric and non-linear nature of ML Models accentuates this problem. Several techniques have been developed to address this potential problem: reducing the complexity of decision trees, expanding the training dataset, adding weak learners, dropouts, regularization, reducing the training time, cross validation etc.. The interpretation problem is a bit more challenging since users demand both, predictive accuracy and some form of interpretability. Several interpretation methods are used currently, like PDP (Partial dependence plot), ALE (accumulated local effects), PFI (permutation feature importance) and ICE (individual conditional expectation) but each has its shortcomings. Some of the challenges with the interpretability methods are:
- Isolating Cause and Effect – This is not often possible with supervised ML models since they only exploit associations and do not explicitly model cause/effect relationships.
- Mistaking Correlation for Dependence – Independent variables have a correlation coefficient of zero but a zero correlation coefficient may not imply independence. The correlation coefficient only tracks linear correlations and the non-linear nature of the models makes this difficult.
- Feature interaction and dependence – An incorrect conclusion can be drawn about the features influence on the target when there are interactions and dependencies between them.
While ML based prepay and credit models offer better predictive accuracy and automatically capture feature interactions and non-linear effects, they are still a few years away from gaining widespread acceptance. A good use for such models, at this stage, would be to use them in conjunction with traditional models. They would be a good benchmark to test traditional models with.
Note: Some of the information on this post was obtained from publicly available sources on the internet. The author wishes to thank Lei Zhao and Du Tang of the modeling group for proofreading this post.
Is Free Public Data Worth the Cost?
No such thing as a free lunch.
The world is full of free (and semi-free) datasets ripe for the picking. If it’s not going to cost you anything, why not supercharge your data and achieve clarity where once there was only darkness?
But is it really not going to cost you anything? What is the total cost of ownership for a public dataset, and what does it take to distill truly valuable insights from publicly available data? Setting aside the reliability of the public source (a topic for another blog post), free data is anything but free. Let us discuss both the power and the cost of working with public data.
To illustrate the point, we borrow from a classic RiskSpan example: anticipating losses to a portfolio of mortgage loans due to a hurricane—a salient example as we are in the early days of the 2020 hurricane season (and the National Oceanic and Atmospheric Administration (NOAA) predicts a busy one). In this example, you own a portfolio of loans and would like to understand the possible impacts to that portfolio (in terms of delinquencies, defaults, and losses) of a recent hurricane. You know this will likely require an external data source because you do not work for NOAA, your firm is new to owning loans in coastal areas, and you currently have no internal data for loans impacted by hurricanes.
Know the Data.
The first step in using external data is understanding your own data. This may seem like a simple task. But data, its source, its lineage, and its nuanced meaning can be difficult to communicate inside an organization. Unless you work with a dataset regularly (i.e., often), you should approach your own data as if it were provided by an external source. The goal is a full understanding of the data, the data’s meaning, and the data’s limitations, all of which should have a direct impact on the types of analysis you attempt.
Understanding the structure of your data and the limitations it puts on your analysis involves questions like:
- What objects does your data track?
- Do you have time series records for these objects?
- Do you only have the most recent record? The most recent 12 records?
- Do you have one record that tries to capture life-to-date information?
Understanding the meaning of each attribute captured in your data involves questions like:
- What attributes are we tracking?
- Which attributes are updated (monthly or quarterly) and which remain static?
- What are the nuances in our categorical variables? How exactly did we assign the zero-balance code?
- Is original balance the loan’s balance at mortgage origination, or the balance when we purchased the loan/pool?
- Do our loss numbers include forgone interest?
These same types of questions also apply to understanding external data sources, but the answers are not always as readily available. Depending on the quality and availability of the documentation for a public dataset, this exercise may be as simple as just reading the data dictionary, or as labor intensive as generating analytics for individual attributes, such as mean, standard deviation, mode, or even histograms, to attempt to derive an attribute’s meaning directly from the delivered data. This is the not-free part of “free” data, and skipping this step can have negative consequences for the quality of analysis you can perform later.
Returning to our example, we require at least two external data sets:
- where and when hurricanes have struck, and
- loan performance data for mortgages active in those areas at those times.
The obvious choice for loan performance data is the historical performance datasets from the GSEs (Fannie Mae and Freddie Mac). Providing monthly performance information and loss information for defaulted loans for a huge sample of mortgage loans over a 20-year period, these two datasets are perfect for our analysis. For hurricanes, some manual effort is required to extract date, severity, and location from NOAA maps like these (you could get really fancy and gather zip codes covered in the landfall area—which, by leaving out homes hundreds of miles away from expected landfall, would likely give you a much better view of what happens to loans actually impacted by a hurricane—but we will stick to state-level in this simple example).
Make new data your own.
So you’ve downloaded the historical datasets, you’ve read the data dictionaries cover-to-cover, you’ve studied historical NOAA maps, and you’ve interrogated your own data teams for the meaning of internal loan data. Now what? This is yet another cost of “free” data: after all your effort to understand and ingest the new data, all you have is another dataset. A clean, well-understood, well-documented (you’ve thoroughly documented it, haven’t you?) dataset, but a dataset nonetheless. Getting the insights you seek requires a separate effort to merge the old with the new. Let us look at a simplified flow for our hurricane example:
- Subset the GSE data for active loans in hurricane-related states in the month prior to landfall. Extract information for these loans for 12 months after landfall.
- Bucket the historical loans by the characteristics you use to bucket your own loans (LTV, FICO, delinquency status before landfall, etc.).
- Derive delinquency and loss information for the buckets for the 12 months after the hurricane.
- Apply the observed delinquency and loss information to your loan portfolio (bucketed using the same scheme you used for the historical loans).
And there you have it—not a model, but a grounded expectation of loan performance following a hurricane. You have stepped out of the darkness and into the data-driven light. And all using free (or “free”) data!
Hyperbole aside, nothing about our example analysis is easy, but it plainly illustrates the power and cost of publicly available data. The power is obvious in our example: without the external data, we have no basis for generating an expectation of losses after a hurricane. While we should be wary of the impacts of factors not captured by our datasets (like the amount and effectiveness of government intervention after each storm – which does vary widely), the historical precedent we find by averaging many storms can form the basis for a robust and defensible expectation. Even if your firm has had experience with loans in hurricane-impacted areas, expanding the sample size through this exercise bolsters confidence in the outcomes. Generally speaking, the use of public data can provide grounded expectations where there had been only anecdotes.
But this power does come at a price—a price that should be appreciated and factored into the decision whether to use external data in the first place. What is worse than not knowing what to expect after a hurricane? Having an expectation based on bad or misunderstood data. Failing to account for the effort required to ingest and use free data can lead to bad analysis and the temptation to cut corners. The effort required in our example is significant: the GSE data is huge, complicated, and will melt your laptop’s RAM if you are not careful. Turning NOAA PDF maps into usable data is not a trivial task, especially if you want to go deeper than the state level. Understanding your own data can be a challenge. Applying an appropriate bucketing to the loans can make or break the analysis. Not all public datasets present these same challenges, but all public datasets present costs. There simply is no such thing as a free lunch. The returns on free data frequently justify these costs. But they should be understood before unwittingly incurring them.
COVID-19 and the Cloud
COVID-19 creates a need for analytics in real time
Regarding the COVID-19 pandemic, Warren Buffet has observed that “we haven’t faced anything that quite resembles this problem” and the fallout is “still hard to evaluate.”
The pandemic has created unprecedented shock to economies and asset performance. The recent unemployment data, although encouraging , has only added to the uncertainty. Furthermore, impact and recovery are uneven, often varying considerably from county to county and city to city. Consider:
- COVID-19 cases and fatalities were initially concentrated in just a few cities and counties resulting in almost a total shutdown of these regions.
- Certain sectors, such as travel and leisure, have been affected worse than others while other sectors such as oil and gas have additional issues. Regions with exposure to these sectors have higher unemployment rates even with fewer COVID-19 cases.
- Timing of reopening and recoveries has also varied due to regional and political factors.
Regional employment, business activity, consumer spending and several other macro factors are changing in real time. This information is available through several non-traditional data sources.
Legacy models are not working, and several known correlations are broken.
Determining value and risk in this environment is requiring unprecedented quantities of analytics and on-demand computational bandwidth.
Need for on-demand computation and storage across the organization
“I don’t need a hard disk in my computer if I can get to the server faster… carrying around these non-connected computers is byzantine by comparison.” ~ Steve Jobs
Front office, risk management, quants and model risk management – every aspect of the analytics ecosystem requires the ability to run large number of scenarios quickly.
Portfolio managers need to recalibrate asset valuation, manage hedges and answer questions from senior management, all while looking for opportunities to find cheap assets. Risk managers are working closely with quants and portfolio managers to better understand the impact of this unprecedented environment on assets. Quants must not only support existing risk and valuation processes but also be able to run new estimations and explain model behavior as data streams in from variety of sources.
These activities require several processors and large storage units to be stood up on-demand. Even in normal times infrastructure teams require at least 10 to 12 weeks to procure and deploy additional hardware. With most of the financial services world now working remotely, this time lag is further exaggerated.
No individual firm maintains enough excess capacity to accommodate such a large and urgent need for data and computation.
The work-from-home model has proven that we have sufficient internet bandwidth to enable the fast access required to host and use data on the cloud.
Cloud is about how you do computing
“Cloud is about how you do computing, not where you do computing.” ~ Paul Maritz, CEO of VMware
Cloud computing is now part of everyday vocabulary and powers even the most common consumer devices. However, financial services firms are still in early stages of evaluating and transitioning to a cloud-based computing environment.
Cloud is the only way to procure the level of surge capacity required today. At RiskSpan we are computing an average of a half-million additional scenarios per client on demand. Users don’t have the luxury to wait for an overnight batch process to react to changing market conditions. End users fire off a new scenario assuming that the hardware will scale up automagically.
When searching Google’s large dataset or using Salesforce to run analytics we expect the hardware scaling to be limitless. Unfortunately, valuation and risk management software are typically built to run on a pre-defined hardware configuration.
Cloud native applications, in contrast, are designed and built to leverage the on-demand scaling of a cloud platform. Valuation and risk management products offered as SaaS scale on-demand, managing the integration with cloud platforms.
Financial services firms don’t need to take on the burden of rewriting their software to work on the cloud. Platforms such as RS Edge enable clients to plug their existing data, assumptions and models into a cloud–native platform. This enables them to get all the analytics they’ve always had—just faster and cheaper.
Serverless access can also help companies provide access to their quant groups without incurring additional IT resource expense.
A recent survey from Flexera shows that 30% of enterprises have increased their cloud usage significantly due to COVID-19.
Cloud is cost effective
“In 2000, when my partner Ben Horowitz was CEO of the first cloud computing company, Loudcloud, the cost of a customer running a basic Internet application was approximately $150,000 a month.” ~ Marc Andreessen, Co-founder of Netscape, Board Member of Facebook
Cloud hardware is cost effective, primarily due to the on-demand nature of the pricing model. A $250B asset manager uses RS Edge to run millions of scenarios for a 45–minute period every day. Analysis is performed over a thousand servers at a cost of $500 per month. The same hardware if deployed for 24 hours would cost $27,000 per month
Cloud is not free and can be a two-edged sword. The same on-demand aspect that enables end users to spin up servers as needed, if not monitored, can cause the cost of such servers to accumulate to undesirable levels. One of the benefits of a cloud-native platform is built-on procedures to drop unused servers, which minimizes the risk of paying for unused bandwidth.
And yes, Mr. Andreeseen’s basic application can be hosted today for less than $100 per month
The same survey from Flexera shows that organizations plan to increase public cloud spending by 47% over the next 12 months.
Alternate data analysis
“The temptation to form premature theories upon insufficient data is the bane of our profession.” ~ Sir Arthur Conan Doyle, Sherlock Holmes.
Alternate data sources are not always easily accessible and available within analytic applications. The effort and time required to integrate them can be wasted if the usefulness of the information cannot be determined upfront. Timing of analyzing and applying the data is key.
Machine learning techniques offer quick and robust ways of analyzing data. Tools to run these algorithms are not readily available on a desktop computer.
Every major cloud platform provides a wealth of tools, algorithms and pre-trained models to integrate and analyze large and messy alternate datasets.
Join fintova’s Gary Maier and me at 1 p.m. EDT on June 24th as we discuss other important factors to consider when performing analytics in the cloud. Register now.
Webinar: Data Analytics and Modeling in the Cloud – June 24th
On Wednesday, June 24th, at 1:00 PM EDT, join Suhrud Dagli, RiskSpan’s co-founder and chief innovator, and Gary Maier, managing principal of Fintova for a free RiskSpan webinar.
Suhrud and Gary will contrast the pros and cons of analytic solutions native to leading cloud platforms, as well as tips for ensuring data security and managing costs.
Click here to register for the webinar.
Chart of the Month: Tracking Mortgage Delinquency Against Non-traditional Economic Indicators by MSA
Tracking Mortgage Delinquency Against Non-traditional Economic Indicators by MSA
Traditional economic indicators lack the timeliness and regional granularity necessary to track the impact of COVID-19 pandemic on communities across the country. Unemployment reports published by the Bureau of Labor Statistics, for example, tend to have latency issues and don’t cover all workers. As regional economies attempt to get back to a new “normal” RiskSpan has begun compiling non-traditional “alternative” data that can provide a more granular and real-time view of issues and trends. In past crises, traditional macro indicators such as home price indices and unemployment rates were sufficient to explain the trajectory of consumer credit. However, in the current crisis, mortgage delinquencies are deteriorating more rapidly with significant regional dispersion. Serious mortgage delinquencies in the New York metro region were around 1.1% by April 2009 vs 30 day delinquencies at 9.9% of UPB in April 2020.
STACR loan–level data shows that nationwide 30–day delinquencies increased from 0.8% to 4.2% nationwide. In this chart we track the performance and state of employment of 5 large metros (MSA).
Indicators included in our Chart of the Month:
Change in unemployment is the BLS measure computed from unemployment claims. Traditionally this indicator has been used to measure economic health of a region. BLS reporting typically lags by months and weeks.
Air quality index is a measure we calculate using level PM2.5 reported by EPA’s AirNow database on a daily basis. This metric is a proxy of increased vehicular traffic in different regions. Using a nationwide network of monitoring sites, EPA has developed ambient air quality trends for particle pollution, also called Particulate Matter (PM). We compute the index as daily level of PM2.5 vs the average of the last 5 years. For regions that are still under a shutdown air quality index should be less than 100 (e.g. New York at 75% vs Houston at 105%)
Air pollution from traffic has increased in regions where businesses have opened in May ’20 (e.g. LA went up from 69% in April to 98% in May). However, consumer spending has not always increased at the same level. We look to proxies for hourly employment levels.
New Daily COVID-19 Cases: This is a health crisis and managing the rate of new COVID-19 cases will drive decisions to open or close businesses. The chart reports the monthly peak in new cases using daily data from Opportunity Insight
Hourly Employment and Hours Worked at small businesses is provided by Opportunity Insight using data from Homebase. Homebase is a company that provides virtual scheduling and time-tracking tools, focused on small businesses in sectors such as retail, restaurant, and leisure/accommodation. The chart shows change in level of hourly employment as compared to January 2020. We expect this is to be a leading indicator of employment levels for this sector of consumers.
Sources of data:
Freddie Mac’s (STACR) transaction database
Opportunity Insight’s Recovery Tracker
Bureau of Labor and Statistics (BLS)’ MSA level economic reports
Environment Protection Agency (EPA)’s AirNow database.