Date: April 27, 2021
This chapter is not about how to develop a scorecard. Rather, it focuses on the key steps, resources needed, and evaluations a manager should know.
I have written two summaries on scorecard development which covers most of the points in this chapter, especially on the key steps and resources needed. You may check them out below.
The system deteriorates over time as the population and economy change. Therefore, as a manager, you need to understand and evaluate the effectiveness of your decision tools. The author mentioned a few tools/ charts to monitor the performance of a model.
- To be sure the score is statistically powerful, use K-S or Divergence Measures.
- To be sure the score is likely to work in the future, use Population Stability Reports.
- To be sure the cutoff is appropriate, use Historical Performance Tables.
K-S plots the percentage of cumulative good and bad account. A manager can look at the separation between the two curves. At the cut-off, the separation of more than 30 points is considered acceptable.
The divergence method plots the score distribution of both good and bad accounts. From the chart, you can inspect how much overlap there is between the good and bad accounts, i.e., how often a good could be mistaken for a bad (or vice versa).
Population stability can be monitored by two summaries. The first summary is called through-the-door population. It plots the expected score distribution with the actual population. A manager then can understand if there is any shift in population distribution.
The next summary is called Characteristic Stability. It summarized the percentage of sample in each bin of different variables and monitors their changes. For example, in the below figure, the Unknown category (bin) changes in Q3 compared with Q1, Q2 and development sample.
Finally, a Charge-Offs by Score is plotted. It summarizes the charge-off percentage in different score ranges. One thing to pay attention to is that the chart should use data with comparability, such as within a given time period, accounts from a particular marketing program, etc. We called it cohort/ vintage.
The author also talks about using two scoring system for decision making. It improves the discriminating power of the system since it improves your decision making in the grey area, something in the middle (cutoff).
The author mentioned a potential problem of using a two-score approach when accounts are first passed through the credit risk model and then the revenue model. The scenario is like this: marketing took the names that passed the risk model and sent offers only to those that passed the revenue model. The end result is that low-risk customers received no offer, while high-risk customers received an offer. They are not enough low-risk accounts to offset the risk of high-risk accounts. I think this problem is not specific to the two-score approach.
This chapter gives a very good overview on the tools a manager can use to monitor a credit score performance.