This example shows how to manage Modelscape™ test metrics and their associated threshold objects using
MetricsHandler produces reports that summarize the metrics and the status of the metrics in the container relative to their thresholds.
For more information about test metrics and thresholds, see Credit Scorecard Validation Metrics and Fairness Metrics in Modelscape. To learn how to write your own metrics, see Test Metrics in Modelscape.
This example shows you how to set up some metrics and thresholds for mock data of a credit scoring model. This example creates a metrics handler object to visualize the metrics and summarize the results. It shows you how to set an overall status to the handler based on various metrics.
Set Up Test Metrics and Thresholds
Use the following random data as an example of training response data (
defaultIndicators) and model predictions (
rng('default'); scores = rand(1000,1); defaultIndicators = double(scores + rand(1000,1) < 1);
Create three metrics:
1) Area under the receiver operating characteristic curve (AUROC),
2) Cumulative accuracy profile (CAP Accuracy) ratio, and
3) Kolmogorov-Smirnov statistic.
For AUROC and CAP Accuracy ratio, set values greater than 0.8 as Pass, values less than 0.7 as Fail, and values between these as Undecided, requiring further inspection. Set no thresholds for the Kolmogorov-Smirnov statistic.
import mrm.data.validation.TestThresholds import mrm.data.validation.pd.* auroc = AUROC(defaultIndicators, scores); aurocThresholds = TestThresholds([0.7, 0.8], ["Fail", "Undecided", "Pass"]); cap = CAPAccuracyRatio(defaultIndicators, scores); capThresholds = TestThresholds([0.6, 0.7], ["Fail", "Undecided", "Pass"]); ks = KSStatistic(defaultIndicators, scores);
Add metrics to a Metrics Handler Object
Add the metrics created in the previous section to a
import mrm.data.validation.MetricsHandler mh = MetricsHandler(); append(mh, auroc, aurocThresholds); append(mh, cap, capThresholds); append(mh, ks); disp(mh)
MetricsHandler with properties: CAP: [1x1 mrm.data.validation.pd.CAPAccuracyRatio] AUROC: [1x1 mrm.data.validation.pd.AUROC] KS: [1x1 mrm.data.validation.pd.KSStatistic]
The handler contains these three metrics that can be accessed as properties of this handler object. This allows you to access constituent metrics' diagnostics and visualizations.
Interrogate Metrics Handlers
report method to view the performance of the model relative to the given metrics.
summaryTable = report(mh); disp(summaryTable)
Metric Value Status Diagnostic ____________________________ _______ ___________ ___________ Area under ROC curve 0.82905 Pass (0.8, Inf) Accuracy ratio 0.65809 Undecided (0.6, 0.7] Kolmogorov-Smirnov statistic 0.51462 <undefined> <undefined>
The model performs well on AUROC, whereas the "Undecided" status on the Accuracy Ratio suggests the model requires a closer look.
When the handler carries complex non-scalar metrics, use arguments
Metrics arguments with
report. For more information, see Fairness Metrics in Modelscape.
Set Overall Status to the Handler
For a handler with many metrics, set an overall status to the handler by associating a 'status interpreter' to the handler. This section shows how to use an interpreter supplied with Modelscape that is compatible with your threshold objects.
mh.StatusInterpreter = @mrm.data.validation.overallStatus; summaryTable = report(mh); disp(summaryTable)
Metric Value Status Diagnostic ____________________________ _______ ___________ ___________ Area under ROC curve 0.82905 Pass (0.8, Inf) Accuracy ratio 0.65809 Undecided (0.6, 0.7] Kolmogorov-Smirnov statistic 0.51462 <undefined> <undefined> Overall NaN Undecided <undefined>
The overall status is in general decided based on the status descriptions of the individual metrics. In the above case, the overall status is the 'worst' of the individual statuses - 'Undecided'.
Thresholding systems with other descriptive strings - for example "Red", "Amber", "Green" require a custom status interpreter to be implemented. To do this, see the instructions before the StatusInterpreter declaration in the MetricsHandler implementation.
Alternatively, modify the interpreter as required.
You can also set the StatusInterpreter for the handler immediately at construction:
mh2 = MetricsHandler('StatusInterpreter', @mrm.data.validation.overallStatus)