How to evaluate Gaussian process regression model with other Evaluation Metrics than resubLoss(gprMdl)/loss?

The fitrgp (https://au.mathworks.com/help/stats/fitrgp.html) used L = resubLoss(gprMdl) for regression loss. How to use other probabilistic evaluation matrics for GPR model , for example, continuous ranked probability score (CRPS) or pinball?

Answer 1

[Aditya Srikar]

To evaluate a Gaussian Process Regression (GPR) model using different probabilistic metrics such as Continuous Ranked Probability Score (CRPS) or pinball loss, you can use the `predict` method of the GPR model and calculate the metric of interest based on the predicted distribution and true target values.

Here are the general steps to evaluate a GPR model using CRPS:

1. Predict the mean and standard deviation of the target variable using the `predict` method of the GPR model:

[yPred, yStd] = predict(gprMdl, X);

2. Calculate the CRPS using the `crps` function from the Statistics and Machine Learning Toolbox. You can represent the predictive distribution as a normal distribution using the predicted mean and standard deviation. Here's an example of how to calculate the CRPS assuming the true target values (`y`) are normally distributed:

N = length(y);
crpScore = 0;
for i = 1:N
    z = (y(i) - yPred(i)) / yStd(i);
    crpScore = crpScore + (2/sqrt(pi)) * ((sin(z)/z)^(2*pi)) + z - (2/sqrt(pi));
end
crpScore = crpScore / N;

3. You can also calculate pinball loss for a quantile q using the `quantile` function and the formula:

q = 0.9; % example quantile
yPred_q = quantile(yPred, q);
pinballScore = mean(max(q*(y - yPred_q), (q-1)*(yPred_q - y)));

By default, the `fitrgp` and `resubLoss` functions in MATLAB use mean squared error (MSE) as the default loss function for GPR. However, by manually implementing a custom loss function as shown above, you can evaluate the GPR model using a different loss metric.