Significance test of ROC value with Perfcurve (obtaining the p value)         

5 views (last 30 days)
I would like to compare two populations of data points using ROC analysis (calculating area under the curve). The ROC values are between 0.5 and 1:00. I would like to examine whether the ROC value is significantly different from 0.5. One approach is to randomly assign the data points to two populations (say 1000 times) and then check the significance. My question is that whether 'perfcurve' can perform such a significance test and provide the p value. If yes, could you please explain in detail (or with example) how the routine should be written. Thanks

Answers (1)

Ilya
Ilya on 19 Aug 2014
I can't explain in detail how the routine should be written because I don't understand how you are comparing the two populations. But I can give you an idea for what you could do. If you have a sufficiently recent version of perfcurve, you can use the 'NBoot' parameter to estimate the curve uncertainty by bootstrap. The 4th output from perfcurve is then a confidence interval for AUC at the test size specified by the 'Alpha' parameter. If this interval does not include 0.5, you can conclude that AUC is statistically different from 0.5 at this value of alpha.
  2 Comments
Marta
Marta on 5 Nov 2015
Hi! Is there any equally simple way to see whether AUCs from different classifiers are significantly different? Eq (3) of http://pubs.rsna.org/doi/pdf/10.1148/radiology.148.3.6878708 suggests a simple method for this that relies on estimates of the standard deviation of the AUC. Would you estimate the std(AUC) using the confidence intervals from the bootstrapping as detailed in your answer?
Many thanks, Marta
Ilya
Ilya on 2 Dec 2015
You can estimate the standard deviation of AUC using confidence intervals from bootstrap. The paper you are referring to prescribes estimating correlation between AUCs produced by two classifiers. This can be done by bootstrap too, but clearly you would need to write your own code for that.

Sign in to comment.

Categories

Find more on ROC - AUC in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!