Significance test of ROC value with Perfcurve (obtaining the p value)
10 views (last 30 days)
I would like to compare two populations of data points using ROC analysis (calculating area under the curve). The ROC values are between 0.5 and 1:00. I would like to examine whether the ROC value is significantly different from 0.5. One approach is to randomly assign the data points to two populations (say 1000 times) and then check the significance. My question is that whether 'perfcurve' can perform such a significance test and provide the p value. If yes, could you please explain in detail (or with example) how the routine should be written. Thanks
Ilya on 19 Aug 2014
I can't explain in detail how the routine should be written because I don't understand how you are comparing the two populations. But I can give you an idea for what you could do. If you have a sufficiently recent version of perfcurve, you can use the 'NBoot' parameter to estimate the curve uncertainty by bootstrap. The 4th output from perfcurve is then a confidence interval for AUC at the test size specified by the 'Alpha' parameter. If this interval does not include 0.5, you can conclude that AUC is statistically different from 0.5 at this value of alpha.