nnstart and test ROC AUC

Hi all,

I'm doing a project now that requires training a large number of pattern recognizing neural nets. It's my first time using them, so I've been using a "simple script" generated from Matlab's neural network toolbox.

during training, I'm able to look at the ROC for the train, test, and validation sets. I'm wondering if it's possible to somehow extract only the test numeric ROC AUC from the neural network after it's final iteration of training.

Any help would be great!

Thanks, J

Answer 1

[Greg Heath]

 clear all, close all, clc, plt=0;
 tic
 [ x, t ] = simpleclass_dataset;
 [ I N ] = size(x)        % [ 2 1000 ]
 [ O N ] = size(t)        % [ 4 1000 ]
 Ntst = round(0.15*N)     % 150 default
 Nval = Ntst              % 150 default
 Ntrn = N-Nval-Ntst       % 700 default
 Ntrneq = prod(size(t))   % 4000 No. of training equations

 % H = No. of hidden nodes    ( I-H-O NN )
 % Nw = (I+1)*H+(H+1)*O = No. of unknown weights
 % NO OVERFITTING <==> Nw < Ntrneq 
 % NO OVERFITTING <==>  H <= Hub  (upperbound)

 Hub = -1 + ceil( ( Ntrneq-O) / (I+O+1) )  % 570

% For a robust design desire Nw << Ntrneq. Therefore, if possible, choose H << Hub. The PATTERNNET default is H = 10. However, the smaller H, the more robust the design. Obtaining 110 designs ( 10 trials each for H = 0:10 shows that ANY OF THESE VALUES CAN YIELD EXCELLENT RESULTS PROVIDED THE RANDOM WEIGHT INITIALIZATION IS NOT TOO POOR. The complete result can be duplicated below by choosing Hmax = 10 with Ntrials = 10. However, the result below for Hmax = 0 with Ntrials = 4 is sufficient to illustrate the concept.

 Hmin = 0
 dH = 1
 Hmax = 0
 Ntrials = 4
 rng(0)                           % Initialize RNG
 j=0
 for h = Hmin:dH:Hmax
     j=j+1
     if h==0
         net = patternnet([]);;
         Nw = (I+1)*O
     else
         patternnet(h);
         Nw = (I+1)*h+(h+1)*O
     end
     for i = 1:Ntrials
         s(i,j) = rng              % Store state of RNG
         net = configure(net,x,t);
         [ net tr y ] = train(net,x,t);
         tstind = tr.testInd;
         ttst = t(:,tstind);
         ytst = y(:,tstind);
         plt=plt+1,figure(plt)
         plotconfusion(ttst,ytst)
         title([ ' TEST SET CONFUSION MATRIX. TRIAL =  ', num2str(10*(j-1)+i )] )
         hold off
         plt=plt+1,figure(plt)
         plotroc(ttst,ytst)
         title([ ' TEST SET ROC. TRIAL =  ', num2str(10*(j-1)+i )] )
         hold off
     end
 end
 toc                                   % 7.3 sec

Hope this helps.

Thank you for formally accepting my answer

Greg

[Jordan]

Thanks for the response Greg. Still it's not totally clear to me. All I really need here is some way to extract the test samples being used from the net object. If I had a list of which samples were being used for testing, I could calculate the AUC of the ROC. I don't need full NN code.

This is what was generated for me by nnstart:

______________

inputs = in'
targets = out
hiddenLayerSize = 30;
net = patternnet(hiddenLayerSize);
net.divideParam.trainRatio = 70/100
net.divideParam.valRatio = 15/100
net.divideParam.testRatio = 15/100
[net, tr] = train(net,inputs,targets);
outputs = net(inputs);
errors = gsubtract(targets, outputs);
performance = perform(net, targets, outputs);

______________

I don't really want to deviate from this code.

[Greg Heath]

I gave you the expressions for ttst and ytst.

What else do you need??

Greg