Unexpected error calling cuDNN: CUDNN_STAT​US_NOT_SUP​PORTED

Hi everyone,

I am running the training of a simple DNN for binary classification on Matlab R2020b and - when the training is complete or interrupted manually - I get the following error message:

Error using trainNetwork (line 183)

Unexpected error calling cuDNN: CUDNN_STATUS_NOT_SUPPORTED.

Error in SVM (line 181)

[cl] = trainNetwork(X',categorical(class'),layers,options)

What am I doing wrong? It was working perfectly until yesterday.

Thanks,

A.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Here is the code:

numClasses = 2;

featureDimension = size(X',1);

numHiddenUnits = 100;

layers = [ ...

sequenceInputLayer(featureDimension)

fullyConnectedLayer(numHiddenUnits)

batchNormalizationLayer

leakyReluLayer

fullyConnectedLayer(numClasses)

softmaxLayer

classificationLayer];

miniBatchSize = 1024;

options = trainingOptions('adam', ...

'MiniBatchSize',miniBatchSize, ...

'MaxEpochs',10^6, ...

'InitialLearnRate',1e-3, ...

'LearnRateSchedule','piecewise', ...

'LearnRateDropFactor',0.9, ...

'LearnRateDropPeriod',500, ...

'Shuffle','every-epoch', ...

'ExecutionEnvironment','gpu', ...

'Verbose',true,...

'ValidationData',{Xtest',categorical(classtest')},...

'ValidationFrequency',1,...

'ValidationPatience',40,...%'L2Regularization',0.0005,...'CheckpointPath',checkpointPath, ...

'Plots','training-progress');

[cl] = trainNetwork(X',categorical(class'),layers,options);

Answer 1

[Aditya Patil]

This is a known issue with cuDNN. Following are some workarounds,

1. Update to newer release of MATLAB. Newer releases might use newer versions of cuDNN that fix the issue.

2. Update cuDNN library. The steps are mentioned below.

3. Decrease mini batch size. The issue occurs due to large mini batch size.

Please follow the steps below to update the cuDNN library:

1. Download an appropriate version of cuDNN for your CUDA toolkit version and OS from https://developer.nvidia.com/cudnn.

a. You can see your CUDA toolkit version from the output of the "gpuDevice" command.

b. Here is a link for more info about installing the CUDA driver and toolkit: https://www.mathworks.com/help/parallel-computing/gpu-support-by-release.html

2. Unzip the downloaded file

3. Close MATLAB

4. Replace the existing library files in the following locations with the new one.

a. Notes:

i. You can get your MATLAB path by running the 'matlabroot' command in MATLAB.

ii. Before making the replacements, you should make a copy of the old files in case anything breaks.

b. For Windows:

i. Replace <yourmatlabpath>\matlab\sys\cuda\win64\cudnn\include\cudnn.h with <yourdownloadpath>\cuda\include\cudnn.h

ii. Replace <yourmatlabpath>\matlab\sys\cuda\win64\cudnn\lib\x64\cudnn.lib with <yourdownloadpath>\cuda\lib\x64\cudnn.lib

iii. Replace <yourmatlabpath>\matlab\bin\win64\cudnn64_7.dll with <yourdownloadpath>\cuda\bin\cudnn64_7.dll

c. For Linux:

i. Replace <yourmatlabpath>/matlab/sys/cuda/glnxa64/cudnn/include/cudnn.h with <yourdownloadpath>/cuda/include/cudnn.h

ii. Replace <yourmatlabpath>/matlab/bin/glnxa64/libcudnn.so with <yourdownloadpath>/cuda/lib64/libcudnn.so

iii. Replace <yourmatlabpath>/matlab/bin/glnxa64/libcudnn.so.7 with <yourdownloadpath>/cuda/lib64/libcudnn.so.7

iv. Replace <yourmatlabpath>/matlab/bin/glnxa64/libcudnn.so.7.5.0 with <yourdownloadpath>/cuda/lib64/libcudnn.so.7.6.x

5. Start MATLAB and verify that your script runs