Bounding Box Is Not Showing For RCNNobjectdetector

Hi professionals,

I managed to get to the bounding box and it is not consistant, sometimes is it showing and working most times it is not!!

The image is appearing with the Region Of Interest but the bounding box is not around that said region!!!

Can someone help to me understanding what I am doing incorrectly or interpret the status of the issue Pretty Pretty Please!!!

everything seems to work but the bounding box!!

Here is my code:

clear
clc
% deepNetworkDesigner
gunfolder = '/Users/mmgp/Desktop/gunsGT';
save('gunlables.mat','gunfolder');

%% Specifying Image Amount In Specified Folder
total_images = numel(gunfolder);
%% Accessing Content of Folder TrainingSet Using Datastore

imds = imageDatastore(gunfolder,'IncludeSubFolders',true,'LabelSource','Foldernames');
%% Setting Output Function(images my have size variation resizing for consistency with pretrain net)
imds.ReadFcn=@(loc)imresize(imread(loc),[227,227]);
%% Counting Images In Each Category "If not equal this will create issues"
tbl=countEachLabel(imds);
%% Making Category The Same Number Of Images
minSetCount=min(tbl{:,2});
%% Splitting Inputs Into Training and Testing Sets
[imdsTrain,imdsValidation] = splitEachLabel(imds,0.7,'randomized');
size(imdsTrain)
%% Loading Pretrained Network
net = alexnet; %Trained on 1million+ images/classify images into 1000 object categories
% analyzeNetwork(net) % Display Alexnet architecture & network layer details

%% Alter InputSize Of 1st Layer/ Alexnet Image requirements is 277 width 277 height by 3 colour channels 
inputSize = net.Layers(1).InputSize;%Displays the input size of Alexnet
%% Counting Total Number Of Images Including Subfolders **IF AVAILABLE**
imgTotal = length(imds.Files);

%% Displaying Multiple Randomized Images Within The Dataset
% a = 4;
% b = 4;
% n = randperm(imgTotal, a*b);
% 
% figure(),
% Idx = 1;
% for j=1:a
%     for k=1:b
%         img=readimage(imds,n(Idx));
%         subplot(a,b,Idx)
%         imshow(img);
%         Idx=Idx+1;
%     end
% end
%% Replace Final Layer/Last 3 Configure For 1000 classes
% Finetuning these 3 layers for new classification
% Extracting all Layers except the last 3
layersTransfer = net.Layers(1:end-3);
%% List the image categories/Clases:
numClasses = numel(categories(imdsTrain.Labels));
layers = [
    layersTransfer
    fullyConnectedLayer(numClasses,'WeightLearnRateFactor',25,'BiasLearnRateFactor',25)
    softmaxLayer
    classificationLayer];

%% Training The Network
% Resizing images in datastore to meet Alexnet's size requirements
% Utilising Augmented Data Store for automatic resizing of training images
%% Augmented Data Store Prevents Over Fitting By Randomly Flipping Along The Vertical Axis
%  Stopping the network from memorizing exact details of the training data
%  Also Randomly Translates them up to 30 pixels horizontally & Vertically
pixelRange = [-30 30];
imageAugmenter = imageDataAugmenter( ...
    'RandXReflection',true, ...
    'RandXTranslation',pixelRange, ...
    'RandYTranslation',pixelRange);
augimdsTrain = augmentedImageDatastore(inputSize(1:2),imdsTrain, ...
    'DataAugmentation',imageAugmenter);

%% Utilising Data Augmentation For Resizing Validation Data
%  implemented without specifying overfit prevention procedures
%  By not specifying these procedures the system will be precise via
%  predicitons
%% Resizing Images, Assists With Preventing Overfitting 
augmentedTrainingSet = augmentedImageDatastore(inputSize ,imdsTrain,'ColorPreprocessing', 'gray2rgb');
augimdsValidation = augmentedImageDatastore(inputSize,imdsValidation,'ColorPreprocessing', 'gray2rgb');

%% Specifying Training Options
%  Keep features from earlier layers of pretrained networked for transfer learning
%  Specify epoch training cycle, the mini-batch size and validation data
%  Validate the network for each iteration during training
%  (SGDM)groups the full dataset into disjoint mini-batches This reaches convergence faster 
%  as it updates the network?s weight value more frequently increase the
%  computationl speed

%% Implementing For ***VISUAL*** Graphical Representations
% options = trainingOptions('sgdm', ...
%     'MiniBatchSize',32, ...
%     'MaxEpochs',20, ...
%     'InitialLearnRate',0.004, ...
%     'Shuffle','every-epoch', ...
%     'ValidationData',augimdsValidation, ...
%     'ValidationFrequency',3, ...
%     'Verbose',true, ...
%     'Plots','training-progress');

%% Implementing **WITH** The RCNN Object Detector

opts = trainingOptions('sgdm',...
    'Momentum',0.9,...
    'MiniBatchSize', 20,...
    'InitialLearnRate', 1e-4,...
    'LearnRateSchedule', 'piecewise', ...
    'LearnRateDropFactor', 0.1, ...
    'LearnRateDropPeriod', 8, ...
    'L2Regularization', 1e-5, ...
    'MaxEpochs', 80);


% opts = trainingOptions('sgdm','MiniBatchSize', 130,'InitialLearnRate', 1e-4,'MaxEpochs', 200);

[height,width,numChannels, ~] = size(imdsTrain);
imageSize = [height width numChannels];
inputLayer = imageInputLayer(imageSize);

%% Training network Consisting Of Transferred & New Layers. 
netTransfer = trainNetwork(augmentedTrainingSet,layers,opts);


%% Classifying Validation Images Utilising Fine-tuned Network
[YPred,scores] = classify(netTransfer,augimdsValidation);

%% Displaying 4 Validation Image Samples With Predicted Labels
% idx = randperm(numel(imdsValidation.Files),4);
% figure
% for i = 1:4
%     subplot(2,2,i)
%     I = readimage(imdsValidation,idx(i));
%     imshow(I)
%     label = YPred(idx(i));
%     title(string(label));
% end

%% Calculating Validation Data Classification Accuracy (Accuracy Labels Predicted Accurately By Network)
YValidation = imdsValidation.Labels;
accuracy = mean(YPred == YValidation);

%% Training the R-CNN detector. Training can take a few minutes to complete.
% Loading .MAT file, the ground truths and the Network layers
load('gTruth.mat');

% Positive and Negative Overlap Range Controls Which Image Patch is Used
rcnn = trainRCNNObjectDetector(gTruth, netTransfer, opts, 'NegativeOverlapRange', [0 0.3]);


%% Testing the R-CNN detector on a test image.
testimg = imread('Gun00011.jpg');
[bbox, score, label] = detect(rcnn, testimg, 'MiniBatchSize', 20);

%% Display strongest detection result.
[score, idx] = max(score)
bbox = bbox(idx, :)
annotation = sprintf('%s: (Confidence = %f)', label(idx), score)
detectedImg = insertObjectAnnotation(testimg, 'rectangle', bbox, annotation);
figure
imshow(detectedImg);
%% Remove the image directory from the path.
rmpath(gunfolder);

Answer 1

[Shashank Gupta]

Hi Matthew Parris,

I see what you are intent to do, But I have some confusion/doubt in your code. I can’t seem to find the RCNN Box regression Layer, the one which is responsible to give you the bounding box coordinates. I am also wondering how you managed to get some bounding box. The code which you share only contains classificationLayer which just seem to classify the number of objects the network should detect. May be try adding "box regression layer" and see if that solve your problem.

Let me know if something good comes up.

[Matpar]

Hi @ Shashank Gupta,

It's has been confusing me as well! My intent was to do transfer learning using alexnet.

Then replace the last 3 layers based on a matlab example https://uk.mathworks.com/help/deeplearning/ref/alexnet.html.

Then add the new layers so that they accept and implement my code.

Then use the rcnn detector for training and testing!

Please guide me if i am wrong I have been working on this assidiously to not get it!

can you show me an example based on your explainations?

Thanx a mill pal for acknowledging my request!

[Shashank Gupta]

Hi,

there is an existing example of RCNN training and testing, I am attaching the link

https://www.mathworks.com/help/vision/ug/faster-r-cnn-examples.html#CreateFastRCNNObjectDetectionNetworkExample-1

you can refer to this. First try to train the same model with the same data, then start customizing it according to your need. but start with the default network.

[Matpar]

Ok @ Shashank Gupta,

I will try this will response soon but question is this the same process like say if I was using Alexnet?

I started using this and I am understading it, it would be shame to now ge the hang of it and leave in the middle when something is not going right!.

Let me know but i will still work on your example as a solution and let you know!

The Example did not work for me I am, how do I point the newly designed network to the rcnnobjectdetector?

[Matpar]

HI @ Shashank Gupta,

I followed the entire example but this does not show how to connect the layers so that the;

%rcnn = trainRCNNObjectDetector(gTruth, lgraph, opts, 'NegativeOverlapRange', [0 0.3]);

see it and conducts the training!!!

The errors I am getting is and it's a tad challenging interpreting them!

ERRORS:

Error using trainRCNNObjectDetector (line 266)
Invalid network.
Error in test4 (line 72)
rcnn = trainRCNNObjectDetector(gTruth, newlgraph, opts, 'NegativeOverlapRange', [0 0.3]);
Caused by:
    Network: Multiple graph components. The layer graph must consist of a single connected component.
    Initial layers of disconnected components:
        layer 'input_1' (component with 177 layers)
        layer 'rcnnBoxFC' (component with 2 layers)
    Network: Too many output layers. The network must have one output layer.
    Detected output layers:
        layer 'rcnnClassification'
        layer 'rcnnBoxDeltas'
    Layer 'rcnnBoxFC': Unused input. Each layer input must be connected to the output of another layer.

Guide me to another example that has this exercise please? the code is not accepting it for some reason unknown to me! just like the previous exercise the old and new layers function as one network referencing this piece of code from the previous exercise:

    %% Step 16 Training network Consisting Of Transferred & New Layers. 
netTransfer = trainNetwork(augmentedTrainingSet,Tlayers,opts)

Now I can call the layers as one from the new variable netTransfer,

I am not seeing this process in the example you displayed please guide me? Thanx in advance for acknowledging my errors thus far as a student willing to learn and understand the examples!

[Matpar]

This where I have gotten to thus far, please help me understand what i am doing?

% Load a pretrained ResNet-50.
net = resnet50;
lgraph = layerGraph(net);

% Remove the last 3 layers. 
layersToRemove = {
    'fc1000'
    'fc1000_softmax'
    'ClassificationLayer_fc1000'
    };
lgraph = removeLayers(lgraph, layersToRemove);

% Specify the number of classes the network should classify.
numClasses = 2;
numClassesPlusBackground = numClasses + 1;

% Define new classification layers.
newLayers = [
    fullyConnectedLayer(numClassesPlusBackground, 'Name', 'rcnnFC')
    softmaxLayer('Name', 'rcnnSoftmax')
    classificationLayer('Name', 'rcnnClassification')
    ];

% Add new object classification layers.
lgraph = addLayers(lgraph, newLayers);

% Connect the new layers to the network. 
lgraph = connectLayers(lgraph, 'avg_pool', 'rcnnFC');

% Define the number of outputs of the fully connected layer.
numOutputs = 4 * numClasses;

% Create the box regression layers.
boxRegressionLayers = [
    fullyConnectedLayer(numOutputs,'Name','rcnnBoxFC')
    rcnnBoxRegressionLayer('Name','rcnnBoxDeltas')
    ];

% Add the layers to the network.
lgraph = addLayers(lgraph, boxRegressionLayers);

% Connect the regression layers to the layer named 'avg_pool'.
lgraph = connectLayers(lgraph,'avg_pool','rcnnBoxFC');

% Select a feature extraction layer.
featureExtractionLayer = 'activation_40_relu';

% Disconnect the layers attached to the selected feature extraction layer.
lgraph = disconnectLayers(lgraph, featureExtractionLayer,'res5a_branch2a');
lgraph = disconnectLayers(lgraph, featureExtractionLayer,'res5a_branch1');

% Add ROI max pooling layer.
outputSize = [14 14];
roiPool = roiMaxPooling2dLayer(outputSize,'Name','roiPool');
lgraph = addLayers(lgraph, roiPool);

% Connect feature extraction layer to ROI max pooling layer.
lgraph = connectLayers(lgraph, featureExtractionLayer,'roiPool/in');

% Connect the output of ROI max pool to the disconnected layers from above.
lgraph = connectLayers(lgraph, 'roiPool','res5a_branch2a');
lgraph = connectLayers(lgraph, 'roiPool','res5a_branch1');


*********************************************************************************
After the above piece of code what is next please guide me!! i really want to learn this!!
********************************************************************************************


%% Step 1 Deploying deepNetworkDesigner to edit Layers physically
gunfolder = '/Users/mmgp/Desktop/gunsGT';
save('gunlables.mat','gunfolder');
load('gTruth.mat')
%% Step 2 Specifying Image Amount In Specified Folder
total_images = numel(gunfolder);

%% Step 3 Accessing Content of Folder TrainingSet Using Datastore
imds = imageDatastore(gunfolder,'IncludeSubFolders',true,'LabelSource','Foldernames');

%% Step 4 Setting Output Function(images may have size variation resizing for consistency with pretrain net)
imds.ReadFcn=@(loc)imresize(imread(loc),[227,227]);

%% Step 5 Counting Images In Each Category "If not equal this will create issues"
tbl=countEachLabel(imds);

%% Step 6 Making Category The Same Number Of Images
minSetCount=min(tbl{:,2});

%% Step 7 Splitting Inputs Into Training and Testing Sets
[imdsTrain,imdsValidation] = splitEachLabel(imds,0.7,'randomized');
size(imdsTrain);

opts = trainingOptions('sgdm',...
    'Momentum',0.9,...
    'InitialLearnRate', 1e-4,...
    'LearnRateSchedule', 'piecewise', ...
    'LearnRateDropFactor', 0.1, ...
    'Shuffle','every-epoch', ...
    'LearnRateDropPeriod', 8, ...
    'L2Regularization', 1e-4, ...
    'MaxEpochs', 10,...
    'MiniBatchSize',20,...
    'Verbose', true);



%rcnn = trainRCNNObjectDetector(gTruth, newlgraph, opts, 'NegativeOverlapRange', [0 0.3]);