train with multiple input to get two classes output

I have two folders

folder_1 with two subfolders(good, bad) each with (900 images and 100 image) respectively

folder_2 with two subfolders(good, bad) each with (900 images and 100 image) respectively. when training with pretrained (resnet50) on the "Deep netwoek designer" console, i get the next error about categorical response? any explaination please.

imds_1 = imageDatastore('C:\Users\Folder_1', ...
    'IncludeSubfolders',true, ...
    'FileExtensions','.jpg', ...
    'LabelSource','foldernames');
[imdsTrain_1,imdsValidation_1] = splitEachLabel(imds_1,0.75); %split the data into training and validation

imds_2 = imageDatastore('C:\Users\Folder_2', ...
    'IncludeSubfolders',true, ...
    'FileExtensions','.jpg', ...
    'LabelSource','foldernames');
[imdsTrain_2,imdsValidation_2] = splitEachLabel(imds_2,0.75); %split the data into training and validation

%  'train_ok.txt' contain the labels of the images in (imdsTrain_1 or imdsTrain_2)           750x1
%  'val_ok.txt'   contain the labels of the images in (imdsValidation_1 or imdsValidation_2) 250x1

labelStore = tabularTextDatastore('train_ok.txt','TextscanFormats','%C',"ReadVariableNames",false);
labelStoreCell = transform(labelStore,@setcat_and_table_to_cell);
train_multi = combine(imdsTrain_1,imdsTrain_2,labelStoreCell);
train_multi.read

labelStore2 = tabularTextDatastore('val_ok.txt','TextscanFormats','%C',"ReadVariableNames",false);
labelStoreCell2 = transform(labelStore2,@setcat_and_table_to_cell);
val_multi = combine(imdsValidation_1,imdsValidation_2,labelStoreCell2);
val_multi.read


%train_multi.read 750x1

    {224×224×3 uint8}    {224×224×3 uint8}    {[Good        ]}
    {224×224×3 uint8}    {224×224×3 uint8}    {[bad         ]}
    {224×224×3 uint8}    {224×224×3 uint8}    {[bad         ]}  ...  

%val_multi.read 250x1
    {224×224×3 uint8}    {224×224×3 uint8}    {[Good        ]}
    {224×224×3 uint8}    {224×224×3 uint8}    {[Good        ]}
    {224×224×3 uint8}    {224×224×3 uint8}    {[bad         ]} ....


function [dataout] = setcat_and_table_to_cell(datain)
validcats = ["Good", "bad"];
datain.(1) = setcats(datain.(1),validcats);
dataout = table2cell(datain);
end

[the cyclist]

Responding to your email pointing me to this question.

Sorry, I don't have experience with the Deep Network Designer.

Which line of code gives the error?

[Rayan Matlob]

@the cyclist, the code is fine.

i got the error on the deep network designer console

deepnetworkdesigner(resnet50)

after adding the training and validation datastore, when i press the train i got the above error.

[KSSV]

Error says, your target should be of class categorical. Check the class of your targets.

[Rayan Matlob]

@KSSV, this is my label: please how to make it categorical?

[Ben]

I think your responses are already categorical - e.g. Deep Network Designer is showing that in your first screenshot.

Have you added a second input layer to resnet50 to consume the 2nd image input? The default resnet50 has only one imageInputLayer, you will need a 2nd imageInputLayer to consume the second image input, and this will have to be connected into the network appropriately.

[Rayan Matlob]

@Ben, this worked as you said , but why this giving accuracy less than 30% and a message (Warning: Input datastore is not shuffleable but trainingOptions specified shuffling. Training will proceed

without shuffling.)

Note: when i worked on the separated data i gor more than 80% for each. thanks

[Ben]

Shuffling is easy to explain - tabularTextDatastore isn't capable to be shuffled (see the isShuffleable(textDatastore) function). We show a warning to let you know that training won't try to shuffle the data, because it can't.

This could be an issue if your training data is organised such that each minibatch in training doesn't get a good proportion of both "Good" and "bad" labels. E.g. if your data is first all the "Good" images, then all the "bad".

It might be preferable to simply read the text in train_ok.txt into memory as a string (e.g. with fileread or readlines), convert it to categorical and use an arrayDatastore.

R.e. the network: Are you concatenating the images on the channel dimension (3)? I'm not sure how the pre-trained resnet50 could work with that since its weights assume the input image has exactly 3 channels. If you want to run the 2nd image input through a separate ResNet, you could make a 2nd copy of the ResNet layers for the 2nd image input, and concatenate just before the last fullyConnectedLayer.

[Rayan Matlob]

@Ben, Dear Sir, every thing you said is just correct, even the concatenation is perfectly worked. The error is disappeared.

one last thing you also mentioned and was correct which is the data, it is first all the "Good" images, then all the "bad", and in order to make it random i used

imdsTrain_1 = shuffle(imdsTrain_1);
imdsTrain_2 = shuffle(imdsTrain_2);

The problem now is the order of both (imdsTrain_1 and imdsTrain_2) is not the same! , and the same for (imdsValidation_1 and imdsValidation_2). how to make the shuffle reorder the images of both (imdsTrain_1 and imdsTrain_2) in the same manner?

[Ben]

You can do cds = combine(imdsTrain_1,imdsTrain_2) and call shuffle(cds). You will want to also combine an arrayDatastore (or other Shuffleable datastore) containing the labels to use this for training.