semanticseg producing marginally different values when inference is repeated

Good afternoon,

I've been using semanticseg to apply a previously trained DAG network model to segment a test set of images. I've noticed that when repeatedly applying the same model to exactly the same image (with the same random seed, unless I am somehow setting this improperly) the output segmentation can differ very slightly. When using [~,~,allScores] = semanticseg(image, network) to view the softmax probability values output by the network per-pixel and per-class, these values do seem to be able to change each time the network is applied. The difference is very small, the values I've inspected seem to be consistent to at least 5 decimal places between segmentations, but I'm curious to know where this small difference is produced since I expected the inference procedure to be entirely deterministic. Could the use of the parallel computing toolbox influence the values in this way?

Thank you for any help you can provide; I hope I haven't overlooked an obvious answer. Fortunately, this small difference in probability score is only pronounced enough to shift the final pixel classification very rarely, on the order of magnitude of something like 1/1,000,000,000 pixels, and so only in cases where the output class probabilities are practically tied to begin with. I've included one such rare example below, in which the maximum-likelihood class for the pixel shifts in response to this variation.

Considering 9 classes, note the first and the last class:

Run 1:

[0.499402880668640, 8.857092470861971e-04, 9.613345497427872e-08, 1.140553695933022e-08, 3.467669529300110e-08, 2.951414890262072e-09, 6.419115834432887e-07, 3.072153194807470e-04, 0.499403357505798]

Run 2:

[0.499403357505798, 8.857083739712834e-04, 9.613336970915043e-08, 1.140553695933022e-08, 3.467669529300110e-08, 2.951414890262072e-09, 6.419103897314926e-07, 3.072150284424424e-04, 0.499402880668640]

Thanks again!

[awezmm]

Good morning Topological Sphere,

How are you setting the random seed?

best

-awezmm

[TopologicalSphere]

I set the random seed by placing rng(aNumber) at the top of the code where, for example, aNumber could be 45929 or 0. I've tried clearing all variables from the workspace and restarting the entire matlab program between segmentation runs but this does not resolve the issue.

I believe I've managed to recreate the issue using the camvid/deeplab demo found at https://www.mathworks.com/help/vision/examples/semantic-segmentation-using-deep-learning.html so that we can compare results.

The process which produces the error is as follows:

First, I run the demo code to download the camvid dataset (though it skips this step on repeated runs) and the pretrained network (so that I do not train a different one at any point which would obviously influence the segmentations). To fully mimic the process I use in my own code, I save the network as a .mat file from which it will be loaded in future segmentation procedures, overwriting the one pulled from the demo in case that were ever pulled a second time.

The demo code also splits the data into training/validation/testing sets using rng(0). This does seem to produce the same split every time, as intended. I stop the demo from running after it checks to see whether or not it should train the network, after the if block following doTraining = false. We will be using the testing set to produce the segmentations.

Now I mimic my segmentation procedure using the following code copied from the demo, though with minibatch size set to 1, a custom out directory, and with a random seed added for consistency (however my impression of DeepLabV3+ was that it didn't have any layers which would induce randomness at inference to be affected by a seed in the first place, though this may be completely mistaken. In either case, a seed is specified). This code is run in a totally seperate script after the demo has created the training/testing split and after loading the saved network from the .mat file.

rng(45929)

outDir = './LocalOut1';

pxdsResults = semanticseg(imdsTest,net, ...
    'MiniBatchSize',1, ...
    'WriteLocation',outDir, ...
    'Verbose',false);

Now I close matlab. I re-launch matlab, run the demo so it creates the training/testing split, load the saved network from the .mat file, change outDir to './LocalOut2' and run the above code. Then I loop through both directories './LocalOut1' and './LocalOut2' and determine whether or not their images are the same using the code below

dir1 = './LocalOut1/';
dir2 = './LocalOut2/';

dir1Info = dir(strcat(dir1,"*.png"));
dir2Info = dir(strcat(dir2,"*.png"));
for i = 1:numel(dir1Info)
    dir1Img = strcat(dir1Info(i).folder,'/',dir1Info(i).name);   
    dir2Img = strcat(dir2Info(i).folder,'/',dir2Info(i).name); 
    
    if ~isequal(imread(dir1Img),imread(dir2Img))
        disp(i)
        disp('inequality')
        segA = imread(dir1Img);
        segB = imread(dir2Img);
        return
    end
end

It returned that the 128th segmentations were different. When I checked this with the below code, they differed by one pixel

%segA is the imread first segmentation, as above
%segB is the imread second segmentation, as above
diffMat = zeros(size(segA),'uint8');
diffMat(segA ~= segB) = 1;
disp(sum(diffMat,'all'))

The linear index of the pixel was 481335. In segA it has value 2 and in segB it has value 8.

I repeated the process exactly as stated above again, and strangely enough there were no differences in any images. I repeated it a third time and a fourth time to discover no differences, but on the fifth time segmentation 128 was different again, at the same index.

Rerunning this experiment (with the closing and opening of matlab between segmentation procedures to completely remove any possibility of workspace variable holdover) but saving and comparing the softmax probability outputs rather than the segmentation class assignments required the following modifications to the code, and showed outputs that varied much, much more consistently, since the differences only occur in the integer segmentations when softmax probabilities between classes are close enough that subtle shifts could influence the final result. Basically, all of the softmax outputs I checked are mildly different.

Pixel 481335 of image 128 fits this description, with a softmax output of approximately 0.3363 for both classes 2 and 8.

%Produce probabilities and save them in a .mat file
rng(45929)

outDir = './LocalOut1';

for i = 1:numel(imdsTest.Files)
    [~,~,allScores] = semanticseg(imread(imdsTest.Files{i}),net);
    save(strcat(outDir,"/Seg",num2str(i),".mat"),'allScores'); 
end

And for the comparison:

dir1 = './LocalOut1/';
dir2 = './LocalOut2/';

dir1Info = dir(strcat(dir1,"*.mat"));
dir2Info = dir(strcat(dir2,"*.mat"));
for i = 1:numel(dir1Info)
    dir1Mat = strcat(dir1Info(i).folder,'/',dir1Info(i).name);   
    dir2Mat = strcat(dir2Info(i).folder,'/',dir2Info(i).name); 
    
    dir1Mat = load(dir1Mat,'allScores');
    allScores1 = dir1Mat.allScores;
    dir2Mat = load(dir2Mat,'allScores');
    allScores2 = dir2Mat.allScores;
    
    if ~isequal(allScores1,allScores2)
        disp(i)
        disp('inequality!')
        return
    else
        disp(i)
    end
end

Apologies for the delay. Hopefully I'm doing something silly with the random number generator which can be easily corrected, or have a misunderstanding about either one of the layers in the deeplab architecture or the way in which the network objects perform computations at inference.

Answer 1

[Mahesh Taparia]

Hi

The slight difference in probability may be because of incorrect use of random seed or you are using dropout layer in the network which is selecting the weights randomly with the specified probability and resulting in minor difference.

[TopologicalSphere]

Thank you for your response!

The network does not include any dropout layers and I don't think I'm using the random seed incorrectly, though this may be possible. Is there something I ought to include in the script aside from placing rng(aSeed) at the top, where aSeed is a constant integer?

[Mahesh Taparia]

Hi

Can you upload your code file once?

[TopologicalSphere]

Please see the above response to awezmm's comment. Apologies for the delay.

Answer 2

[Greg Heath]

Use

clear all, close all, clc, rng(0)

on the 1st line

Thank you for formally accepting my answer

Greg

[TopologicalSphere]

Please see the above response to awezmm's comment. Apologies for the delay.