Different results accross multiple runs

Hello,

I'm facing a problem that when i run same classification algorithm multiple times, I find that the accuracy results differ. I fixed the seed value using rng function, set the learnables (weights and biases) of the network manulay using Xavier(Glorot), and I also restrictied to use only one CPU and not to use GPU. Any Help?

I've read that it's accepted to have slightly differences among multiple runs and I have to get the average and the STD of the results and use them as the final score of my algorithm is it true if so please give me a refereance for that. Thanks in advance.

Answer 1

[Joss Knight]

It might help to follow some of the suggestions here, even if you are not using a GPU:

https://uk.mathworks.com/help/deeplearning/ug/reproduce-gpu-network-training.html

You should be able to get deterministic results for everything by controlling the rng seed as long as your execution environment is not changing (e.g. a laptop is throttling, memory usage is changing due to execution of other applications and so forth).

[Ahmed]

Thanks, for your help. I'll check out the link. I already use the same PC and not changing any thing in the algorithm and I can see that loss differs slightly in first epochs of the embeddings network and unfortunately the difference increases. I also use a second classifier that takes these embeddings and get different in accuracy about 4% maximum.

I checked the svm classifier and it gives same results when training data is the same. So the problem is in the feature embeddings network.

I wonder if the ram usage differs across different runs, the results changes?!

[Joss Knight]

It's hard to intuit how, if everything is on the CPU. Are you sure you are running the same code, resetting the rng before you do anything else, and re-creating all the networks and datastores? Also, you cannot use background preprocessing for your data.

[Joss Knight]

Hi! I see you posted some code but then deleted it. Hopefully this is because you worked out how to get reproducible results. If not, let me know and I can look into it further.