How can I freeze layers for training a network with multiple outputs and reduce time for the training?
15 views (last 30 days)
Show older comments
Since I try to train the model with multiple outputs, I have an issue to use 'deepNetworkDesigner' and its Training option.
So, I follow the direct training method on this example with freezing net's layers using the code below
lgraph = net.layerGraph;
target = 290
for i = 1 : target
try
L = freezeWeights(lgraph.Layers(i));
lgraph = replaceLayer(lgraph,lgraph.Layers(i).Name,L);
catch
end
end
net = dlnetwork(lgraph)
I checked WeightLearnRateFactor and BiasLearnRateFactor become zero, which means the layers are frozen.
However, it still takes too much time on the stage of training.
So,
Q1: is this the right way to freeze layers for the training multiple output network?
Q2: How can I reduce time for training, ignoring the layers which are frozen.
Here is the base code from example that I used for network training
[loss,gradients,state] = dlfeval(@modelLoss,net,X,T1,T2);
function [loss,gradients,state] = modelLoss(net,X,T1,T2)
[Y1,Y2,state] = forward(net,X,Outputs=["softmax" "fc_2"]);
lossLabels = crossentropy(Y1,T1);
lossAngles = mse(Y2,T2);
loss = lossLabels + 0.1*lossAngles;
gradients = dlgradient(loss,net.Learnables);
end
0 Comments
Accepted Answer
Aniketh
on 12 Jul 2023
Yeah this a right way to freeze the layers, as long as you are not seeing any drastic changes in the output it shouldn't be any different for the case of predicting multiple outputs.
To further reduce training time and ignore the frozen layers, you can modify the modelLoss function to only compute and backpropagate gradients for the unfrozen layers using the find function in matlab. Here's a sample code for how this will work:
function [loss, gradients, state] = modelLoss(net, X, T1, T2)
[Y1, Y2, state] = forward(net, X, Outputs = ["softmax" "fc_2"]);
lossLabels = crossentropy(Y1, T1);
lossAngles = mse(Y2, T2);
loss = lossLabels + 0.1 * lossAngles;
% Compute gradients only for unfrozen layers
unfrozenLayers = find([net.Layers.WeightLearnRateFactor] > 0);
gradients = dlgradient(loss, net.Learnables(unfrozenLayers));
end
More Answers (0)
See Also
Categories
Find more on Image Data Workflows in Help Center and File Exchange
Products
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!