warmupLearnRate

Warm-up learning rate schedule

Since R2024b

Description

A warm-up learning rate schedule object ramps up the learning for a specified number of iterations.

Tip

To easily use a warm-up learning rate schedule with the default options, set the LearnRateSchedule argument of the trainingOptions function to "warmup".

Creation

Description

schedule = warmupLearnRate creates a warmupLearnRate object.

This syntax is equivalent to setting the LearnRateSchedule argument of the trainingOptions function to "warmup".

schedule = warmupLearnRate(Name=Value) specifies additional properties using one or more name-value arguments. For example, InitialFactor=0.5 specifies an initial scaling factor of 0.5.

Properties

InitialFactor — Initial scaling factor

0.3 (default) | positive scalar

Initial scaling factor, specified as a positive scalar.

The software starts by scaling the base learning rate by InitialFactor and transitions toward scaling the learning rate by FinalFactor.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

FinalFactor — Final scaling factor

1 (default) | positive scalar

Final scaling factor, specified as a positive scalar.

The software starts by scaling the base learning rate by InitialFactor and transitions toward scaling the learning rate by FinalFactor.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

NumSteps — Number of warm-up steps

5 (default) | finite positive integer

Number of warm-up steps, specified as a finite positive integer.

If FrequencyUnit is "iteration", then the software updates the learning rate each iteration for NumSteps iterations. If FrequencyUnit is "epoch", then the software updates the learning rate each epoch for NumSteps epochs.

warmupLearnRate objects are finite learning rate schedules, so NumSteps is a finite positive integer.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

FrequencyUnit — Frequency unit

"iteration" (default) | "epoch"

Frequency unit, specified as "iteration" or "epoch".

If FrequencyUnit is "iteration", then the software updates the learning rate each iteration for NumSteps iterations. If FrequencyUnit is "epoch", then the software updates the learning rate each epoch for NumSteps epochs.

Examples

Create Warm-Up Learning Rate Schedule

Create a warm-up learning rate schedule with the default settings.

schedule = warmupLearnRate

schedule = 
  warmupLearnRate with properties:

    InitialFactor: 0.3000
      FinalFactor: 1
    FrequencyUnit: "iteration"
         NumSteps: 5

Specify this schedule as a training option.

options = trainingOptions("adam",LearnRateSchedule=schedule);

As an alternative to creating a warm-up learning rate schedule object, to easily use a warm-up learning rate schedule with the default options, specify the LearnRateSchedule argument of the trainingOptions function as "warmup".

options = trainingOptions("adam",LearnRateSchedule="warmup");

Customize Warm-Up Learning Rate Schedule

Create a warm-up learning rate schedule that ramps up the learning rate for 10 iterations, starting with an initial scaling factor of 0.5.

schedule = warmupLearnRate( ...
    NumSteps=10, ...
    InitialFactor=0.5)

schedule = 
  warmupLearnRate with properties:

    InitialFactor: 0.5000
      FinalFactor: 1
    FrequencyUnit: "iteration"
         NumSteps: 10

Specify this schedule as a training option.

options = trainingOptions("adam",LearnRateSchedule=schedule);

Algorithms

Warm-Up Learning Rate Schedule

A warm-up learning rate schedule object ramps up the learning for a specified number of iterations.

The software uses this formula to calculate the learning rate:

$$\alpha =\{\begin{array}{cc}\left({\gamma }_0+\frac{k-1}{N}\left({\gamma }_N-{\gamma }_0\right)\right){\alpha }_0& \text{if }k\le N\\ {\gamma }_N{\alpha }_0& \text{if }k>N\end{array},$$

where:

• α₀ is the base learning rate, specified by the InitialLearnRate option of the trainingOptions function.

• γ₀ is the initial scaling factor, specified by the InitialFactor argument.

• γ_N is the final scaling factor, specified by the FinalFactor argument.

• N is the number of steps, specified by the NumSteps argument.

If FrequencyUnit is "iteration", then the variable k denotes the iteration number. If FrequencyUnit is "epoch", then the variable k denotes the epoch number and each iteration of the epoch uses the same learning rate.

This plot shows an overview of the warm-up learning rate schedule with the default settings.

Version History

Introduced in R2024b