## Define Custom Regression Output Layer

**Tip**

Custom output layers are not recommended, use the `trainnet`

function and specify a custom loss function instead. To specify a custom backward function
for the loss function, use a `deep.DifferentiableFunction`

object. For more
information, see Define Custom Deep Learning Operations.

To train a neural network using mean absolute error loss, use the
`trainnet`

function and specify the loss function
`"mae"`

.

If you want to use a different loss function for your regression problems when you use the
`trainNetwork`

function, then you can define a custom regression
output layer using this example as a guide. This example shows how to create a custom
regression output layer with the mean absolute error (MAE) loss.

To define a custom regression output layer, you can use the template provided in this example, which takes you through the following steps:

Name the layer – Give the layer a name so it can be used in MATLAB

^{®}.Declare the layer properties – Specify the properties of the layer.

Create a constructor function (optional) – Specify how to construct the layer and initialize its properties. If you do not specify a constructor function, then the software initializes the properties with

`''`

at creation.Create a forward loss function – Specify the loss between the predictions and the training targets.

Create a backward loss function (optional) – Specify the derivative of the loss with respect to the predictions. If you do not specify a backward loss function, then the forward loss function must support

`dlarray`

objects.

A regression MAE layer computes the mean absolute error loss
for regression problems. MAE loss is an error measure between two continuous random variables.
For predictions *Y* and training targets *T*, the MAE loss
between *Y* and *T* is given by

$$L=\frac{1}{N}{\displaystyle \sum}_{n=1}^{N}\left(\frac{1}{R}{\displaystyle \sum}_{i=1}^{R}\left|{Y}_{ni}-{T}_{ni}\right|\right),$$

where *N* is the number of observations and
*R* is the number of responses.

### Regression Output Layer Template

Copy the regression output layer template into a new file in MATLAB. This template outlines the structure of a regression output layer and includes the functions that define the layer behavior.

classdef myRegressionLayer < nnet.layer.RegressionLayer % ... % & nnet.layer.Acceleratable % (Optional) properties % (Optional) Layer properties. % Layer properties go here. end methods function layer = myRegressionLayer() % (Optional) Create a myRegressionLayer. % Layer constructor function goes here. end function loss = forwardLoss(layer,Y,T) % Return the loss between the predictions Y and the training % targets T. % % Inputs: % layer - Output layer % Y – Predictions made by network % T – Training targets % % Output: % loss - Loss between Y and T % Layer forward loss function goes here. end function dLdY = backwardLoss(layer,Y,T) % (Optional) Backward propagate the derivative of the loss % function. % % Inputs: % layer - Output layer % Y – Predictions made by network % T – Training targets % % Output: % dLdY - Derivative of the loss with respect to the % predictions Y % Layer backward loss function goes here. end end end

### Name the Layer and Specify Superclasses

First, give the layer a name. In the first line of the class file, replace the
existing name `myRegressionLayer`

with
`maeRegressionLayer`

. Because the layer supports acceleration, also
include the `nnet.layer.Acceleratable`

class. For more information
about custom layer acceleration, see Custom Layer Function Acceleration.

classdef maeRegressionLayer < nnet.layer.RegressionLayer ... & nnet.layer.Acceleratable ... end

Next, rename the `myRegressionLayer`

constructor function (the first
function in the `methods`

section) so that it has the same name as the
layer.

methods function layer = maeRegressionLayer() ... end ... end

#### Save the Layer

Save the layer class file in a new file named
`maeRegressionLayer.m`

. The file name must match the layer
name. To use the layer, you must save the file in the current folder or in a folder
on the MATLAB path.

### Declare Layer Properties

Declare the layer properties in the `properties`

section.

By default, custom output layers have the following properties:

`Name`

— Layer name, specified as a character vector or string scalar. For`Layer`

array input, the`trainnet`

and`dlnetwork`

functions automatically assign names to layers with the name`""`

.`Description`

— One-line description of the layer, specified as a character vector or a string scalar. This description appears when the layer is displayed in a`Layer`

array. If you do not specify a layer description, then the software displays`"Classification Output"`

or`"Regression Output"`

.`Type`

— Type of the layer, specified as a character vector or a string scalar. The value of`Type`

appears when the layer is displayed in a`Layer`

array. If you do not specify a layer type, then the software displays the layer class name.

Custom classification layers also have the following property:

`Classes`

— Classes of the output layer, specified as a categorical vector, string array, cell array of character vectors, or`"auto"`

. If`Classes`

is`"auto"`

, then the software automatically sets the classes at training time. If you specify the string array or cell array of character vectors`str`

, then the software sets the classes of the output layer to`categorical(str,str)`

.

Custom regression layers also have the following property:

`ResponseNames`

— Names of the responses, specified a cell array of character vectors or a string array. At training time, the software automatically sets the response names according to the training data. The default is`{}`

.

If the layer has no other properties, then you can omit the `properties`

section.

The layer does not require any additional properties, so you can remove the
`properties`

section.

### Create Constructor Function

Create the function that constructs the layer and initializes the layer properties. Specify any variables required to create the layer as inputs to the constructor function.

To initialize the `Name`

property at creation, specify the input
argument `name`

. Add a comment to the top of the function that explains
the syntax of the function.

function layer = maeRegressionLayer(name) % layer = maeRegressionLayer(name) creates a % mean-absolute-error regression layer and specifies the layer % name. ... end

#### Initialize Layer Properties

Replace the comment `% Layer constructor function goes here`

with
code that initializes the layer properties.

Give the layer a one-line description by setting the
`Description`

property of the layer. Set the
`Name`

property to the input argument `name`

.
Set the description to describe the type of layer and its size.

```
function layer = maeRegressionLayer(name)
% layer = maeRegressionLayer(name) creates a
% mean-absolute-error regression layer and specifies the layer
% name.
% Set layer name.
layer.Name = name;
% Set layer description.
layer.Description = 'Mean absolute error';
end
```

### Create Forward Loss Function

Create a function named `forwardLoss`

that returns the MAE loss
between the predictions made by the network and the training targets. The syntax for
`forwardLoss`

is ```
loss = forwardLoss(layer, Y,
T)
```

, where `Y`

is the output of the previous layer and
`T`

contains the training targets.

For regression problems, the dimensions of `T`

also depend on the type of
problem.

Regression Task | Example | |
---|---|---|

Shape | Data Format | |

2-D image regression | 1-by-1-by-R-by-N, where
R is the number of responses and
N is the number of observations | `"SSCB"` |

2-D Image-to-image regression | h-by-w-by-c-by-N,
where h, w, and
c are the height, width, and number of channels
of the output, respectively, and N is the number of
observations | `"SSCB"` |

3-D image regression | 1-by-1-by-1-by-R-by-N, where
R is the number of responses and
N is the number of observations | `"SSSCB"` |

3-D Image-to-image regression | h-by-w-by-d-by-c-by-N,
where h, w, d,
and c are the height, width, depth, and number of
channels of the output, respectively, and N is the
number of observations | `"SSSCB"` |

Sequence-to-one regression | R-by-N, where
R is the number of responses and
N is the number of observations | `"CB"` |

Sequence-to-sequence regression | R-by-S-by-N,
where R is the number of responses,
N is the number of observations, and
S is the sequence length | `"CBT"` |

For example, if the network defines an image regression network with one response and has
mini-batches of size 50, then `T`

is a 4-D array of size
1-by-1-by-1-by-50.

The size of `Y`

depends on the output of the previous layer. To ensure
that `Y`

is the same size as `T`

, you must include a layer
that outputs the correct size before the output layer. For example, for image regression
with *R* responses, to ensure that `Y`

is a 4-D array of
the correct size, you can include a fully connected layer of size *R*
before the output layer.

A regression MAE layer computes the mean absolute error loss
for regression problems. MAE loss is an error measure between two continuous random variables.
For predictions *Y* and training targets *T*, the MAE loss
between *Y* and *T* is given by

$$L=\frac{1}{N}{\displaystyle \sum}_{n=1}^{N}\left(\frac{1}{R}{\displaystyle \sum}_{i=1}^{R}\left|{Y}_{ni}-{T}_{ni}\right|\right),$$

where *N* is the number of observations and
*R* is the number of responses.

The inputs `Y`

and `T`

correspond to
*Y* and *T* in the equation, respectively. The
output `loss`

corresponds to *L*. To ensure that
`loss`

is scalar, output the mean loss over the mini-batch. Add a
comment to the top of the function that explains the syntaxes of the function.

```
function loss = forwardLoss(layer, Y, T)
% loss = forwardLoss(layer, Y, T) returns the MAE loss between
% the predictions Y and the training targets T.
% Calculate MAE.
R = size(Y,3);
meanAbsoluteError = sum(abs(Y-T),3)/R;
% Take mean over mini-batch.
N = size(Y,4);
loss = sum(meanAbsoluteError)/N;
end
```

Because the `forwardLoss`

function only uses functions that support `dlarray`

objects, defining the `backwardLoss`

function is optional. For a list of functions that support `dlarray`

objects, see List of Functions with dlarray Support.

### Completed Layer

View the completed regression output layer class file.

classdef maeRegressionLayer < nnet.layer.RegressionLayer ... & nnet.layer.Acceleratable % Example custom regression layer with mean-absolute-error loss. methods function layer = maeRegressionLayer(name) % layer = maeRegressionLayer(name) creates a % mean-absolute-error regression layer and specifies the layer % name. % Set layer name. layer.Name = name; % Set layer description. layer.Description = 'Mean absolute error'; end function loss = forwardLoss(layer, Y, T) % loss = forwardLoss(layer, Y, T) returns the MAE loss between % the predictions Y and the training targets T. % Calculate MAE. R = size(Y,3); meanAbsoluteError = sum(abs(Y-T),3)/R; % Take mean over mini-batch. N = size(Y,4); loss = sum(meanAbsoluteError)/N; end end end

### GPU Compatibility

If the layer forward functions fully support `dlarray`

objects, then the layer is GPU compatible. Otherwise, to be GPU compatible, the layer functions must support inputs and return outputs of type `gpuArray`

(Parallel Computing Toolbox).

Many MATLAB built-in functions support `gpuArray`

(Parallel Computing Toolbox) and `dlarray`

input arguments. For a list of functions that support `dlarray`

objects, see List of Functions with dlarray Support. For a list of functions that execute on a GPU, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox). To use a GPU for deep learning, you must also have a supported GPU device. For information on supported devices, see GPU Computing Requirements (Parallel Computing Toolbox). For more information on working with GPUs in MATLAB, see GPU Computing in MATLAB (Parallel Computing Toolbox).

The MATLAB functions used in `forwardLoss`

in
`maeRegressionLayer`

all support `dlarray`

objects, so the layer is GPU compatible.

### Check Output Layer Validity

Check the layer validity of the custom classification output
layer `maeRegressionLayer`

.

Create an instance of the layer `maeRegressionLayer`

,
attached to this example as a supporting file.

`layer = maeRegressionLayer('mae');`

Check the layer is valid using `checkLayer`

. Specify the
valid input size to be the size of a single observation of typical input to the
layer. The layer expects a
1-by-1-by-*R*-by-*N* array inputs, where
*R* is the number of responses, and *N* is
the number of observations in the mini-batch.

```
validInputSize = [1 1 10];
checkLayer(layer,validInputSize,'ObservationDimension',4);
```

Skipping GPU tests. No compatible GPU device found. Skipping code generation compatibility tests. To check validity of the layer for code generation, specify the CheckCodegenCompatibility and ObservationDimension options. Running nnet.checklayer.TestOutputLayerWithoutBackward ........ Done nnet.checklayer.TestOutputLayerWithoutBackward __________ Test Summary: 8 Passed, 0 Failed, 0 Incomplete, 2 Skipped. Time elapsed: 0.17995 seconds.

The test summary reports the number of passed, failed, incomplete, and skipped tests.

### Include Custom Regression Output Layer in Network

You can use a custom output layer in the same way as any other output layer in Deep Learning Toolbox. This section shows how to create and train a network for regression using the custom output layer you created earlier.

The example constructs a convolutional neural network architecture, trains a network, and uses the trained network to predict angles of rotated, handwritten digits. These predictions are useful for optical character recognition.

Load the example training data.

[XTrain,~,TTrain] = digitTrain4DArrayData;

Create a layer array including the regression output layer
`maeRegressionLayer`

.

```
layers = [
imageInputLayer([28 28 1])
convolution2dLayer(5,20)
batchNormalizationLayer
reluLayer
fullyConnectedLayer(1)
maeRegressionLayer('mae')]
```

layers = 6x1 Layer array with layers: 1 '' Image Input 28x28x1 images with 'zerocenter' normalization 2 '' 2-D Convolution 20 5x5 convolutions with stride [1 1] and padding [0 0 0 0] 3 '' Batch Normalization Batch normalization 4 '' ReLU ReLU 5 '' Fully Connected 1 fully connected layer 6 'mae' Regression Output Mean absolute error

Set the training options and train the network.

```
options = trainingOptions('sgdm');
net = trainNetwork(XTrain,TTrain,layers,options);
```

Training on single CPU. Initializing input data normalization. |========================================================================================| | Epoch | Iteration | Time Elapsed | Mini-batch | Mini-batch | Base Learning | | | | (hh:mm:ss) | RMSE | Loss | Rate | |========================================================================================| | 1 | 1 | 00:00:00 | 28.28 | 25.1 | 0.0100 | | 2 | 50 | 00:00:04 | 14.27 | 11.3 | 0.0100 | | 3 | 100 | 00:00:07 | 14.90 | 11.9 | 0.0100 | | 4 | 150 | 00:00:11 | 10.21 | 8.0 | 0.0100 | | 6 | 200 | 00:00:15 | 10.15 | 7.9 | 0.0100 | | 7 | 250 | 00:00:19 | 11.57 | 9.0 | 0.0100 | | 8 | 300 | 00:00:23 | 10.86 | 8.4 | 0.0100 | | 9 | 350 | 00:00:27 | 9.94 | 7.7 | 0.0100 | | 11 | 400 | 00:00:31 | 9.97 | 7.2 | 0.0100 | | 12 | 450 | 00:00:36 | 8.88 | 6.7 | 0.0100 | | 13 | 500 | 00:00:40 | 9.39 | 5.9 | 0.0100 | | 15 | 550 | 00:00:44 | 8.73 | 6.2 | 0.0100 | | 16 | 600 | 00:00:48 | 8.95 | 6.6 | 0.0100 | | 17 | 650 | 00:00:52 | 8.01 | 5.7 | 0.0100 | | 18 | 700 | 00:00:56 | 8.35 | 6.2 | 0.0100 | | 20 | 750 | 00:01:00 | 7.13 | 5.6 | 0.0100 | | 21 | 800 | 00:01:05 | 7.50 | 5.5 | 0.0100 | | 22 | 850 | 00:01:09 | 7.11 | 5.6 | 0.0100 | | 24 | 900 | 00:01:12 | 7.43 | 5.4 | 0.0100 | | 25 | 950 | 00:01:17 | 6.66 | 4.9 | 0.0100 | | 26 | 1000 | 00:01:21 | 6.82 | 4.8 | 0.0100 | | 27 | 1050 | 00:01:25 | 6.65 | 5.1 | 0.0100 | | 29 | 1100 | 00:01:29 | 7.39 | 5.9 | 0.0100 | | 30 | 1150 | 00:01:33 | 7.10 | 5.4 | 0.0100 | | 30 | 1170 | 00:01:35 | 6.64 | 5.0 | 0.0100 | |========================================================================================| Training finished: Max epochs completed.

Evaluate the network performance by calculating the prediction error between the predicted and actual angles of rotation.

[XTest,~,TTest] = digitTest4DArrayData; YPred = predict(net,XTest); predictionError = TTest - YPred;

Calculate the number of predictions within an acceptable error margin from the true angles. Set the threshold to be 10 degrees and calculate the percentage of predictions within this threshold.

thr = 10; numCorrect = sum(abs(predictionError) < thr); numTestImages = size(XTest,4); accuracy = numCorrect/numTestImages

accuracy = 0.7622

## See Also

`trainnet`

| `trainingOptions`

| `dlnetwork`

| `checkLayer`

| `findPlaceholderLayers`

| `replaceLayer`

| `PlaceholderLayer`