Training a non-linear function using deep learning (layers and options)

I have a code that was mainly taken from mathworks question and I wanted to understand the code thoroughly.

the code is given at the end of the question.

this was an answer to the question

How can I train a deep neural network for the approximation of a nonlinear function in 1D?

https://www.mathworks.com/matlabcentral/answers/627883-how-can-i-train-a-deep-neural-network-for-the-approximation-of-a-nonlinear-function-in-1d#answer_525853

In code given at the bottom, how do we control the hidden layers and number of neurons?

Also what is the difference between

fullyConnectedLayer(8, "Name", "myFullyConnectedLayer1")

&

fullyConnectedLayer(1, "Name", "myFullyConnectedLayer2")

what does the 8 and 1 signify?

Also in the featureInput layer what does 1 signify in the

 featureInputLayer(1, "Name", "myFeatureInputLayer", 'Normalization','rescale-symmetric')

%% Define function to be approximated
fnc = @(x) x.^3;

%% Define the training data
XTrain = linspace(-1,1,80)';
YTrain = fnc(XTrain);

%% Define a layer architecture
layers = [ ...
    featureInputLayer(1, "Name", "myFeatureInputLayer", 'Normalization','rescale-symmetric')
    fullyConnectedLayer(8, "Name", "myFullyConnectedLayer1")
    tanhLayer("Name", "myTanhLayer")
    fullyConnectedLayer(1, "Name", "myFullyConnectedLayer2")
    regressionLayer("Name", "myRegressionLayer")
    ];

%% Define options for the training
opts = trainingOptions('adam', ...
    'MaxEpochs',1000, ...
    'InitialLearnRate',0.01,...
    'Shuffle','every-epoch', ...
    'Plots','training-progress', ...
    'MiniBatchSize',128, ...
    'Verbose',false);

%% Train the network
[trainedNet, info] = trainNetwork(XTrain, YTrain, layers, opts);

%% Create some sample test data
numRand = 100;
XTest = sort(2.*rand(numRand, 1) - 1);
YTest = predict(trainedNet, XTest);

%% Compare the expected and the predicted solution of the nonlinear function
plot(XTrain, YTrain, '-sblack', XTest, YTest, '-vr');
legend('exact', 'predicted')
grid on;
xlabel('x')
ylabel('f(x) = x^3')

Answer 1

[Shantanu Dixit]

Hi Fahad,

• The first input argument in fullyConnectedLayer is the outputSize, i.e , the number of neurons in the next layer. So, fullyConnectedLayer(8, "Name", "myFullyConnectedLayer1") means 8 neurons in the next layer (all fully connected). Similarly for fullyConnectedLayer(1, "Name","myFullyConnectedLayer2") which is the last layer in this neural network (since th function to be approximated is 1-Dimensional in nature).
• featureInputLayer() takes in the numFeatures as the first input argument, as we are approximating a function which takes an 1-dimensional input, numFeatures = 1 here.

Code workflow (Attached code below with relevant comments):

• The initial function is defined using an anonymous function fnc = @(x) x.^3.
• 80 data points in the range (-1,1) are generated from the function, where XTrain are the input data points and YTrain are the output values of the function.
• A neural network is designed with one input layer, one hidden layer with 8 fully connected neurons and a tanh activation function, one output layer with one fully connected neuron, and a regression layer specified for the output.
• The trainingOptions function is used to specify the training parameters, such as the optimizer (adam), maximum number of epochs, learning rate, shuffle interval, mini-batch size, plots, and verbosity.
• The trainNetwork function is used to train the neural network on the training data using the defined layers and opts.
• To test the accuracy of the trained network, 100 random test data points are generated and passed through the trained network using predict function and the output is stored in the variable YTest. A plot is then generated to compare the predicted and actual function outputs.

%% Define function to be approximated --> fnc = x^3
fnc = @(x) x.^3;

%% Define the training data
XTrain = linspace(-1,1,80)';               % Generating 80 datapoints (uniformly spaced x values between -1,+1)
YTrain = fnc(XTrain);

%% Defining a layer architecture

% The neural network will have four layers:
% feature input layer, fully connected hidden layer, output layer, and regression layer
layers = [ ...
    featureInputLayer(1, "Name", "myFeatureInputLayer", 'Normalization','rescale-symmetric') % Input layer
    fullyConnectedLayer(8, "Name", "myFullyConnectedLayer1") % 8 neurons, fully connected
    tanhLayer("Name", "myTanhLayer") % Activation function: Hyperbolic tangent
    fullyConnectedLayer(1, "Name", "myFullyConnectedLayer2") % Output layer (1 neuron)
    regressionLayer("Name", "myRegressionLayer") % Regression layer for output layer
    ];

% Training options for the network / Hyperparameters for training the
% neural network

opts = trainingOptions('adam',             % Optimizer = ADAM
    'MaxEpochs',1000,                      % Maximum number of epochs
    'InitialLearnRate',0.01,               % Learning rate
    'Shuffle','every-epoch',               % Shuffle data every epoch
    'Plots','training-progress',           % Display training progress
    'MiniBatchSize',128,                   % Batch size
    'Verbose',false);                      % Disable command window output

    
% Train the network / trains the network on the training set which is XTrain and YTrain 
[trainedNet, info] = trainNetwork(XTrain, YTrain, layers, opts);


%% Create some sample test data

numRand = 100;                           % number of points to generate randomly
XTest = sort(2.*rand(numRand, 1) - 1);   % Generate a column vector of test points between -1,+1
YTest = predict(trainedNet, XTest);      % Pass the test points through the trained network and get the predicted outputs

%% Compare the expected and the predicted solution of the nonlinear function
plot(XTrain, YTrain, '-sblack', XTest, YTest, '-vr');
legend('exact', 'predicted')
grid on;
xlabel('x')
ylabel('f(x) = x^3')

Useful documentations:

Feature input layer - MATLAB - MathWorks India

Fully connected layer - MATLAB - MathWorks India

Options for training deep learning neural network - MATLAB trainingOptions - MathWorks India