Deep Neural LSTM Network Issues

I am training a Deep Neural Net with a regression layer in the end. I have 20 inputs and a sequence of output with 10 steps. I tried using both LSTM and BiLISTM layers with something like 100 to 200 hidden units. I have also included two fully connected layers with some 100 hidden nodes or so. But no matter what I do and change learning rate etc.. the progress comes to a plateau very quickly like in a couple of epochs and remains so for the rest of the training. Tried changing learning rate, min batch size, Number of hidden units, adding relu and not adding relu etc etc.. you name it. But I can not improve the accracy to more than about 0.9 (validation MSE). Is there anything else I can try to improve?

Answer 1

[Udit06]

Hi Pappu,

I understand that you aim to improve your validation set accuracy of the LSTM based deep neural network. You can try out the following approaches to improve your accuracy metric:

1) You can normalize your input features in the range [0, 1] to stabilize the training process and improve convergence.

2) You can use dropouts in your network to prevent overfitting and get a generalized model. You can refer to below MathWorks documentation to understand more about dropouts.

https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.dropoutlayer.html

3) Instead of using a fixed learning rate, you can use Adam optimizer which utilizes adaptive learning rate based on the gradient magnitude. You can refer to below MathWorks documentation to understand more about Adam optimizer.

https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.trainingoptionsadam.html

I hope this helps.

[Pappu Murthy]

Tried those ideas that you have suggested already but not much help at all.