What is a good validation RMSE value?

9 views (last 30 days)
Manny
Manny on 9 Dec 2024
Commented: Manny on 9 Dec 2024
Hi everyone
I am new to MATLAB and modelling. I have a LSTM deep learning model that I am experimenting with. I need some help in determining if the model is a good fit for my data. I am not sure where to start so I will start with validation RMSE. If there are other metrics to look at please let me know.
I have three versions of the model. There are close to 3000 data points used in the model. 80% is used for training. 20% is used for testing/validation.
Model 1 22 sec, validation RMSE = 0.35918
Model 2 34 sec, validation RMSE = 0.065824
Model 3 41 sec, validation RMSE = 0.50482
My elementary knowledge of modelling tells me that the lower the RMSE the better. So, if I use this approach then Model 2 is the winner. Is this enough? Should I be looking at other ways to assess quality of fit?
Here is some basic information about the data set:
Average 17.45502778
Min 8.8901784
Max 74.563979
Standard Deviation (Sample) 6.500568667
Standard Deviation (Population) 6.499343878
Thank you

Answers (1)

Jacob Mathew
Jacob Mathew on 9 Dec 2024
Hi Manny,
Assuming you have calculated the Root Mean Square Error on the original data itself, you can consider calculating the Normalised RMSE which is denoted using the formula:
where,
  • n is the number of observations.
  • are the observed values.
  • are the predicted values.
  • and are the maximum and minimum observed values, respectively
This effectively calculates the RMSE accounting for the variability in your data. In your case, this turns out to be:
  • 0.005469 for Model 1
  • 0.001002 for Model 2
  • 0.007686 for Model 3
All 3 values are close to zero which imply the predictions have little error against the observered value. This generally hints that the model is well fitted to the data provided the data is sampled randomly to create the test, train and validation data and appropriate data preprocessing was done.
  1 Comment
Manny
Manny on 9 Dec 2024
Thank you @Jacob Mathew. This is very helpful.
You made a comment re how the RMSE was calculated. I am not sure either myself. MATLAB produced a screen that shows the results. Here it is:
Does this address your comment?

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!