What is the exact interpretation of LogLikelihood, AIC, BIC and RMSE in population pharmacokinetic model development in Simbiology.? How to compute Objective Function Value in Simbiology?
I use Simbiology for population PK-PD model development. During the model fitting of data, I understand that the model diagnostics play a major decisive role in selecting the suitable model. Hence would like to make it clear regarding the interpretation of the model diagnostics.
If for example, I have two models. First model: DFE= 411, LogLikelihood = - 807.6 (minus 807.6), AIC = 1633.2 , BIC = 1647.2 and RMSE = 1.92 Second model: DFE= 410, LogLikelihood = - 888.8 (minus 888.8), AIC = 1797.6 , BIC = 1813.2 and RMSE = 0.34 Which among the model is better and why? What are the individual interpretation of DFE, LogLikelihood, AIC, BIC and RMSE?
In PK-PD research paper generally, they take Objective Function value as decisive model diagnostics. What is the Objective Function Value in Simbiology? I did some literature search and found that Objective Function Value is -2 times LogLikelihood value? So should I multiply the LogLikelihood value given in Simbiology by - 2 to obtain Objective Function Value? Moreover, if the LogLikelihood value is multiplied with -2 then the entire interpretation will be changing (as minus will reverse the direction). So please guide in this regard and give your valuable inputs.
8 Comments
Time Descending
There is no universal criterion that people agree on for selecting a model. Log likelihood, AIC, and BIC are among the most common criteria, and that's why we provide them as outputs when fitting. I'll share a little more of what I know about these criteria.
Log likelihood has a relatively clear statistical meaning and derivation. As you might expect from the name, it is a statistical measure of how likely the estimated model parameters are given the data. However, people don't generally like to use log likelihood as a criterion to for choosing between models that have different numbers of parameters. That's because using the log likelihood alone can encourage overfitting, since you can generally improve the log likelihood by adding parameters to a model.
To overcome this, the AIC and BIC were introduced to penalize additional parameters in a model. The difference between the two is how they weight additional parameters. Wikipedia has a decent summary of the comparisons of AIC and BIC.
When you use the fit program in the SimBiology desktop or sbiofit from the command line, the objective function that is minimized is generally a transformation of the negative log likelihood that has the same minimum. ( nlinfit is the only exception to this. It uses a weighted least squares objective function that is only equivalent to the negative log likelihood in specific cases.) We use these transformations to make fitting more efficient and robust. You can find more information here about the objective functions used.
Thanks Arthur for a detailed and prompt reply. You have touched upon all aspects except for one - RMSE. What is the importance of RMSE in selection of right model?
I overlooked RMSE because I don't use it for model selection. In some cases (for example, a single constant error model), minimizing the RMSE is equivalent to minimizing the negative log likelihood (NLL). In such cases, using RMSE for model selection is equivalent to using NLL. But in the general case, I think NLL, AIC, and BIC are more useful since they take into account the error models used.
Thanks Arthur Goldspie. Things are sorted out to perfection. Classy support by MATLAB community.
Hi Paveer,
SimBiology estimates parameter by the method of maximum likelihood. Depending on the error models you use, it might convert this into a Least Square problem during the optimization but it will always return the value of the LogLikelihhood. So, the higher the LogLikelihood, the better.
Now, it is common to use -2*LogLikelihood instead to compare fits. This is, I believe, what you call 'Objective Function'. It is also common to penalize model complexity by adding a term proportional to the number of parameters to estimate. This is essentially what AIC is. BIC is similar as it also penalizes fits with a large number of data points. So here, the lower the AIC or BIC, the better.
You can find more explanations about the optimization problem that SimBiology poses for parameter estimation here: Maximum Likelihood Estimation
You can also find a description of AIC, BIC, DFE and all other measures returned here: LeastSquaresResult object
Thank you Jeremy Huard for the prompt reply. I have a query in your reply. When you are saying higher the likelihood better is the model, what are you referring to with the word higher? That is among -888.8 and -807.6, mathematically -807.6 is higher. But i think that you are referring to -888.8 (that is absolute value of -888.8, in which case 888.8 is greater than 807.6).
Please clarify this.
I mean the signed number and not its absolute value. As Arthur described, the higher the likelihood, the more likely it is to be the model to select. I'm oversimplifying it but it's good way to remember it. So, in your case if both models have the same number of parameters, then the model that leads to a LogLikelihood of -807.6 might be the one to select. If the number of parameters is different, I would suggest to consider AIC or BIC as a criterion to select the right model. Here the lower, the better. But in your case it's still model #1.
Thanks Jeremy Huard. It's clear now. Great support.
