Simbiology data fitting program is constantly running nonstop

Hey guys,

Does anyone have experience with fitting the data in Simbiology but the program just running for more than 30 minutes? I am experienceing this recently with Matlab R2020b. I am trying to fitting biodistribution data into my 15 compartments PBPK model. My model worked without error and warning when I first estimated 6 parameters I am interested in with simulation program build in the Simbiology app. After I got similar simulation to my animal data. I switched to data fitting program build in the Simbiology app. Here came the problem. Everytime when I run the data fitting, the figure of "Progress Plot for Isqnonlin" did pop up and ran correctly with the porgress showed in the figure and showed the End time and Elapsed time after the run. However, the program just keep running nonstop for more than 30 minute. I took so long that I can only force quite my matlab. There's no result and summery come up after waiting for 30 minute. All the setings for the fitting I used are defult setup in the program. How could I deal with this problem?

Answer 1

[Sietse Braakman]

Hi Jesse,

It is not uncommon for estimation programs to take that long. The duration of the fitting process depends on many factors:

• The complexity of your model: If you have a larger model (more ODEs and algebraic equations), it will take longer to numerically solve the model. Generally, the optimization process requires the equations to be solved for each iteration (pooled fit) and/or each individual in your dataset at each iteration (unpooled fit and mixed effects estimation)
• The effective stop time for the simulation. During a fit, this would be determined by the last time point in your dataset. If you only have to simulate your model for 1 day vs 1 month, that makes a difference
• The solver settings for the ODE solver: if you tolerances are unnecessarily low, the ODE solver will take more steps than is necessary to simulate the model. You can set these tolerances in the simulation settings window. I found this documentation page helpful wrt the absolute and relative tolerance: Absolute and Relative Tolerance
• The solver settings for the optimization algorithm: if your termination criteria are too restrictive, the optimization may never terminate. Generally, you have a termination criteria for the objective function (you can glean what a fair termination criterium from the log-likelihood in the progress plot), for the parameter estimate, for the first-order optimality (a measure of the gradient at the optimal point that can also be gleaned from the progress plot, only for gradient-based optimization methods), and the maximum number of iterations (this is not a desirable termination criterion because it doesn't represent a mathematical optimum).
• The number of parameters you are estimating. The higher the number, the higher the dimensionality of your search space.
• The optimization algorithm: local optimization algorithms (lsqnonlin, fmincon, fminsearch, fminunc, nlinfit) are generally fastest, global optimization algorithms (scattersearch, patternseach, ga, particleswarm) generally require many more iterations due to their mathematical approach. Mixed effects algorithms, also require more iterations than local algorithms, so they may also take longer. Especially for the mixed effects algorithms, the default termination criteria tend to be conservative, so you may want loosen these criteria. We are also aware of the nlmefitsa algorithm taking a very long time to converge.
• There may be an identifiability issue where some of the parameters you are trying to estimate are not identifiable. See this webinar for more info on identifiability and sensitivity analysis for parameter estimation: https://www.rosaandco.com/webinars/2019/parameter-estimation-to-establish-confidence-model-predictions

Some of the things you could do:

• Make sure your optimization is still progressing after 30 minutes by looking at the progress plot - if so, either wait longer, use parallel computing toolbox (will only help on unpooled, mixed effects or global optimization fits) to speed up the optimization, try changing the termination criteria.
• Investigate identifiability of your parameters, e.g. using the AliasingScore app for SimBiology models
• Reduce the number of parameters you are estimating

Let me know how you get on.

Best,

Sietse

[Fulden Buyukozturk]

Hi Jesse,

In addition to what Sietse already wrote, a suggestion could be to set a reasonable value on max wall clock (via the simulation settings if using the app).

A very common reason things appear stuck during a fit is because of a very slow simulation. Even when things appear to have “converged”, a subsequent simulation could step into a “bad” set of parameter values. So setting a reasonable max wall clock might help.

Fulden

[Sietse Braakman]

Jesse, I have one additional question - how many individuals are there in your dataset? Could it be that plotting takes a long time? In that case, you can turn plotting off in the fit program and see if that solves your problem. (See screenshot)

Another thought, related to Fulden's suggestion, is that it could be that the final estimates from your fit, cause your model to be very stiff. Your ODE solver will then have to take very small steps to meet its tolerances, leading to long simulation times

[Jesse Chao]

Hi Sietse and Fulden,

Thank you for all your preasure suggestions. I have a little update about my problem. When I posted this question, I ran Matlab R2020b on my macbook pro 2016 with macOS Big Sur and I also tried to run in Parallels (vertal machine) with window 10. Both systems had the same issue which is that after their was no progress in the progressing plot, the model analysis app still showed 'program running' and no final results and summaries generated after 30 minutes waiting in the model analysis app. Thus, I asked my friend to run my model also in matlab R2020b in windows-based laptop with widow 10. It turns out that he can successfully get the fitting results with same code and setting as mine althought the fitting is not quite good. At least, I can get some summaries about my guessing.

@Sietse: First, following your suggestions, I have maked sure that there is no more progress in the progressing plot after about 3 minutes waiting, and the fit has been coverged was showen on the plot. However, the model analysis app still showed 'program running' and no final results and summaries generated after 30 minutes waiting in the model analysis app. Second, I have also try to use the Aliasing Score app. However, it seems that the data I want to analysis is a represented as a repeat assignment in my model. Thus, the app is not able to run. In addition, the majority comparments and parameters are assigned as initial assignments. I don't know how to applied all the calculated value to my model and run the Aliasing Score app. Third, I have tried to reduced my parameters or targets, but it seems that this issue still presented. Fourth, my data is just a n=1 with 5 time points from 5 different organs biodistribution data. I also tried out your suggestion to turn off the plotting, but the problem still presented.

@Fulden: This is a very good point that I have not be aware of. Just for a quick test, I set my max wall clock to 180s compared to my runnign time showed in my progressing plot in my first post is a resenable value. However, the problem didn't be solved. The fit has been coverged was showen on the plot but the model analysis app still showed 'program running' and no final results and summaries generated after 10 minutes waiting in the model analysis app.

I am just guessing is this some internal problems of the software compatibility, becasue this only happened in mac-based laptope. It seems that the Simbiology somehow stucked at the point that the fit has been coverged and couldn't execute the results and summaries for a bad fit or evan a complex model, because my model is quite huge and complicated .

What do you think about my guessing? Is there any issue about matlab compatibility with mac has been reported which had similarity to my problem?

I am looking forward to hearing from you. Thank you very much.

Answer 2

[Jesse Chao]

Just a brief update, I have successfully fit the data through writting my own script instead of using Simbiology Model Analyzer app. However, if I use Simbiology Model Analyzer to fit, the problem mentioned above is still unsolved.

[Fulden Buyukozturk]

Hi Jesse, it would be very diffucult to reproduce this issue on our end without the project file. Could you share it? If so, you can contact me directly from my profile.