MATLAB / Simulink runs slower in batch mode than interactively (Windows)

We have hundreds of Requirements (SLREQX), models (SLX, SLDD) and Test Manager files (MLDATX) and traceability between them (SLMX) and a script which are meant to scan our entire repository and perform reasonablenss checks & collect metrics about these files.

While debugging these MATLAB scripts, I experience a typical runtime of 10 minutes on a Windows Server 2019 host (details below). This duration was measured using 'tic' and 'toc' in the script.

In "production" I want the script to run headlessly via a Continuous Integration (CI) agent on the same host PC, but in this mode the script takes more than 1 hour (60 minutes). It is expected that the same tasks take significantly longer (more than 5x) in headless / batch mode? What else should I investigate to enable a run-time similar to interactive / graphical mode?

The CI agent runs as a functional/service account, but I have confirmed this account's MATLAB preferences are identical to mine (such as the Java Heap Memory preferences - set to 8 GB).

On the CI agent, the script is launched via the following method (Windows batch script):

$MATLAB2021ROOT\matlab.exe -wait -noFigureWindows -sd "$WORKSPACE" -batch "$SCRIPT_COMMAND; quit force;"

Where SCRIPT_COMMAND is (paraphrasing)

• open Simulink Project
• change directory & add additonal folder to MATLAB path
• run reasonablenss checks & collect metrics <- this is the part that is taking the longest, but is 10 minutes in graphical mode

Host PC details

• MATLAB 2021 Update 4
• Xeon processor @ 3GHz, 96 cores (2 physical CPUs)
• 192GB RAM, NVME storage
• Windows Server 2019

Thank you

[Yongjian Feng]

Hi Nathan,

This could be caused by different setup/configuration of the Host PC. To compare apple to apple, it makes sense to try the same script headless on the developement machine. This will be a better comparison.

Thanks,

[Yonnatan Bar]

Same issue here ,

When launching Matlab and running and generating a code from the APP , it takes 20-25 minutes.

When trying the same on the same computer using headless through CMD , 2 Hours .

Same branch , same pc , same apple.

[Guillaume]

Same issue here,

My Simulink Test Manager runs in 35 minutes in interactive mode but when triggered by my GitHub Action it takes 5 hours !!! On the same machine obviously, which is also a Windows Server 2019...

[Nathan]

@Yongjian Feng - I have done as suggested and executed the same script headlessly on the development machine in question (i.e. executed the same Windows Batch script that Jenkins CI would have executed). I still observe that the task completes very quickly compared to when CI does the same task: 13 minutes vs more than 90 minutes.

I am beginning to wonder if the functional account that performs the Jenkins CI tasks may be somehow configured differently that my personal account which I am using for testing. Although I am not sure where to begin my investigation ...

[Michael]

i have the same issue, did you find any solution?

[Nathan B]

@Michael - still no solution, still dealing with very slow MATLAB /Simulink jobs in our CI environment

Open to ideas if you have any ...

[Michael]

@Nathan B please see my comment here

Answer 1

[Michael]

I made some progress.

My setup is as following:

• Linux RHEL9 hosting Jenkins as a Container (run by Podman)
• Windows 2022 Server as a Jenkins Agent with MATLAB 2024b Installation

For executing MATLAB commands I use the MATLAB Plugin for Jenkins (https://plugins.jenkins.io/matlab/), but I built a Wrapper function to redirect all temporary directories to the job's workspace, because otherwise MATLAB might hang if there are multiple jobs running at the same time.

After check-out from our VCS, a simple Script collects all tests from all the models into a test suite and then runs them. This takes about 5 to 6 minutes on my local machine, and about 30 min when done via the Jenkins Job on the agent.

What I found out with ProcMon is that even on the remote Agent the script exuction actually only takes 5-6 min but MATLAB keeps running and performing unidentified background work for 25 min before eventually closing. I could not stop MATLAB with any MATLAB command like bdclose all, exit or quit.

What I found to resolve the issue is to NOT cleanup the workspace, specifically the "cache" folder generated by Simulink. What I do instead is to leave the workspace as is and only update any out-of-date files which are under source control. When the cache is not deleted, the Pipeline takes as long as the local run (5-6 min).

We use this workaround for CI builds, but cannot use it for Release builds. So it is not really a solution for us, but what can you do.

I am honestly quite frustrated about MATLAB's intransparent background tasks.

We also needed quite some time to figure out that MATLAB shares sldd-Caches between Instances. If you do not redirect the temporary directories by changing environment variables, MATLAB instances just hang because the read/write on those Caches lead to a deadlock. In my opinion the MATLAB Plugin for Jenkins should provide a solution for this out-of-the-box, since running multiple Jobs concurrently is a common theme in Continuous Integration.

[Josh Kahn]

Are you using Process Advisor for managing your pipeline? If so, we have a utility that lets you redirect the cache:

https://www.mathworks.com/help/slcheck/padv/ref/padv.util.dictionarycache-class.html

[Michael]

We are not using Process Advisor, since we are not meeting the criteria for the support package (MATLAB/Simulink Version and MATLAB Projects).

I don't have experience with Process Advisor - I understood from you that it might solve the Caching-Problem, but does it also fix the problem that MATLAB performs unidentified background work for 25 min even after my script has finished?

[Nathan B]

@Michael thank you for sharing this - could you clarify a few items?

"leave the workspace as is" - does that mean essentially don't take any steps with the CI tool to delete the temporary files, or reset to ONLY what is tracked by version control? I believe we have tried this and did not see significant change but want to confirm this is what you meant.

Our Simulink Projects have startup shortcuts which set the Cache directory to be in the root of the project (Simulink.fileGenControl('set', 'CacheFolder'), so each of mulitple parallel jobs should not be interfering with each other.

"redirect the temporary directories by changing environment variables" - what environment variables are these?

[Michael]

@Nathan B

"leave the workspace as is":

Yes. Initially I wanted to cleanup the directory completely, meaning to delete the contents of the workdirectory with a pipeline step. this step could be either the cleanWs() plugin, deleting by my VCS plugin (which supports cleaning of any files which are not under source control) or by scripting the cleanup.

I am not doing any cleanup now, but only UPDATE the files which are under source control which have been changed since the last build. This means that any cached files or files generated during the pipeline execution will persist.

"Simulink directory redirection":

As you already pointed out, we are also redirecting the cache and also the build folder to the root directory. We do this as follows:

CachePath = fullfile(pwd, 'cache');
BuildPath = fullfile(pwd, 'build');        
Simulink.fileGenControl('set',...
    'CacheFolder', CachePath,...
    'CodeGenFolder', BuildPath,...
    'createDir', true)

"redirect the temporary directories by changing environment variables":

MATLAB uses a lot of funny temporary directories which are shared. This is especially funny because they differ from Linux to Windows. If i remember correctly Linux uses TMPDIR and Windows TMP, but for some reason when we use our Jenkins Controller (Running on Podman in Linux) to run a Job on our Windows Server (where MATLAB is executed) suddenly MATLAB uses TMPDIR on Windows.

Anyway, I wrote the following Groovy Script and put it in a Jenkins Pipeline Library to Wrap every call to the MATLAB Plugin, effectively redirecting all temporary directories I could idenfity on Linux and MATLAB:

def call(String matlabCommand) {
    echo "Setting MATLAB instance directories"
    def currentDir = pwd()
    def tempDirs = "${currentDir}/_MATLAB_INSTANCE_DIRS"
    def tmpDir = "${tempDirs}/TMP"
    def tempDir = "${tempDirs}/TEMP"
    def tmpDirDir = "${tempDirs}/TMPDIR"
    def localAppDataDir = "${tempDirs}/LOCALAPPDATA"
    bat "(if exist \"${tmpDir}\" rmdir /s /q \"${tmpDir}\") && mkdir \"${tmpDir}\""
    bat "(if exist \"${tempDir}\" rmdir /s /q \"${tempDir}\") && mkdir \"${tempDir}\""
    bat "(if exist \"${tmpDirDir}\" rmdir /s /q \"${tmpDirDir}\") && mkdir \"${tmpDirDir}\""
    bat "(if exist \"${localAppDataDir}\" rmdir /s /q \"${localAppDataDir}\") && mkdir \"${localAppDataDir}\""
    withEnv(["TMP=${tmpDir}", "TEMP=${tempDir}", "TMPDIR=${tmpDirDir}", "LOCALAPPDATA=${localAppDataDir}"]) {
        echo "Running MATLAB Command ${matlabCommand} with activity timeout"
        timeout(activity: true, time: 2, unit: 'HOURS') {
            runMATLABCommand(command:
            """
            ver;
            fprintf('--- Execution START at %s ---\\n', datetime(datetime('now')));
            ${matlabCommand};
            bdclose all
            fprintf('--- Execution END at %s ---\\n', datetime(datetime('now')));
            """
            )
        }
    }
}

By taking the above measures I could prevent the pipeline from taking 5 to 6 times as long as when I would run it locally. I hope this resolves your issue as well. Please let me know how it goes; I am very interested.

At this point I feel like saying the following when working with batch-execution of MATLAB:

Good Luck

(last statement might include some sarcasm)