call Matlab code from c# / .NET threads: multi-thread scalability: is MathWorks.MATLAB.Engine best option?

Hi Experts,

Typically NET & Matlab synergies is a common topic of interest. Having said that, just simply calling a function here and there, might be of little interest / an easy use-case. A more important use-case would be multi-threading and how would calling matlab functionality fare within a highly parallel / multi-threaded .NET program architecture.

So, this is a question on best way to design on the following:

1) .NET c# application for realtime live data feed processing, able to spin up undreds of parallel threads when needed.

2) Each thread needs to call a Matlab function, passing various data such as a list of static file names that every thread with share, and a list of parameters specific to each thread. The Matlab piece has to be scalable and not impact too much the paralel thread execution.

3) NET threads would typically get back the result from the Matlab function call and carry out their execution.

Questions:

a) What's best approach for this, from a Matlab integration point of view? (having scalability and speed in mind)

b) Are there any examples anywhere of this?

Answer 1

[Hassaan]

1. MATLAB Engine API for .NET: MATLAB provides the Engine API for .NET, which allows you to call MATLAB from .NET applications. However, when it comes to multi-threaded applications, MATLAB does not inherently support multi-threaded calls to the Engine API, as each MATLAB engine session runs in a single-threaded mode. To work around this, you could instantiate multiple MATLAB engine sessions from different .NET threads, but this approach has overhead and may not scale well with hundreds of threads due to the resource-intensive nature of MATLAB sessions.
2. MATLAB Production Server: This is a more scalable solution for integrating MATLAB algorithms into web, database, and enterprise applications. MATLAB Production Server is designed for high-availability and scalable deployments, allowing you to manage the load across multiple MATLAB workers. It supports RESTful API calls, so your .NET application can easily interact with MATLAB functions over HTTP requests. This approach decouples the .NET application threads from the MATLAB execution threads, potentially providing better scalability and parallel execution without the direct overhead on the .NET application side.
3. Compiling MATLAB Code: Another approach is to compile MATLAB code into .NET assemblies using MATLAB Compiler SDK. This allows you to call MATLAB code as if it were a native .NET function, potentially running within your .NET application's threads. However, the performance and scalability would depend on how well the compiled code manages resources and handles concurrency, as the MATLAB Compiler might not parallelize operations that were parallelizable within MATLAB itself.

Design Considerations

• Data Sharing vs. Data Isolation: Sharing static data (like file names) between threads is straightforward, but ensuring thread-safe access to this data is crucial. For dynamic data or results from MATLAB functions, consider how data is returned to the .NET application to avoid bottlenecks or data corruption.
• Concurrency and Parallelism in MATLAB: Some MATLAB functions are inherently multi-threaded and can utilize multiple cores. When calling such functions from .NET, ensure that the MATLAB session or compiled code can efficiently use the available hardware resources without contention.
• Error Handling and Robustness: Ensure that the system is designed to handle failures in MATLAB function calls, such as timeouts or execution errors, especially in a multi-threaded context where such issues can complicate troubleshooting and recovery.

Examples and Resources

Direct examples of such a specific integration scenario might be hard to come by, as implementations can vary widely based on the specifics of the .NET application and the MATLAB functions being used. However, MATLAB's documentation and forums are valuable resources for exploring specific aspects of MATLAB integration:

• MATLAB Engine API for .NET Documentation: For details on calling MATLAB from .NET applications.
• MATLAB Production Server Documentation: For insights into deploying and managing scalable MATLAB applications.
• MATLAB Compiler SDK Documentation: For guidance on compiling MATLAB code into .NET assemblies.

-----------------------------------------------------------------------------------------------------------------------------------------------------

If you find the solution helpful and it resolves your issue, it would be greatly appreciated if you could accept the answer. Also, leaving an upvote and a comment are also wonderful ways to provide feedback.

It's important to note that the advice and code are based on limited information and meant for educational purposes. Users should verify and adapt the code to their specific needs, ensuring compatibility and adherence to ethical standards.

Professional Interests

• Technical Services and Consulting
• Embedded Systems | Firmware Developement | Simulations
• Electrical and Electronics Engineering

Feel free to contact me.