I wanted to rum my code on parallel HPC cluster however I face with an erorr.
5 views (last 30 days)
Show older comments
Starting parallel pool (parpool) using the 'carya-multinode' profile ...
Connected to parallel pool with 48 workers (PreferredPoolNumWorkers).
6 Comments
Damian Pietrus
on 28 Nov 2023
This is where things can get a little bit more complicated, so I'm going to generalize and make some assumptions about your setup.
Based on your first post, it looks like your default cluster profile is called 'carya-multinode'. I'm going to assume that this is an HPC cluster of some sort, and that it has a scheduler such as Slurm or PBS. These schedulers can accept lots of arguments on how to shape your job (which queue it goes to, how much memory your job needs, etc). At the very least, what we want to do is tell the scheduler how many MATLAB workers, and by extension, how many cores we want. We can do this with the parpool command:
c=parcluster;
pool = gcp('nocreate');
numWorkers = 50;
if isempty(pool)
pool = c.parpool(numWorkers);
end
As you can see, we can change the numWorkers value to indicate how many cores we'd like the job to run on. It's then up to the scheduler to determine where those cores are going to land on the HPC cluster. Once the pool is open on one or more nodes, the parfor in your code will run across those workers.
There's more to consider when working with HPC, but that's the quick version of choosing a worker count.
If you'd like to reach out to me directly, please feel free to send an email to support@mathworks.com. Please include a link to this post and ask for me (Damian Pietrus), and the front-line technicians should pass it along.
Walter Roberson
on 28 Nov 2023
See https://www.mathworks.com/help/parallel-computing/discover-clusters-and-use-cluster-profiles.html for information on discovering and managing cluster profiles.
Each cluster profile is named. To choose to execute on a specific cluster configuration, you would use
c = parcluster('ClusterNameGoesHere');
To run your code on multiple nodes with multiple cores, you need to first find out the name that has been associated with that configuration.
Answers (0)
See Also
Categories
Find more on Parallel Computing Fundamentals in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!