Discover Clusters and Use Cluster Profiles
Parallel Computing Toolbox™ comes pre-configured with the cluster profiles
Threads for running parallel
code on your local desktop machine.
Control parallel behavior using the Parallel menu on the MATLAB® Home tab.
You can use the Parallel menu to:
Discover other clusters running on your network or on Cloud Center. Click Parallel > Discover Clusters. For more information, see Discover Clusters.
Create and manage cluster profiles using the Cluster Profile Manager. Click Parallel > Create and Manage Clusters. For more information, see Create and Manage Cluster Profiles.
Create and Manage Cluster Profiles
Cluster profiles let you define certain properties for your cluster, then have these properties applied when you create cluster, job, and task objects in the MATLAB client. Some of the functions that support the use of cluster profiles are
Manage cluster profiles using the Cluster Profile Manager. To open the Cluster Profile Manager, on the Home tab, in the Environment section, select Parallel > Create and Manage Clusters.
You can use the Cluster Profile Manager to:
Discover other clusters running on your network or on Cloud Center. For more information, see Discover Clusters.
Create a cluster in the cloud, such as Amazon® AWS®. For more information, see Create Cloud Cluster.
Add cluster profiles and modify their properties. For more information, see Add and Modify Cluster Profiles.
Import and export cluster profiles. For more information, see Import and Export Cluster Profiles.
Specify profile properties. For more information, see Edit Number of Workers and Cluster Settings.
Validate that a cluster profile is ready for use in MATLAB.
You can let MATLAB discover clusters for you. Use either of the following techniques to discover those clusters which are available for you to use:
On the Home tab in the Environment section, select Parallel > Discover Clusters
In the Cluster Profile Manager, select Discover Clusters
This opens the Discover Clusters dialog box, where you can search for MATLAB Parallel Server™ clusters:
If you select On your network, you see a new window. Select this option if your clusters use a MATLAB Job Scheduler, Microsoft® Windows® HPC server or other third-party scheduler cluster. As clusters are discovered, they populate a list for your selection. If you already have a profile for any of the listed clusters, those profile names are included in the list. If you want to create a new profile for one of the discovered clusters, select the name of the cluster you want to use, and select Next. The subsequent dialog box lets you choose if you want to set the created profile as your default. This option is not supported in MATLAB Online.
If you select On MathWorks Cloud Center, you search for clusters running on Cloud Center. To access these clusters, you must provide your MathWorks Account login information.
Requirements for Cluster Discovery
Cluster discovery is supported for MATLAB Job Schedulers, Microsoft Windows HPC server clusters, other third-party scheduler clusters, and MathWorks Cloud Center clusters. If you need to integrate your scheduler with MATLAB Parallel Server, or create a cluster profile for a different supported scheduler, see Get Started with MATLAB Parallel Server (MATLAB Parallel Server). The following requirements apply to cluster discovery:
MATLAB Job Scheduler — MATLAB Job Scheduler clusters support two different means of discovery:
Multicast: The discover clusters functionality uses the multicast networking protocol from the client to search for head nodes where a MATLAB Job Scheduler is running. This requires that the multicast networking protocol is enabled and working on the network that connects the MATLAB Job Scheduler head nodes (where the schedulers are running) and the client machines. This form of discovery might be limited to the client local subnet, and therefore not always able to discover a MATLAB Job Scheduler elsewhere in your network.
DNS SRV: An alternative discovery technique is to search for clusters by DNS service records.
The Domain Name System (DNS) is a standard for identifying host names with IP addresses, either on the Internet or in a private network. Using DNS allows discovery of MATLAB Job Scheduler clusters by identifying specific hosts rather than broadcasting across your network.
A DNS service (SRV) record defines the location of hosts and ports of services, such as those related to the clusters you want to discover. Your system administrator creates DNS SRV records in your organization’s DNS infrastructure. For a description of the required record, and validation information, see DNS SRV Record (MATLAB Parallel Server).
HPC Server — The discover clusters functionality uses Active Directory Domain Services to discover head nodes. HPC Server head nodes are added to the Active Directory during installation of the HPC Server software.
Third-party Schedulers – The discover clusters functionality uses a configuration file to discover third-party scheduler clusters. For more information, see Configure for Third-Party Scheduler Cluster Discovery (MATLAB Parallel Server).
Cloud Center — The discover clusters functionality requires a working network connection between the client and the Cloud Center web services running in mathworks.com.
Create Cloud Cluster
You can create clusters in Cloud Center cloud services directly from the Cluster Profile Manager. In the Cluster Profile Manager, select Create Cloud Cluster.
Sign up with your MathWorks® Account and complete the required steps. Then, you can create a cloud cluster and configure parameters, such as the number of machines or the number of workers per machine. For more information on each of the available parameters, see Create a Cloud Cluster. When you complete all the steps, MATLAB creates a new cluster profile for you. You can modify its properties from the Cluster Profile Manager.
To manage your licenses, test cloud connectivity, or manage your cloud clusters in MathWorks Cloud Center, go to Cluster Profile Manager toolstrip > CLOUD section.
Add and Modify Cluster Profiles
With the Cluster Profile Manager, you can add a cluster profile for a MATLAB job scheduler or a third-party scheduler. If you need to set up your cluster for use with MATLAB, see Get Started with MATLAB Parallel Server (MATLAB Parallel Server).
The following example provides instructions on how to add and modify profiles using the Cluster Profile Manager.
Suppose you want to create a profile to set several properties for jobs to run in a MATLAB Job Scheduler cluster. The following example illustrates a possible workflow, where you create two profiles differentiated only by the number of workers they use.
In the Cluster Profile Manager, select Add Cluster Profile > MATLAB Job Scheduler. This specifies that you want a new profile for a MATLAB Job Scheduler cluster.
This creates and displays a new profile, called MJSProfile1.
Double-click the new profile name in the listing, and modify the profile name to be
Select Edit in the tool strip so that you can set your profile property values.
In the Description field, enter the text
MJS with 4 workers, as shown in the following figure. Enter the host name for the machine on which the MATLAB Job Scheduler is running, and the name of the MATLAB Job Scheduler. If you are entering information for an actual MATLAB Job Scheduler already running on your network, enter the actual names. If you are unsure about the MATLAB Job Scheduler names and locations on your network, ask your system administrator for help.
If the MATLAB Job Scheduler is using a nondefault
BASE_PORTsetting as defined in the
Hostproperty in the cluster profile must be appended with this
BASE_PORTnumber. For example,
Scroll down to the Workers section, and for the Range of number of workers, enter the two-element vector
[4 4]. This specifies that jobs using this profile require at least four workers and no more than four workers. Therefore, a job using this profile runs on exactly four workers, even if it has to wait until four workers are available before starting.
You might want to edit other properties depending on your particular network and cluster situation.
Select Done to save the profile settings.
To create a similar profile with just a few differences, you can duplicate an existing profile and modify only the parts you need to change, as follows:
In the Cluster Profile Manager, right-click the profile name
MyMJSProfile1in the list and select Duplicate.
This creates a duplicate profile with a name based on the original profile name appended with
Double-click the new profile name and edit its name to be
Select Edit to allow you to change the profile property values.
Edit the description field to change its text to
MJS with any workers.
Scroll down to the Workers section, and for the Range of number of workers, clear the
[4 4]and leave the field blank.
Select Done to save the profile settings and to close the properties editor.
You now have two profiles that differ only in the number of workers required for running a job.
When creating a job, you can apply either profile to that job as a way of specifying how many workers it should run on.
You can see examples of profiles for different kinds of supported schedulers in the MATLAB Parallel Server installation instructions at Configure Your Cluster (MATLAB Parallel Server).
Import and Export Cluster Profiles
Cluster profiles are stored as part of your MATLAB preferences, so they are generally available on an individual user
basis. To make a cluster profile available to someone else, you can export it to a
.mlsettings file. In this way, a repository of profiles
can be created so that all users of a computing cluster can share common
To export a cluster profile:
In the Cluster Profile Manager, select (highlight) the profile you want to export.
Select Export > Export. (Alternatively, you can right-click the profile in the listing and select Export.)
If you want to export all your profiles to a single file, select Export > Export All
In the Export profiles to file dialog box, specify a location and name for the file. The default file name is the same as the name of the profile it contains, with a
.mlsettingsextension appended; you can alter the names if you want to.
Note that you cannot export profiles for Cloud Center personal clusters.
Profiles saved in this way can then be imported by other MATLAB users:
In the Cluster Profile Manager, select Import.
In the Import profiles from file dialog box, browse to find the
.mlsettingsfile for the profile you want to import. Select the file and select Open.
The imported profile appears in your Cluster Profile Manager list. Note that the list contains the profile name, which is not necessarily the file name. If you already have a profile with the same name as the one you are importing, the imported profile gets an extension added to its name so you can distinguish it.
You can also export and import profiles programmatically with the
Export Profiles for MATLAB Compiler
You can use an exported profile with MATLAB
Compiler™ and MATLAB
Compiler SDK™ to identify cluster setup information for running compiled
applications on a cluster. For example, the
setmcruserdata (MATLAB Compiler) function can
use the exported profile file name to set the value for the key
ParallelProfile. For more information and examples of
deploying parallel applications, see Pass Parallel Computing Toolbox Profile at Run Time (MATLAB Compiler), and Use Parallel Computing Toolbox in Deployed Applications (MATLAB Compiler SDK).
A compiled application has the same default profile and the same list of
alternative profiles that the compiling user had when the application was
compiled. This means that in many cases the profile file is not needed, as might
be the case when using the
Processes profile for local
workers. If an exported file is used, the first profile in the file becomes the
default when imported. If any of the imported profiles have the same name as any
of the existing profiles, they are renamed during import (though their names in
the file remain unchanged).
Edit Number of Workers and Cluster Settings
After you create a cluster profile, you can specify the number of workers and other profile properties:
NumWorkers: the number of workers to start a pool. The actual pool size might be limited by licensing, cluster size, and cluster profile settings. See Pool Size and Cluster Selection
NumThreads: the number of computational threads to use on each worker. You can change
NumThreads, so that your workers can run in multithreaded mode and use all the cores on your cluster. This allows you to increase the number of computational threads
NumThreadson each worker, without increasing the number of workers
NumWorkers. If you have more cores available, increase
NumThreadsto take full advantage of the built-in parallelism provided by the multithreaded nature of many of the underlying MATLAB libraries. For details, see Run MATLAB on multicore and multiprocessor machines .
Do not increase the number of threads across all workers on a machine to exceed the number of physical cores. In other words, make sure that
NumWorkers x NumThreads≤ number of physical cores on your machine. Otherwise you might have reduced performance.
Use Your Cluster from MATLAB
To run parallel language functions, such as
batch, on a cluster, set the cluster profile as default, or
use cluster objects.
Specify Default Cluster
To set a cluster profile as the default, use one of the following ways:
On the Home tab in the Environment section, select Parallel > Select a Default Cluster, and from there, all your profiles are available. The default profile is indicated. You can select any profile in the list as the default.
The Cluster Profile Manager indicates which is the default profile. You can select any profile in the list, then select Set as Default.
You can get or set the default profile programmatically by using the
parallel.defaultClusterProfilefunction. The following sets of commands achieve the same thing:
Specify Cluster Programmatically (
parcluster function creates a
cluster object in your workspace according to the specified profile. The profile
identifies a particular cluster and applies property values. For example,
c = parcluster('MyMJSProfile1')
This command finds the cluster defined by the settings of the profile named
MyMJSProfile1 and sets property values on the cluster
object based on settings in the profile. Use a cluster object in functions such
batch. By applying different
profiles, you can alter your cluster choices without changing your MATLAB application code.
setmcruserdata (MATLAB Compiler) |
- Run Code on Parallel Pools
- Scale Up from Desktop to Cluster
- Pass Parallel Computing Toolbox Profile at Run Time (MATLAB Compiler)
- Use Parallel Computing Toolbox in Deployed Applications (MATLAB Compiler SDK)
- Verify Network Communications for Cluster Discovery (MATLAB Parallel Server)
- Get Started with MATLAB Parallel Server (MATLAB Parallel Server)
- Clusters and Clouds