Using matlab full parallelisation with pattern search + parfor

Hello everybody,

I am running Matlab R2011b and I am optimising a function which is not everywhere differentiable, using patternsearch.

The problem is that my function evaluation is very costly, because each step requires the nonlinear estimation of around 200 set of parameters with lsqnonlin (100 parameters of the [x,y] dimension and other 100 with [x,y,z,w] dimension).

Using a parfor loop with a poolsize of 8 cores, each step takes around 1-3 min. However, with an average of 1000 function evaluations, the total run time will be around 17 - 50 hours.

Would there be any better way to improve this run?

 matlabpool open 8

 patternsearch(x)

   until convergence
     parfor
   end
 end

Kind regards.

Answer 1

[Alan Weiss]

I am sorry, I do not understand your question. Are you asking whether it is better to run patternsearch in parallel, with the 'UseParallel' option set to 'always', or if it is better to run the nonlinear estimation in parallel using a parfor loop of your own devising? Or are you asking something else?

Please be a bit more specific, and then perhaps we can advise you.

Alan Weiss

MATLAB mathematical toolbox documentation

Answer 2

[Marcelo Silva]

Hi Alan,

My question concerns what would be the best strategy to follow yes. If I do a parallel pattern search with serial nonlinear parameter estimation, each estimation takes roughly 30 min.

If I do serial pattern search with parallel parameter estimation, I have more workload in pattern search, but each parameter estimation takes 1-3 min.

Taking into account that each pattern search step requires that huge parameter estimation, I was wondering if there would be a more efficient strategy, better than just parallelising the parameter estimation and running the pattern serially.

Kind regards.

Answer 3

[Alan Weiss]

You have touched on a thorny subject. I don't think there is ever a definitive answer. One general recommendation is to parallelize the innermost loop of any optimization, as it has the potential of providing the highest utilization of all the processors, but in any particular problem this might not be the fastest thing to do.

You can probably figure out for yourself what is best in your particular problem. You must have parallel hardware of at least 10, maybe even 50 processors, in order to get a speedup of a factor of 10 to 30 in serial computation of parameter estimation.

Now, how many dimensions is your problem, I mean how many entries in the x0 input? Call this number N. patternsearch will distribute evaluations to either N+1 or 2N processors, depending on which poll algorithm you use. If this number N+1 or 2N matches the number of processors you have, then it might be worthwhile using patternsearch in parallel, and parameter estimation in serial. If the number of processors is greater than N+1 or 2N, then you will certainly be better off using parallel parameter estimation, because many processors would be idle during patternsearch parallel poll. If the number of processors is less than N+1 or 2N, then it might not matter very much which strategy you employ, but I would do parallel parameter estimation anyway, because it is more likely to fully utilize the parallel processing, keeping all processors busy.

Good luck,

Alan Weiss

MATLAB mathematical toolbox documentation

Answer 4

[Marcelo Silva]

Thank you Alan,

About the number of processors to use, Matlab 2011b only allows to use a maximum pool size of 8 cores. Would there be any way of using a higher number of cores?

[Jason Ross]

For 2011b, you can run 12 local workers. The default is the number of compute cores your machine has, but you can override this by editing the properties of the local cluster (Parallel, Manage Cluster Profiles, edit the "local" profile, change NumWorkers). Keep in mind that this may actually decrease performance (or have no effect on it) if your CPUs are already fully utilized.

To go beyond 12 cores, you need to use MDCS in one way or another -- either the standalone Job Manager or by integrating with an existing scheduler.

Reference: Release Notes for 2011b http://www.mathworks.com/help/distcomp/release-notes.html

Number of Local Workers Increased to Twelve You can now run up to 12 local workers on your MATLAB client machine. If you do not specify the number of local workers in a command or configuration, the default number of local workers is determined by the value of the local scheduler's ClusterSize property, which by default equals the number of computational cores on the client machine.

Answer 5

[Marcelo Silva]

About running matlab using MDCS, I would like to ask you 2 questions:

1) It seems that in the beginning matlab starts SMPD in all the workers. Does my code executes in all the workers at the same time or only when I use my parfor cycle? This is because I would expect all the works to communicate when using my parfor cycle.

2) Although I am able to get my job output correctly (the output is a .mat file), I get this error on my cluster .OUT file, whcih is shown by all matlab workers:

%-------------------------------------------------------------------------------------------------------------%

[3]Error using distcomp/getSchedulerUDDConstructor (line 30)

[3]Unable to supply constructor for scheduler . This is an unknown type

[3]Error in distcomp_evaluate_filetask>iCreateSchedulerObject (line 474)

[3]schedulerConstructor = distcomp.getSchedulerUDDConstructor(type);

[3]Error in distcomp_evaluate_filetask>iSetup (line 407)

[3]sched = iCreateSchedulerObject(root, schedulerType, storage);

[3]Error in distcomp_evaluate_filetask (line 41)

[3]runprop = iSetup(handlers, mdceDebugEnabled, varargin);

%-------------------------------------------------------------------------------------------------------------%

I am running my job using this file:

 sched = findResource('scheduler', 'type', 'torque');

 set(sched, 'ClusterSize', 8);
 set(sched, 'ClusterOsType', 'unix');
 set(sched, 'RcpCommand', 'scp');
 set(sched, 'RshCommand', 'ssh');

 set(sched, 'ClusterMatlabRoot', '/vsc-mounts/leuven-apps/matlab/R2011b');

 set(sched, 'ResourceTemplate', '-l nodes=2:ppn=6,mem=8gb');
 set(sched, 'SubmitArguments', '-l walltime=2:00:00');

 set(sched, 'DataLocation', '/user/leuven/306/vsc30684/.matlab/local_scheduler_data/R2011b');

 job = createMatlabPoolJob(sched, 'PathDependencies', {'/user/leuven/306/vsc30684/Model Discrimination/CTMI_Schwaab/'});

 set(job, 'MinimumNumberOfWorkers', 8);
 set(job, 'MaximumNumberOfWorkers', 8);

 task = createTask(job, @run, 0, {5,1});

 tasks = get(job, 'Tasks');
 set(tasks, 'CaptureCommandWindowOutput', true);

 submit(job);

 waitForState(job, 'finished');
 destroy(job);

[Jason Ross]

If you are using local and MDCS, I could see things getting confused with the data location setting. I'm assuming that the /user/leuven directory is visible to all the cluster nodes -- make a directory like /user/leuven/torqueshared and then set the DataLocation there.

[Marcelo Silva]

I tried to create a directory there, but I don't have sufficient privileges. I also tried to create one in '/scratch/leuven/306/vsc30684', but still the same problem appears in some of the workers.

The code I'm using needs a lot of files (some of them in a private folder). Is it ok to define 'pathdependencies' to the main folder (which includes the private folder)? I am not sure about the dfference between path or file dependencies.

[Jason Ross]

For the worker to be able to work, it's going to need access to the files. If you are using a common file system between all workers, it needs to be accessible to every one. If the file system can't be found, something is going to error when it can't find needed data or code.

If you are using the term "private" in terms of the MATLAB definition of a "private" function, that's fine -- so long as the function is available to all of the workers. If you are meaning "private" as in "read access will be denied", that's going to be problematic if the workers can't read the data that they need to.

Another question -- is it possible that the file system is overloaded and can't serve the data quickly enough? Does this work with a smaller number of workers? If you set up a cluster profile, does it validate successfully? (from what you list above you should have all of the information needed to make a cluster profile and attempt to validate it in the Parallel > Manage Cluster Profile menu)