Log file of minimization done in parallel using batch
4 views (last 30 days)
Show older comments
Hi,
I'm working on a problem that has an outer minimization that searches over one set of parameters, and an inner minimization that for every iteration of the outer one, solves n systems of equations, where each system depends on the parameter values of the iteration of the outer loop. As all systems of the inner loop are independent of each other, I can use the parallel features of MATLAB to speed up the outer minimization. This work perfectly when using the 'local' profile in my laptop and I save a lot of time by using PARFOR in the inner minimization, compared with the same situation without parallelizing the inner loop.
However, I need to submit this as a batch job to a remote cluster, and I would like to be able to monitor the job. In particular, I would like to see the value of the objective function at each iteration, as it is shown in my laptop when 'Display','iter' is specified in options.
I have tried using
job = batch('script', 'matlabpool', 4,'CaptureDiary',true)
However, using
diary(job)
while the job is running returns nothing. It seems that I'll have to wait to see the output of diary() until the job is finished. Is there any way to observe what I want? I would like to have a log file as the one that is generated when I run the script as a batch job without using the parallel features.
Thanks,
Fernando
0 Comments
Accepted Answer
Edric Ellis
on 9 Jul 2012
Edited: Edric Ellis
on 24 Nov 2020
At this time, it's not possible to get the diary output from a batch job until after the job has completed execution. You could manually write status output to a file using fopen/fprintf/fclose.
EDIT
Recent versions of Parallel Computing Toolbox do support incremental updates to the diary of a batch job. For example:
c = parcluster('local');
j = batch(c, 'for idx = 1:5, disp(idx), pause(2), end');
% Wait for the job to start running.
wait(j, 'running');
% Print the diary in a loop.
while ~strcmp(j.State, 'finished')
fprintf('Diary at time: %s\n', string(datetime));
diary(j);
pause(2);
end
Results in output like this:
Diary at time: 24-Nov-2020 13:57:01
Warning: The diary of this batch job might be incomplete because the job is still running.
--- Start Diary ---
1
2
3
--- End Diary ---
More Answers (0)
See Also
Categories
Find more on Simple Batch Processing in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!