Save a variable during parfor loop

Hi all,

I have read some post about saving variables in a parfor loop with a self-defined function parsave

function  parsave(fname,x)

%UNTITLED2 Summary of this function goes here

%   Detailed explanation goes here

save(fname,'x');

end

I tried the above function but it does not works. I have to save a variable during parfor loop because I have a huge data to compute due to job exceed the maximum day of usage for the supercomputer partition before it can save any output. My sample of code is shown below,

load rry.mat
n = length(ea); %n = 172439820
% ea_temp = zeros(size(ea));

saveInterval = 100; %1e6, planning to save every 1e6 points
saveFilename = 'ea_temp.mat';  % Change this to your desired filename


pc = parcluster('local');
parpool(pc,str2num(getenv('SLURM_CPUS_PER_TASK')))

tic
parfor ii = (ny+1):n  % n=120 for testing
    
    % My computations here
    
    if mod(ii, saveInterval) == 0
        % Save data every 1e6 iterations and overwrite the previous file
        parsave(saveFilename, ea);
    end
                
end

% Save the final data after parfor is completed
save('ea_only.mat','ea')

Is there anyone have a solution for a similar problem or any suggestion is appreciated.

Thank you.

Keith

[Matt J]

What do you mean by "Save data every 1e6 iterations and overwrite the previous file"? A parfor loop runs in parallel, so there is no such thing as "previous". The iterations of the loop are running at the same time.

[Damian Pietrus]

Matt is correct here -- iterations in your parfor loop are not guaranteed to run in any particular order and operate completely independently from one another. You don't want to save to the same file in a parfor loop as multiple workers could be accessing the file at the same time, leading to corruption.

What are you actually looking to save in your code?

Answer 1

[Jeff Miller]

Maybe break up your computations into segments, run the parfor within each segment, and save the results at the of each segment, something like this:

load rry.mat
n = length(ea); %n = 172439820
% ea_temp = zeros(size(ea));

saveInterval = 100; %1e6, planning to save every 1e6 points
saveFilename = 'ea_temp.mat';  % Change this to your desired filename


pc = parcluster('local');
parpool(pc,str2num(getenv('SLURM_CPUS_PER_TASK')))

nsegments = ceil(n/saveInterval);
for isegment = 1:nsegments
    
  ii_start = (isegment-1) * saveInterval + 1;
  ii_end = min( isegment*saveInterval, n);
  
  parfor ii = ii_start:ii_end
    
    % My computations here
    
  end % parfor

  save(saveFilename, ea);
                
end % for isegment

% No need for another save here--the last save has everything

Answer 2

[Walter Roberson]

parfor divides the range up into chunks according to the number of pool members. Furthermore, it generally starts from the end of the range.

So if the maximum were (say) 100, and there were (say) 4 pool members, then it would start out by allocating 86-100 to the first member, 71-85 to the second pool member, 56-70 to the third pool member, 41-55 to the fourth pool member, and start those running. The first pool member to finish would be handed the next (approximately) 7 available, the second pool member to finish would get the 7 after that and so on. And the last 12-ish would be handed out one at a time.

Thus, your method of recording the "highwater" every so-many iterations is doomed to failure. Iterations are performed out of order and generally from the end towards the beginning.