reentering a parallel pool

1 view (last 30 days)
Emre Senol
Emre Senol on 10 Jun 2020
Commented: Emre Senol on 11 Jun 2020
Hi everyone, I try to parallel function evaluation on workers. Suppose I have 10 different vectors that will be evaluated through same function. I want to retrieve outputs when they become ready. Also, when the certain number of outputs is reached, I want to evaluate the retrieved outputs through a different function without waiting to fetch all of the results. Moreover, when the unretrived data of the agents become ready, I want to use it on the same fc2 function. I want to incomplete parfeval futures to use when they finished on the same fc2 function. Also, It will be a continuing process. Whenever 5 of the retrieved results become ready, they will be used as an input for fc2 function and output of the fc2 function will be input for parfeval function again and again. I wonder is it possible or not? Here is my code.
while itr<itrmax
for j=1:nbagents
f(j) =parfeval(p,@evaluate,1,agent(j));
end
counter=0;
finishedjobs=[];
for idx = 1:nbagents
[completedIdx,value] = fetchNext(f);
finishedjobs=[finishedjobs completedIdx];
agent(completedIdx)=value;
counter=counter+1;
if counter==5
break;
end
end
for i=1:counter
output(i)=agent(finishedjobs(i))
end
target=fc2(output(i));
itr=itr+1;
end

Answers (1)

Sean de Wolski
Sean de Wolski on 10 Jun 2020
Look at using a DataQueue to transmit intermediate data from one parfeval future that's running continuously.
web(fullfile(docroot, 'parallel-computing/parallel.pool.dataqueue.html'))
  5 Comments
Sean de Wolski
Sean de Wolski on 11 Jun 2020
I don't think that's possible with parfeval, unfortunately. It can be done with spmd, though that's synchronous. Maybe one of the Parallel Devs will chime in.
Emre Senol
Emre Senol on 11 Jun 2020
Thank's for your help.

Sign in to comment.

Categories

Find more on Parallel Computing Fundamentals in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!