reentering a parallel pool

Hi everyone, I try to parallel function evaluation on workers. Suppose I have 10 different vectors that will be evaluated through same function. I want to retrieve outputs when they become ready. Also, when the certain number of outputs is reached, I want to evaluate the retrieved outputs through a different function without waiting to fetch all of the results. Moreover, when the unretrived data of the agents become ready, I want to use it on the same fc2 function. I want to incomplete parfeval futures to use when they finished on the same fc2 function. Also, It will be a continuing process. Whenever 5 of the retrieved results become ready, they will be used as an input for fc2 function and output of the fc2 function will be input for parfeval function again and again. I wonder is it possible or not? Here is my code.

while itr<itrmax

for j=1:nbagents

f(j) =parfeval(p,@evaluate,1,agent(j));

end

counter=0;

finishedjobs=[];

for idx = 1:nbagents

[completedIdx,value] = fetchNext(f);

finishedjobs=[finishedjobs completedIdx];

agent(completedIdx)=value;

counter=counter+1;

if counter==5

break;

end

end

for i=1:counter

output(i)=agent(finishedjobs(i))

end

target=fc2(output(i));

itr=itr+1;

end

Answer 1

[Sean de Wolski]

Look at using a DataQueue to transmit intermediate data from one parfeval future that's running continuously.

web(fullfile(docroot, 'parallel-computing/parallel.pool.dataqueue.html'))

[Emre Senol]

I do not have trouble to get the output from the worker. I want to evaluate the received output from fc2 function again through evalute function without canceling the other jobs. whenever, I send them to the pool, there is an error message that says only 1 pool can be active at a time.

[Sean de Wolski]

You can only have one parallel pool at a time. Do you want to send them back to another parfeval future? It's not clear what you want. What you have above would be best done with a DataQueue.

[Emre Senol]

I want to send them back to the same parfeval future. Is it possible? I've read the DataQueue help documentation but I could not find the answer.

[Sean de Wolski]

I don't think that's possible with parfeval, unfortunately. It can be done with spmd, though that's synchronous. Maybe one of the Parallel Devs will chime in.

[Emre Senol]

Thank's for your help.