Retrieve results preprocessed by a passed functor (parfor)

Hello there, I'm desperately trying to retrieve results from simulations distributed to worker threads by means of "parfor". I'm trying to distribute a parameter-sweep, passing in a preparation function handle and a collector handle object. The latter is supposed to look at the generic results, calculate interesting values and then store them for the sweep caller. So it is kind of a functional design, to keep the simulation config and stuff away from the experiment code.

Following is the respective method in my Model-Abstraction-Class:

   function run_sim_sweep( self, duration, omega_path, torque_path, prepare_func, collector, start_i, end_i )

            collector_slices = collector.make_slices( end_i - start_i + 1 );

            parfor i = start_i : end_i
                feval( prepare_func, self, i );
                local_result = self.run_sim( duration, omega_path, torque_path );
                collector_slices{i}.collect( self, i, local_result );
                collector_slices{i}.hack2 = i;
            end

            collector.fold_slices( collector_slices );

        end
  end

While it worked great with the single collector in serial loops, I had to slice the collector into an array of collectors for Matlab not to bark at me. But it still doesn't work the way I imagine it to work.

Here's my diagnostics: The slice's collect method stores the index i in a property "hack" and the loop stores the index too in the property "hack2". After the loop finishes, "hack" contains "2" for every element of the collector_slices cell array, whereas "hack2" contains "1" for every element.

(both collector, and the collector_slice-class it slices into are handle classes)

Can you spot an obvious mistake I made?

Best regards, Michael

[jgg]

What does the object in collector_slices{i} look like?

parfor tends to work better if you create that object in the loop, then assign it to cell i of a cell array, rather than creating an array of objects before the loop then assigning to them.

So, replace

 collector_slices = collector.make_slices( end_i - start_i + 1 );

with

 collector_slices = cell( end_i - start_i + 1,1);

and in your loop replace the collector slice code with

   slice_i = collector.make_slice(1);
   slice_i.collect( self, i, local_result );
   slice_i.hack2 = i;
   collector_slices{i} = slice_i;

this might help, but I'm not sure.

[decimad]

Hello jgg, thank you for your comment.

I adapted the code for your suggestion, which led to Matlab warning about "collector" being a broadcast variable (might harm performance). Then, while doing that, I found that the collect method shadowed the argument "i" by a loop counter of max count = 2 in my example, which explains the 2 I encountered everywhere -.- gosh.

Now the real problem seems not to be the collecting code, but the feval-code that alters the "self" object, which implicitly gets serialized in the caller thread and deserialized in all worker threads, if my understanding is correct. I might be running into a bad behaviour of my code to support this.

Is there an easy way to check/debug compatibility of the serialize -> deserialize roundtrip? I can't debug it with parfor and I'm really a newbie to Matlab. That seems to be a pretty basic question :-(

Kind Regards, Michael

[decimad]

Yep, that was it. I needed the ConstructOnLoad-Attribute in order for Property-Observers to be installed after deserialization.

Your hint in this direction guided me to the real problem though really, so many thanks indeed!

[jgg]

You should provide an answer in the "Answer" section so other people can see how you resolved your problem.

[decimad]

Yup, done. Thank you!

Answer 1

[decimad]

Alright,

I resolved the issue and as it often happens, the initial estimation of the cause was way off. There was no problem with the data-collecting code.

The real problem was that the run_sim_sweep-method-containing class instance will be serialized and then deserialized on every worker thread of the pool (save, load roundrip). By default, Matlab will not call the class constructor on load. Since I install property listeners in the constructor, the resulting object in the worker thread was not watching changes to its properties. This could be overcome by specifying the class attribute "classdef (ConstructOnLoad=true) ...", which makes Matlab call the class constructor with 0 arguments on load.