Use spmdReduce to Achieve MPI_Allreduce Functionality

Generate Data on Workers

Start a parallel pool of four workers.

parpool("Processes",4);

Starting parallel pool (parpool) using the 'Processes' profile ...
Connected to parallel pool with 4 workers.

When you perform data parallel or single program, multiple data (SPMD) computations, you often have a result variable on all workers and want to perform an operation that combines those values.

For example, inside an spmd block, assign a value to x on each worker. This code creates a 1-by-2 variant array on each worker that is slightly different on each worker because of the worker index returned by the spmdIndex function.

spmd
    x = (spmdIndex*(1:2)).^2 + 3
end

Worker 1: 
  
  x =
  
       4     7
  
Worker 2: 
  
  x =
  
       7    19
  
Worker 3: 
  
  x =
  
      12    39
  
Worker 4: 
  
  x =
  
      19    67
  

Perform Simple Summation Reduction

You can calculate the element wise addition of x across all workers using the spmdPlus function and duplicate the result on every worker.

spmd
    s = spmdPlus(x)
end

Worker 1: 
  
  s =
  
      42   132
  
Worker 2: 
  
  s =
  
      42   132
  
Worker 3: 
  
  s =
  
      42   132
  
Worker 4: 
  
  s =
  
      42   132
  

The variables you assign inside an spmd block are represented on the client as Composite objects. You can retrieve the result from a specific worker by indexing into the Composite, similar to cell array indexing.

Retrieve the value of s on worker 1 to the client.

sClient = s{1}

sClient = 1×2

    42    132

You can also use spmdPlus to return the result to a single worker by specifying the worker index. The other workers receive an empty vector. Calculate the element wise addition of x across all workers using the spmdPlus function, and store the result only worker 1.

spmd
    s = spmdPlus(x,1);
end

Display the Composite object s on the client. Only the variable on worker 1 has two elements. The variable on the other workers are empty.

s

 
s =
 
   Worker 1: class = double, size = [1  2]
   Worker 2: class = double, size = [0  0]
   Worker 3: class = double, size = [0  0]
   Worker 4: class = double, size = [0  0]
 

Concatenate Results Data

If your results variable is small, you can concatenate the results variable across the workers by using the spmdCat function, and you can choose the dimension to concatenate along.

spmd
    y1 = spmdCat(x,1); % Concatenate along rows.
    y2 = spmdCat(x,2); % Concatenate along columns.
end
y1{1}

ans = 4×2

     4     7
     7    19
    12    39
    19    67

y2{1}

ans = 1×8

    4    7    7    19    12    39    19    67

Use spmdReduce to Perform Reduction Operations

The spmdPlus and spmdCat functions are special cases of the spmdReduce function. The spmdReduce function enables you to perform any reduction operation by applying any binary function to a variable that exists on all workers in an spmd block. You can use spmdReduce not only to sum or concatenate a variable across workers, but also to find the minimum or maximum, and perform other useful operations.

For example, use spmdReduce to calculate the element-wise product of variable x across all workers. Specify a function handle to the times function. The reduction operation treats x as a matrix distributed by rows across the workers, so the result, p ,is a 2 element row vector containing the product of each column of x across all workers.

spmd
    p = spmdReduce(@times,x);
end
p{1}

ans = 1×2

    6384    347529

You can also find the element-wise maximum and minimum of x across all workers. M{1} and m{1} return row vectors containing the maximum and minimum values of each column of x across all workers.

spmd
    M = spmdReduce(@max,x);
    m = spmdReduce(@min,x);
end
M{1}

ans = 1×2

    19    67

m{1}

ans = 1×2

    4    7

When you need to perform multiple reductions, such as computing several statistics of the same distributed array, it is more efficient to group them into a single spmdReduce call using a custom reduction function. This reduces the total number of messages exchanged between workers and improves performance. For clarity, the code in this example show single reductions, but you can generally combine them when possible.

Perform Logical Operations Across Workers

MATLAB provides built-in associative operations for logical arrays, such as logical AND, OR, and XOR. Use the function handles to the and, or, and xor functions with spmdReduce to perform these operations and combine results column-wise across all workers.

First, create a logical array y on each worker. For example, compare the elements of x to the value 4. This creates a logical array where each element is true if the corresponding element in x is greater than 4.

spmd
    y = x > 4;
end

Perform logical operations on the elements of y across the workers. Display the results on worker 1. These logical operations enable you to combine logical data across all workers using standard logical rules.

spmd
    yand = spmdReduce(@and,y);
    yor  = spmdReduce(@or,y);
    yxor = spmdReduce(@xor,y);
end
yand{1}

ans = 1×2 logical array

    0    1

yor{1}

ans = 1×2 logical array

    1    1

yxor{1}

ans = 1×2 logical array

    1    0

Perform Bitwise Operations Across Workers

Use spmdReduce to perform bitwise operations, such as bitwise AND, OR, and XOR, across all workers. These operations use the function handles to the bitand, bitor, and bitxor functions to combine data column-wise across all workers.

For example, inside an spmd block, apply bitwise operations to variable x defined on all workers and display the results from worker 1. These operations enable you to combine data across workers using standard bitwise logic.

spmd
    xbitand = spmdReduce(@bitand,x);
    xbitor = spmdReduce(@bitor,x);
    xbitxor = spmdReduce(@bitxor,x);
end
xbitand{1}

ans = 1×2

    0    3

xbitor{1}

ans = 1×2

    31    119

xbitxor{1}

ans = 1×2

    28    112

Find Locations of Global Minimum and Maximum Values

In parallel computations, it is often useful to find the global minimum or maximum values across workers and also determine which worker holds those values. You can do this by combining custom reduction functions with spmdReduce.

Locate Maximum Value

To find both the global maximum value and its location across workers, define a custom reduction function. The custom function compares the local maximum value from two workers at a time, and returns the greater one and its location as a cell array.

function out = locateMaximum(in1,in2)
% in1 and in2 are cell arrays: {localMax,maxLoc}
% Each contains a scalar max value and its corresponding location
if in1{1} >= in2{1}
    out = in1;
else
    out = in2;
end
end

Apply the locateMaximum function across all workers. For the input to the reduction function, create a cell array, where the first element is the maximum of the local data and the second element is a structure that details the worker and local array index.

spmd
    [localMax,idx] = max(x);
    maxLoc = struct("worker",spmdIndex,"arrayIdx",idx);
    result = spmdReduce(@locateMaximum,{localMax,maxLoc});
    maxval = result{1};
    maxloc = result{2};
end

Display the value and location of the global maximum value stored on worker 1.

maxval{1}

ans = 
67

maxloc{1}

ans = struct with fields:
      worker: 4
    arrayIdx: 2

Locate Minimum Value

Similarly, you can find the global minimum value and its location by defining another custom reduction function that compares the local minimum value from two workers at a time, and returns the lesser one and its location as a cell array.

function out = locateMinimum(in1,in2)
% in1 and in2 are cell arrays: {localMin,minLoc}
% Each contains a scalar min value and its corresponding location
if in1{1} <= in2{1}
    out = in1;
else
    out = in2;
end
end

Apply this function across all workers. As before, the input to the reduction function is a cell array, where the first element is the minimum value of the local data and the second element is a structure that details the worker and local array index.

spmd
    [localMin,idx] = min(x);
    minLoc = struct("worker",spmdIndex,"arrayIdx",idx);
    result = spmdReduce(@locateMinimum,{localMin,minLoc});
    minval = result{1};
    minloc = result{2};
end

Display the value and location of global minimum value.

minval{1}

ans = 
4

minloc{1}

ans = struct with fields:
      worker: 1
    arrayIdx: 1