Why is this sparse multiplication failing?

Hi,

Why does Matlab run out of memory on the multiplication of these two sparse matrices?

The first one is of size 17249876309 x 29 but only has 7 non-zeros elements that are all on different rows and on different (albeit adjacent) columns.

The second one is of size 29 x 82 and has 432 non-zeros elements.

I can certainly exploit the nature of the sparsity in these two matrices, do the multiplication and reconstruct the big matrix.

Why does Matlab fail at doing this?

Thanks

[James Tursa]

What MATLAB version & OS? 32-bit or 64-bit?

[Patrick Mboma]

Windows 64-bit

Answer 1

[sloppydisk]

You could use the relevant indices to perform the multiplication:

b = sparse(17249876309, 29);
b(430: 490) = 1;
c = sparse(ones(29, 82));

[row, col] = find(b);
d = sparse(size(b, 1), size(c, 2));
d(row, 1:end) = b(row, :)*c; 

spy(d)
h=gcf; 
set(h.Children,'Ylim',[400 500]);

However it's probably not even necessary to construct the whole matrix again. Note that you have to use 1:end instead of : in the assignment, otherwise you get a memory error.

[Patrick Mboma]

Based on your suggestion, I implemented the following

ra=size(a,1);  cb=size(b,2);
[row,~] = find(a);
vals=a(row,:)*b;
[ii,jj,v]=find(vals);
ii=row(ii);
r = sparse(ii,jj,v,ra,cb);

It does not solve all my problems but it is certainly an improvement. Thanks

[sloppydisk]

What problems remain?

[Patrick Mboma]

problems that I think cannot be solved for now: the matrices grow bigger and bigger and at some point I can no longer do

r = sparse(ii,jj,v,ra,cb);

[sloppydisk]

What are you trying to do anyway? There must be an easier way, right?