Sparse for arrays with lots of NaN.

Hi,

I am using very large arrays containing a lot fo NaNs. If they were zeros, I could use 'sparse' to save a lot of memory, but since they are NaNs this does not work. Does a sparse datatype exist for arrays with a lot of NaNs?

Answer 1

[Matt J]

No, but why do you insist on using NaNs instead of zeros?

[Laurent]

In that case I will have to change to zeros indeed. Thanks for the fast answer!

[Matthew Breakey]

I have the same problem. My matrix contains wind speeds where wind speeds are valid for the range of 0 to 100 and invalid/missing records (sensor failure) are indicated with NaN. I would like to use sparse() to minimize matrix size while still differentiating between a valid record with a value of 0 and invalid records with no value (NaN).

[Matt J]

How often does sensor failure occur? Is it really often enough to benefit from a sparse representation? That would mean that at least half of your records are invalid.

But in any case, you could, for example, add +1 to all the valid wind speed values so that they are encoded into the interval [1,101]. Then reserve 0 for missing data. Since you clearly don't plan to do linear algebra with this matrix, I don't think this should interfere with anything.

[Walter Roberson]

Construct a sparse logical array of locations that are nan. Zero those locations out in the other array and sparse that (assuming that less than about 1/3 of the entries remain as non-zero, to make it worth while to construct the sparse array.)

If more than about 2/3 of your locations are nan, then it might instead make sense to create a logical array of locations which were originally 0 and not nan, and zero out the nan values and sparse the result of that; then the entries in the logical array would say where the "true" zeros are, and the rest of the zeros in the second array would be assumed to be nan.