How can I create a MxN tall array, out of tall column vectors without known and not equal row numbers?

Hi there,

again I'm struggeling with tall arrays and want to ask you for help.

I have a data set consisting of voltage values. Data was collected on a few hundreds of electrodes. So voltage values are stored row-wise per electrodes, that are columnwise ordered.

Now, in my program the crossings of a threshold are calculated by looping through the columns relusting in tall column vectors. Depending on the number of detected crossings the number of values differs. I need the values evaluated (using gather in the end) for further processing, but I want to do that for the crossings of all electrodes all together, I would like to store each tall column vector in one tall variable.

But I don't know how. Since the number of values differ, horzcat is not possible. I thought I could preallocate by a zero tall array, with the highest possible number of values as row number, but I guess I cannot index that to store a tall column vector with unknown number of rows.

I appreciate any ideas and hints.

Thank you so mch in advance

Eva

ds = datastore(fullfile(filepath,filename));

names = ds.VariableNames;
len = length(names); %first column contains time

%len = number of electrodes + 1
for col = 2:len
    nam = names{1,col};
    ds.SelectedVariableNames = {nam};
    dat = table2array(tall(ds));

    crossings = goBinning(dat, parameter, filterCoeff);
    cr = gather(crossings);
    positions = getLoc(dat,crossings); %searching for precise location by max slope in spike
end

Answer 1

[Guillaume]

Instead of horizontally concatenating the results, which as you say is not possible since they don't have the same number of rows, why not vertically concatenate them with a second column indicating which column each row corresponds to:

ds = datastore(fullfile(filepath,filename));
t = tall(ds);
allcrossings = [];
for col = 2:width(t)   %much simpler way to iterate over the columns
    coldata = t.(col);
    crossings = goBinning(coldata, paramter, filterCoeff);
    colvalue = crossings;  %duplicate tall array
    colvalue(:) = col;     %and set to column index
    allcrossings = [allcrossings; colvalue, crossings];
end
allcrossings = gather(allcrossings);

No idea if in practice it's more efficient than multiple gather.

[Eva-Maria Weiss]

Hi Guillaume,

yeah, seems like a pretty smart solution. I'll give it a try and will report about the gerformance!

Thank you so much!

Eva