Concatenating multiple tables in a cell array

Hello Everyone, I have a cell array with hundreds of tables in it, It's a 1165x1 cell array. The first 499 are empty cells but the rest have tables in them. from T{500} to T{end}

Most of the tables are 20x153 tables but some of them have different amount of columns (I have identified about 10 out of the 500) which have columns ranging from 130 to 157 columns. Is there a way I could concatenate vertically all the tables in the cell array with 20x153 dimensions ignoring these outliers?

Thank you.

[Walter Roberson]

The entries that have fewer than 153 columns: you want to ignore those entries completely, right?

The entries that have more than 153 columns: do you want to ignore those entries completely, or do you want the first 153 columns of them to be used?

[Daniel Abraham]

Well I was actually looking for an easy way out haha,I think I’d prefer the second option if it’s possible.

Answer 1

[Walter Roberson]

mask = cellfun(@(c) size(c,2) == 153, T);
Tfull = vertcat(T{mask});

[Daniel Abraham]

It helps a lot, thanks so much

[Peter Perkins]

A slight modification should allow you to keep the first 153 variables of tables that have more variables than that:

mask = cellfun(@(c) size(c,2) >= 153, T);
T2 = cellfun(@(tbl) tbl(:,1:153), T(mask), 'UniformOutput',false);
Tfull = vertcat(T2{:});

[Walter Roberson]

Yes, that should work well.

[Daniel Abraham]

Okay, I'll try it and see. Thank you for your input @Peter Perkins, Hey, @Walter Roberson, I tried this code about 5 times but on my computer, it takes about 7hrs on average to import about 700 xml files. What do you think from your experience, could it be much faster or is that how long it should take that specific amount of files running this code on an average Mac. Would like to know your thoughts.

[Walter Roberson]

What code are you using to import the XML files?

Are the XML files stored on hard drive or on SSD?

[Daniel Abraham]

okay this the code. The xml files are stored on a one drive. But I think I get a notofication that MATLAB downloads to my drive which is an SSD when starting to run the code.

%% Import xml files, well about half of it
xmlFiles = dir('*.xml');
N = length(xmlFiles);
T = cell(N,1); 
for i = 500:N %The reason I chose 500 is because importing from 1 never completes
    %  There are about 1,200 files in all
    T{i} = readtable(xmlFiles(i).name) ; 
end
celldisp(T) % to display activity in command window

%% Concatenating all tables

mask = cellfun(@(c) size(c,2) == 153, T); 
T_merged = vertcat(T{mask});

[Walter Roberson]

That code does not download from OneDrive to local disk, but if possible you would really rather read from local disk than from OneDrive. The performance loss for reading from OneDrive can be substantial. Even just starting by copying the files to local drive can improve performance a lot.

[Daniel Abraham]

I see, thanks a lot, I appreciate your suggestions