How do you turn empty brackets ("[]") in a table into a NaN?

Hi, I have a large dataset that I'm having some troubles with. There are a lot of areas in the dataset where there are missing datapoints. The issue is that some of the analyses I'm trying to run won't do so on this dataset unless the empty brackets are turned into NaNs.

I was trying to turn these empty brackets into NaNs using the isempty function, but on a table, it just doesn't work. Even if I call "isempty(dataset(i,j))" and i know it to be the empty brackets, MATLAB returns a logical 0, meaning no its not empty.

The only way I've been able to get isempty to work on the dataset is by turning my entire table into a cell. Unfortunately, that's just not feasible because then I'll lose all the variables in my table when i run "table2cell(x)". This is the script though:

dataset = Data_EMA(:,14); %Data_EMA is the name of the master dataset. I took out just one row from it to run this test script

for i = 1:height(dataset)       %running script through i'th row, j'th column
    if isempty(dataset{i,1})==1 %running script through i'th row, j'th column
        dataset{i,1} = NaN;     %if the i,j is an empty cell, it sets it to a NaN
    end
end

The big issue here, as I've stated, is that I have to turn my table into a cell. Once I turn it back into a table, I'll lose all my variable names. I was thinking a solution here would be to write a giant for loop that runs through each column, saves the variable names, and then keeps concatenating each column to a new table and writes the old variable name back. This is obviously computationally expensive (going to take a very long time to run) so I'm wondering if anyone has any alterantive ways because I don't think its that big of an issue to where I would have to go down the route of using a loop that saves variable names, etc.

If someone has an idea, please let me know! I've attached the dataset to this post.

[Stephen23]

"Even if I call "isempty(dataset(i,j))" and i know it to be the empty brackets, MATLAB returns a logical 0, meaning no its not empty."

For the simple reason that a scalar cell array is not empty:

S = load('Data_EMA.mat')

S = struct with fields:
    Data_EMA: [3121×380 table]
     EMAData: [1×1 struct]
         ans: 1
     dataset: {3121×1 cell}
           i: 3121

S.Data_EMA.Data_EMA14

ans = 3121×1 cell array
    {[       9]}
    {[       6]}
    {[       7]}
    {[       7]}
    {[       7]}
    {0×0 double}
    {[       6]}
    {[       8]}
    {0×0 double}
    {[       7]}
    {[       0]}
    {0×0 double}
    {[       7]}
    {0×0 double}
    {[       8]}
    {[  2.5000]}

S.Data_EMA.Data_EMA14(6) % parentheses refer to the cell array itself.

ans = 1×1 cell array
    {0×0 double}

Read what MATLAB is telling you: 1x1 cell array, it states very clearly. Is 1x1 empty? (hint: no).

However the content of a scalar cell array certainly might be empty:

S.Data_EMA.Data_EMA14{6} % curly braces returns the cell content

ans =

     []

The different indexing to refer to a cell array and its content is explained here:

https://www.mathworks.com/help/matlab/matlab_prog/access-data-in-a-cell-array.html

[BA]

Thanks, that's really helpful and appreciated!

Answer 1

[Walter Roberson]

load Data_EMA;
mask = cellfun(@isempty,Data_EMA.Data_EMA14);

Data_EMA.Data_EMA14(1:10)

ans = 10×1 cell array
    {[       9]}
    {[       6]}
    {[       7]}
    {[       7]}
    {[       7]}
    {0×0 double}
    {[       6]}
    {[       8]}
    {0×0 double}
    {[       7]}

 
Data_EMA.Data_EMA14(mask) = {NaN};
Data_EMA.Data_EMA14 = cell2mat(Data_EMA.Data_EMA14);

Data_EMA.Data_EMA14(1:10)

ans = 10×1

9 6 7 7 7 NaN 6 8 NaN 7

[Walter Roberson]

If you are reading this data in using readtable() then there are better ways of handling this.

[BA]

Thank you! This method works great. I'm using load('X.mat') to import my dataset

[Walter Roberson]

But how did the data get into the table that was saved in the file?

[BA]

This is the code I ran:

Data_EMA = load('Data_EMA.mat');
Data_EMA = Data_EMA.Data_EMA;

When I saved it, I saved it using this command:

save('Data_EMA');

[Walter Roberson]

In order to load() you must have had a mat file to load. Did someone provide that mat file to you, or did you have some other code that reads data from a file, puts it in a table, and stores the table in the mat file?

[BA]

Oh I see what you're saying. I made the dataset using a different script. Originally, the dataset was separate with like 10 different files, and I used outerjoin to combine it into one and then save it as a .mat

Now whenever I need to run an analysis, I just load up the dataset using load('X.mat')

[Walter Roberson]

When you originally read the data from the files, it would have been better if you had done

filename = 'AppropriateName.txt';
opts = detectImportOptions(filename);
opts = setvartype(opts, FIELDNUMBER, 'double');
table_for_this_file = readtable(filename, opts);

[BA]

Thanks, I actually didn't know that. Will be using that from now on.