Why does probabililty distribution for 3D matrix not work with NaN values?

Hi,

I am trying to calculate the critical probability for each grid point in a 3D matrix. To do so I have some sample data, calculated a test statistic and tested with a small sample to calculate the probability distribution and critical probability (all worked fine). Now I tried the same for the whole dataset (all grid points) and it won't work anymore, the code seems to stop when there are NaN values. My plan is to perform the calculation for each data point along the third dimension and skip each grid point, which is filled with NaN. This is where I am stuck now! I have tried several ways to omit NaNs but it doesn't seem to work.

This is the part of my code:

%% 2) calculate critical probability (lambda) for each grid point
f = zeros(size(U_stat,1),size(U_stat,2),100);
xi = zeros(size(U_stat,1),size(U_stat,2),100);
lambda_x_neg= zeros(size(U_stat,1),size(U_stat,2));
lambda_x_pos=zeros(size(U_stat,1),size(U_stat,2));
nonanlocs = ~isnan(U_stat); 

for i = 1: size(U_stat,1)
    for j = 1:size(U_stat,2)
   [f(nonanlocs(i,j,:)),xi(nonanlocs(i,j,:))]=ksdensity(squeeze(U_stat(nonanlocs(i,j,:))));   
% then calculate critical value lambda x for each grid point
        lambda_x_neg(i,j) = xi(i,j,(find(f(i,j,:)<=0.05 & xi(i,j,:)<=0,1,'last')));
        lambda_x_pos(i,j) = xi(i,j,(find(f(i,j,:)<=0.05 & xi(i,j,:)>0,1,'first')));
end 
end

Many thanks for your help in advance!

Answer 1

[darova]

Try this way

%% 2) calculate critical probability (lambda) for each grid point
lambda_x_neg= zeros(size(U_stat,1),size(U_stat,2));
lambda_x_pos=zeros(size(U_stat,1),size(U_stat,2));
nonanlocs = ~isnan(U_stat); 
f = U_stat*0;
xi = U_stat*0;

for i = 1: size(U_stat,1)
    for j = 1:size(U_stat,2)
        ix = nonanlocs(i,j,:)
        [f(i,j,ix),xi(i,j,ix)] = ksdensity(squeeze(U_stat(i,j,ix)));   
        % then calculate critical value lambda x for each grid point
        k1 = find(f(i,j,:)<=0.05 & xi(i,j,:)<=0,1,'last');
        k2 = find(f(i,j,:)<=0.05 & xi(i,j,:)>0,1,'first');
        lambda_x_neg(i,j) = xi(i,j,k1);
        lambda_x_pos(i,j) = xi(i,j,k2);
    end 
end

[darova]

Remember that first value can be zero

[Hanna H]

Thanks for that, this has definitely improved my code. I don't get any error messages about NaNs anymore (that seems solved), however there is a different error message for ksdensity now: 'X must be a non-empty vector or two-column matrix'.

[darova]

Is it possible that all value are NaN? SO the vector is empty

[Hanna H]

Some grid points have no entires (and only consist out of NaNs). These grid points also only have NaNs in the third dimension and these points I was trying to skip!

[darova]

Add if statements inside for loop

for i = 1: size(U_stat,1)
    for j = 1:size(U_stat,2)
        ix = nonanlocs(i,j,:)
        if any(ix)
            [f(i,j,ix),xi(i,j,ix)] = ksdensity(squeeze(U_stat(i,j,ix)));   
            % then calculate critical value lambda x for each grid point
            k1 = find(f(i,j,:)<=0.05 & xi(i,j,:)<=0,1,'last');
            k2 = find(f(i,j,:)<=0.05 & xi(i,j,:)>0,1,'first');
            lambda_x_neg(i,j) = xi(i,j,k1);
            lambda_x_pos(i,j) = xi(i,j,k2);
        else
            lambda_x_neg(i,j) = nan;
            lambda_x_pos(i,j) = nan;
        end
    end 
end

[Hanna H]

Thanks a lot. I just had to change the index in the third dimension of the ouput variables f and xi in your suggestioned code and now its working!

[f(i,j,:),xi(i,j,:)] = ksdensity(squeeze(U_stat(i,j,ix)));   

Answer 2

[KSSV]

You can skip the computation using isnan. It gives you 1 as out put if number is nan and 0 if number is not a nan. Read about isnan.

if ~isnan(num)
% do what you want 
end

You can also fill nan values using fillmissing. Or you can do inteprolation and get the values at the NaN's.

[Hanna H]

Using isnan is actually what i tried, but unfortunately it didn't work (see my code above)!