How can I use detrend in a matrix with NaN?

Hello,

So, I have a matrix 5551*1601 (time*distance) containing numbers and NAN’s. Some rows are just with NaN’s. I want to use linear detrend for every row. Apparently the detrend function does not work with matrix containing NaN. To solve this problem, I tried it to make it step by step using polyfit function for a linear regression. Then just remove the linear regression from my data. Here is the code I used:

% % Applying linear detrend for each barline

for i = 1:numel(E_blimline_1_m(:,1));

    % Creates a matrix time as x in polyfit funtion:
    E_along_M(i,:) = -2000:2.5:2000; 

    % Determines the nan position:
    E_Bl1_nan = ~isnan(E_blimline_1_m(i,:)); 

    % Execute polyfit function with NaN's for linear trend (y = b*x+m) giving b and m
    E_Bl1_polyfit(i,:) = polyfit(E_along_M(i,E_Bl1_nan),E_blimline_1_m(i,E_Bl1_nan),1);
    % Attributes values for y of linear equation 
    E_Bl1_polyval(i,:) = polyval(E_Bl1_polyfit(i,:),E_along_M(i,:));
end
% Create a matrix with detrend lines by removing the trend from the original values
E_Bl1_detrend = bsxfun(@minus,E_blimline_1_m, E_Bl1_polyval);

With this code it seems to give the correct result. However, because I have some rows only with NaN’s, while running it constantly appear this message when using the polyfit function: Warning: Polynomial is not unique; degree >= number of data points.

So my question is, there is some way avoid that message?

And, there is a simplest way of doing a linear detrend on a matrix with NaN’s without using a loop?

Thanks in advanced,

Sara

Answer 1

[Naomi Krauzig]

Hi Sara,

here is an available function for detrending with NaNs.

function detrended = detrendNaN3(A,t)

%DETRENDNAN3 Detrends a matrix with (or without) NaNs into the third dimension using linear least squares.

%

% Input Arguments:

% - A: NxMxK matrix (double)

% - t: Optional 1xK time vector (double) indicating the measurement

% time of each slice. If not given, the slices are assumed

% to be evenly spaced.

%

%% Default time and data formatting

% create default t if not given

if nargin<2

t = 1:size(A,3);

end

% time to same format as A and same NaN positions

t = bsxfun(@times,permute(t,[3 1 2]),ones(size(A)));

t(isnan(A)) = NaN;

%% Calculation

% mean of time

xm = nanmean(t,3);

% mean of A

ym = nanmean(A,3);

% calculate slope using least squares

a = nansum(bsxfun(@times,bsxfun(@minus,t,xm),bsxfun(@minus,A,ym)),3)./nansum(bsxfun(@minus,t,xm).^2,3);

% calculate intercept

b = ym - a.*xm;

% calculate trend

trend = bsxfun(@plus,b,bsxfun(@times,a,t));

% remove trend

detrended = A-trend;

end

Hope it helps,

Naomi