filtfilt function returning NaN at certain frequencies

32 views (last 30 days)
I'm hoping someone can help me sort this out. I'm trying to use the filtfilt function with a butterworth filter, and the maximum frequency range I can specify depends on the sampling frequency Fs. For example:
Fs=16kHz, my maximum bandpass range is [121 7944]. Any frequency lower than 121 or higher than 7944 causes filtfilt to return NaN.
Fs=32kHz, my maximum bandpass range is [209 15900]. Any frequency lower than 209 or higher than 15900 causes filtfilt to return NaN.
Similar questions have been asked in the forum before, but the answers were usually to check the data for NaN or Inf values. I have neither in my data.
The program I'm developing needs to be generic enough to accept any sampling rate and user-specified bandpass. I want to add a warning if the user specifies a bandpass that will cause problems, but I'm not sure how to calculate the limit. I'm sure a large part of my problem is unfamiliarity with filters. Is there some nuance in filter design that I'm missing? Any help would be appreciated. Thanks.
Michael
  5 Comments
Jan
Jan on 28 Aug 2013
@Jeremy: And again, which version of FILTFILT are you using? Matlab < or >= 2011b, Octave's version, the implementation in EEGLAB or the compiled MEX from the FileExchange? I couldn't reproduce any of the problems in this thread because in every description a single important detail is missing. But I'm really eager to solve this.
Jeremy
Jeremy on 28 Aug 2013
Edited: Jeremy on 28 Aug 2013
2012b, 64-bit. I am just using the built-in butter and filtfilt functions.
edit: My neighbor ran this on 2012b 32-bit and got a valid result, I ran on 2010b 64-bit and still got an invalid result (not NaNs, but numbers in the -10^60 to +10^144 range.

Sign in to comment.

Answers (6)

Cameron
Cameron on 4 Mar 2013
I found the source of my problem. I am using the EEGLab toolbox. It appears that it causing a problem with the native filtfilt fxn. Removing the EEGlab toolbox from my paths list fixed the problem.

Jan
Jan on 28 Aug 2013
x = randn(35000,1);
[b, a] = butter(10, 60/5000, 'high');
x = filtfilt(b,a,x);
Inside filtfilt the reflection method for the start values need to solve the linear system (from version 2009a):
zi = sp \ (b(2:nfilt).' - a(2:nfilt).'*b(1));
But for the filter parameters a and b of the Butterworth filter, the condition of sp is 2.3e19. As a rule of thumb you loose k significant digits in the solution of the linear system, when the matrix has the condition 10^k. In this case loosing 19 digits mean, that the results are random. Huge values and overflows can be expected.
Obviously the reflection method proposed by Fredrik Gustafsson (Determining the initial states in forward-backward filtering, IEEE Transactions on Signal Processing, pp. 988--992, April 1996, Volume 44, Issue 4) has a limitation here. Unfortunately I do not have access to this paper to find out, what is said in the discussion about this problem. Sending an enhancement request to TMW would be a good idea.
The FiltFiltM implemtation suffers from exactly the same problem, but it does not use a sparse method to solve this system, such that a warning message appears. I'm not sure, why this does not happen for the sparse version.
  3 Comments
Jan
Jan on 29 Aug 2013
You could use a modified copy of FILTFILT or FiltFiltM from the FileExchange, which check the condition of the reflection matrix at first. When the condition is above 10^10, the method to reduce transitional effects at the start and end of the signal will fail. A simple solution is to disable the tricks to reduce the transitional effects. A warning should appear at least, but it is not a bug to admit, that the the filter parameters do not allow to apply trials to suppress noise induced by the filter algorithm. Physical filters e.g. in electric circuits suffer from the same effects.
For FiltFiltM:
K = ...
if condest(K) < 1e10
IC = K \ (b(2:Order) - a(2:Order) * b(1));
else
IC = 0;
warning('JSimon:FiltFiltM:TransitionalNoise', ...
'Cannot damp transitional effects at the edges.');
end
Jeremy
Jeremy on 29 Aug 2013
Edited: Jeremy on 29 Aug 2013
Talked to TMW, the workaround is to use the filter spec object, seems to work well. No issues with long records or extreme cutoff frequencies.
d = fdesign.highpass('N,F3dB',10,60/5000);
hd = design(d,'butter');
x=randn(35000,1);
y = filtfilt(hd.sosMatrix,hd.ScaleValues,x);

Sign in to comment.


Jan
Jan on 7 Feb 2013
Edited: Jan on 7 Feb 2013
filtfilt is based on the calculations of filter, which is a deterministic function. When you call it with finite values, infinite values are only replied in case of overflows.
I would be helpful if you post the relevant part of the code instead of describing it by some text. E.g. your maximum bandpass range of [121 7944] looks mysterious. How do you define the filter parameters exactly? Usually the cut-of frequencies are normalized by the Nyquist frequency, such that only values between 0 and 1 are accepted. Perhaps your test for finite values of the input has a bug and inspite of your assumption the input does contain NaNs.
Anyhow, whenever I've seen reports of such magic behaviour, there have been very real bugs at unexpected locations. Usually Matlab's toolbox functions have been shadowed by user-defined functions or variables. And without any doubt, Matlab's filter accepts the full range of possible frequencies from 0 to 1 (exclusively for numerical reasons). You can see an implementation as M-file with exactly the same results here: Answers: Use Filter Constants. So you can use this M-function instead of Matlab's built-in filter() and check in every step, where any why the non-finite values occur.
  3 Comments
Michael
Michael on 7 Feb 2013
One quick follow-up. I stepped through the filtfilt function and the problem lies with this line:
y = [2*x(1)-x((nfact+1):-1:2);x;2*x(len)-x((len-1):-1:len-nfact)];
According to the comments, this command is used to Extrapolate beginning and end of data sequence using a "reflection method". Slopes of original and extrapolated sequences match at the end points. This reduces end effects.
If I run my original dataset (x) through the rest of the code, no problem. But when it's transformed into y, the first filter() call returns NaN for over half the dataset.
Jan
Jan on 4 Mar 2013
@Michael: This sounds such strange, that I believe there is another bug anywhere else.

Sign in to comment.


Cameron
Cameron on 3 Mar 2013
I am having the same problem. I am using OSX 10.8.2 with 2012a. filtfilt was working fine but now using the 'butter' filter is only returning NaN's. If I used the code in the example found in doc filtfilt the 'equiripple' works fine but the the 'butter' again returns garbage. I wonder is this has something to do with a Java update or OSX Update.
rng default;
x=ecg(500)'+0.25*randn(500,1); %noisy waveform
h=fdesign.lowpass('Fp,Fst,Ap,Ast',0.15,0.2,1,60);
d=design(h,'equiripple'); %Lowpass FIR filter
y=filtfilt(d.Numerator,1,x); %zero-phase filtering
y1=filter(d.Numerator,1,x); %conventional filtering
subplot(211);
plot([y y1]);
title('Filtered Waveforms');
legend('Zero-phase Filtering','Conventional Filtering');
subplot(212);
plot(ecg(500));
title('Original Waveform');
h=fdesign.lowpass('N,F3dB',12,0.15);
d1 = design(h,'butter');
y = filtfilt(d1.sosMatrix,d1.ScaleValues,x);
plot(x,'b-.'); hold on;
plot(y,'r','linewidth',3);
legend('Noisy ECG','Zero-phase Filtering','location','NorthEast');
  2 Comments
Malcolm Lidierth
Malcolm Lidierth on 4 Mar 2013
Works fine for me with:
Operating System: Mac OS X Version: 10.8.2 Build: 12C60 Java Version: Java 1.6.0_41-b02-445-11M4107 with Apple Inc. Java HotSpot™ 64-Bit Server VM mixed mode --------------------------------------------------------------------------------------------------------
Jan
Jan on 4 Mar 2013
Does dbstop if NaNInf reveal the line, which creates the NaNs?

Sign in to comment.


Cameron
Cameron on 4 Mar 2013
Right, that is my point. It used to work for me and then post this last OSX update where apple mucked around with Java because of the security issue it is no longer working.
MatLab Version 7.14.0.739 (R2012a) OSX 10.8.2
java version "1.7.0_05" Java™ SE Runtime Environment (build 1.7.0_05-b06) Java HotSpot™ 64-Bit Server VM (build 23.1-b03, mixed mode)

Carlo E.D. Riboldi
Carlo E.D. Riboldi on 16 Dec 2013
I had a very similar issue on dataset sampled at very high frequency and very long. NaN showed up without apparent regularity when applying filtfilt to the signal, changing the order of the Butterworth filter (butter) between 4 and 8 and the cut-off frequency.
I tried the implementation of the filter through filter objects (fdesign), but this didn't fix the problem.
I downloaded the small package FilterM from Matlab Central, and this apparently fixed it all (<http://www.mathworks.com/matlabcentral/fileexchange/32261-filterm)>. I recompiled the included Mex-file with MSVC2008.
I am using Matlab 7.9.0, R2009b.

Categories

Find more on Measurements and Spatial Audio in Help Center and File Exchange

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!