How to smoothen the noisy part of the data?

2 views (last 30 days)
I have taken out 10 examples of data sets. These data sets show gradual variation except towards the end where they show haphazard behavior. I have tried to correlate the problem with a previous answer related to signal smoothing (link): I have a very surface knowledge of which they have used.
In an ideal situation I would like to retain the original data as long as it varies gradually. Gradually varies refers to data which:
  1. has sudden jump but then it varies smoothly (can be seen initially in plot 3 and 7).
  2. does not have spikes where a spike is sudden up and down.
Keeping in mind the above results, I have adopted the reference code in the link and made some modifications e.g. movmean to movmedian and reduced the sample size of mov to 5 because data set is small. This can mostly work for me but it shows some inconsistent behavior at the locations encircled in the screenshot attached. Sometimes at the start the points do not match. Sometimes in the midway, it chooses the noisy points. I am not much proficient in the data filtering. I would like to smoothen the disturbed data without disturbing the acceptable data at all. I Will appreciate your help in this regard.
If it does not work, I would simply have to remove the noisy data.
clc
clear all
close all
%% Data have been sorted in x and y
load('chk.mat')
x = chkdta(:,1);
dta = chkdta(:,2:end);
figure
for i= 1:10
nexttile
hold on; grid on; box on;
% data
y = dta(:,i);
y_out = y;
% create the low pass filtered / smoothed version of y
k = [5 5];
ys = smoothdata(y,'movmedian',k); % smoothed version of y
% let's detect when we have too much noise content and replace y with ys in
% those sections
[B,A] = butter(2,0.03,'high');
yhp = filtfilt(B,A,y);
% logic signal
ls = abs(yhp);
k = [5 5];
ls = movmedian(ls.^2,k);
ls = ls./max(ls);
idx = (ls>0.1);
y_out(idx) = ys(idx);
plot(x,y,':.r',DisplayName='original')
plot(x,y_out,'--ob',DisplayName='filtered')
end
  1 Comment
John D'Errico
John D'Errico on 22 Sep 2024
Unfortunately, your eye/brain is very good at knowing what you want those curves to look like. But writing an automatic scheme that willl be robust, and consistently perfect when faced with a variety of outliers is not oing to be trivial. Good luck.

Sign in to comment.

Accepted Answer

Star Strider
Star Strider on 22 Sep 2024
See if using the filloutliers function will do what you want —
clc
clear all
close all
%% Data have been sorted in x and y
load('chk.mat')
x = chkdta(:,1);
dta = chkdta(:,2:end);
dta = filloutliers(dta, 'linear'); % <— ADDED
figure
for i= 1:10
nexttile
hold on; grid on; box on;
% data
y = dta(:,i);
y_out = y;
% create the low pass filtered / smoothed version of y
k = [5 5];
ys = smoothdata(y,'movmedian',k); % smoothed version of y
% let's detect when we have too much noise content and replace y with ys in
% those sections
[B,A] = butter(2,0.03,'high');
yhp = filtfilt(B,A,y);
% logic signal
ls = abs(yhp);
k = [5 5];
ls = movmedian(ls.^2,k);
ls = ls./max(ls);
idx = (ls>0.1);
y_out(idx) = ys(idx);
plot(x,y,':.r',DisplayName='original')
plot(x,y_out,'--ob',DisplayName='filtered')
end
This runs in MATLAB Online and seems to produce tthe result you want. I can’t run this here, since I get some sort of weird ‘Authentication failed’ error. MathWorks must be doing site maintenance. (The problem didn’t exist a few hours ago.)
.
  2 Comments
Muha
Muha on 23 Sep 2024
I have tried this but with other parameters. The linear seems to work better. Thank you. Should have found this out. Do'h! Thank you for providing the solution.
Cheers

Sign in to comment.

More Answers (0)

Products


Release

R2024a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!