How to Remove Anomalous Patterns in Large Dataset ?

Q: how can I best remove anomolous data (anomolous trend in data) as shown in the attached plot (upper left)? The dataset is a raw 1-sec to 30-sec retimed noise matrix LAeq (sound pressure levels, dBA) vs. a Low Frequency sound vector (25-125Hz). 12MM data points in the raw file, 389K data points in the retimed file. Thanks for any help.

Answer 1

[Image Analyst]

I'd sort your data by x value then scan it with a window to find the upper most data point in the window. I'd then fit a line to those upper points and delete any data points above a line shift just a little bit below that line. movmax might work. Then deed into polyfit(x,y,1). Otherwise if there are spurious points in the upper line, I'd use RANSAC: fitPolynomialRANSAC I'd I'd have tried something but unfortunately you didn't read the posting guidelines and you forgot to attach your actual data. Maybe I'll check back later.

If you have any more questions, then attach your data and code to read it in with the paperclip icon after you read this:

TUTORIAL: How to ask a question (on Answers) and get a fast answer

[Doug Leaffer]

Thank you. I will prep the data into a .mat file to attach it.

[Doug Leaffer]

It seems the problem with anomalous data may be the data interpolation across the (missing) time gap. See new figure, below. I am attaching the dataset .mat file. I have used the following code (below) to retime the raw (1-sec) sampling data [in filename: firstUniqueRowsTT] to 30 sec averages. Then the gaps are filled erroneously, linearly. How can I best fill the gaps with NaN or not interpolate over the gaps so that a time series plot shows only the true averaged data or blanks ?

dt = minutes(0.5);

Site0 = retime(firstUniqueRowsTT, 'regular','linear','TimeStep',dt);

[Image Analyst]

If you don't have data there, and it's so noisy, then why try to interpolate it? Just delete it or set it to nan. Why does it need to be some value?

[Doug Leaffer]

Thank you Image Analyst.