I am having trouble smoothing my data.

I have some data I need to do a bit of analysis on but I am having trouble getting it in the correct form. I have attached a fig for your viewing pleasure.

As you can see I have two set of data, one is beautiful (relatively smooth and continuous in red) the other is all over the place. The spikes you see are do to gap fills and I am not sure why it behaves this way but it seems that these "gap-fills" are interpolated data but scaled up. I need to connect the higher "gap-fills" with the lower values but I am not sure how to go about it. I have tried a few things but nothing is working as well as I'd like. First I got rid of the spikes and replaced them with NaN's then interpolated those gaps but it was not what I needed. Any help is appreciated! Thanks!

Answer 1

[Image Analyst]

Why do you need the gaps filled instead of just removing them?

newSignal = signal(signal<6500);
newTimes = times(signal<6500);
plot(newTimes, newSignal);

If you really need the new, repaired signal at all the same times that the old signal has (for some reason, but why???), then use interp1

goodSignal = signal(signal<6500);
goodTimes = times(signal<6500);
repairedSignal = interp1(goodTimes, goodSignal, times);

[daniel]

I need the entire interval, If I remove them I lose most of the data. The resultant vector is truncated horizontally giving me a false set, and the interpolation gives me very unrealistic values.

[Image Analyst]

Why do you say that the interpolation of the missing values with interp1() will give you very unrealistic values? Because you have no information about the missing values other than the values on either side of the bad stretches, interpolation would be the maximum likelihood answer. What basis or proof can you offer that the interpolated values are very unrealistic?

[daniel]

If you read the question you might have noticed me saying that I had already interpolated it. I can say with confidence that those interpolated values are not realistic values because I know how the data is structured and I know exactly what it "should" look like. Why are you so aggro?

[Image Analyst]

daniel, let me try to make my case to you. I tried to recreate your signal by making a sine wave and then adding huge spiky noise to it. Then I removed the spikes and interpolated what's left. It gave a pretty darn good representation of the perfect cosine wave signal. It is not "very unrealistic", at least to my eyes. I think it looks great. What do you think?

clc;    % Clear the command window.
close all;  % Close all figures (except those of imtool.)
clear;  % Erase all existing variables. Or clearvars if you want.
workspace;  % Make sure the workspace panel is showing.
format long g;
format compact;
fontSize = 20;

times = 1 : 600;
signal = 50*cosd(times) + 6100;
% Add noise
randomLocations = randperm(length(signal), 100);
signal(randomLocations) = 8200;
% Plot it.
subplot(2, 1, 1);
plot(signal, 'b-', 'LineWidth', 2);
title('Original, Corrupted Signal', 'FontSize', fontSize);
grid on;
% Enlarge figure to full screen.
set(gcf, 'Units', 'Normalized', 'OuterPosition', [0 0 1 1]);

% Find "good" parts of the signal.
goodSignal = signal(signal<6500);
goodTimes = times(signal<6500);
repairedSignal = interp1(goodTimes, goodSignal, times);
subplot(2, 1, 2);
plot(repairedSignal, 'b-', 'LineWidth', 2);
grid on;
title('Repaired Signal', 'FontSize', fontSize);

[daniel]

This is isn't a pissing contest if you don't like my explanation it's cool, I am not here to get into an argument. I came on here to get some help, which I did. This isn't randomly generated data this is real life, I need as much of the original signal as possible. If you don't understand "why" I am sorry, enjoy your day ;)