Is there a way to shift the data upwards automatically?

Hi folks,

I have a graph as follows. I am looking to shift the valley (red) upwards to be in keeping with the preceeding data. Is there a way to do this automatically? I am unaware of the best method of achieving this currently.

I've attached a copy of the data for reference.

[the cyclist]

Is this something you need to do once, for only this dataset, or automate for many possible datasets?

If that latter, might there be multiple valleys in a single dataset?

I don't think you will find something already written, that is exactly what you need. You'll need to decide exactly how to define the valley (because you do not want to adjust every small dip), and exactly what you want to do to adjust it.

[Teshan Rezel]

@the cyclist hi there, it's something I need to do for multiple datasets. However, there is al;ways only one pronounced valley in each dataset like the one circled above, due to how the data is gathered.

[the cyclist]

Also, you might want to upload one example of the data, so people can try out solutions on real data.

[Teshan Rezel]

@the cyclist good idea, thanks! I've ammended my post

Answer 1

[the cyclist]

I tried plotting Height vs. Temperature, since that is what your plot shows. But my graph looks completely different from yours.

r = readtable('https://www.mathworks.com/matlabcentral/answers/uploaded_files/652735/Results.xls','Sheet','Test Piece 1-BS 1');

Warning: Column headers from the file were modified to make them valid MATLAB identifiers before creating variable names for the table. The original column headers are saved in the VariableDescriptions property.
Set 'VariableNamingRule' to 'preserve' to use the original column headers as table variable names.

figure
plot(r.Temperature,r.Height,'.')

Perhaps make it easier for us to help you, by only uploading the two variables you are plotting above, in a MAT file?

EDIT (in response to comment):

I've uploaded the new data file. Also, this whole thing should have been a comment rather than an answer. But I'll leave it here.

[Teshan Rezel]

@the cyclist apologies for this, I think I accidentally uploaded the wrong file! I've put the correct one up now, and the sheet for test piece 1 should be the best data to demonstrate the issue.

[the cyclist]

I've edited my code above to use the new data.

It's not clear to me what you think the fix should do. In your original question, you suggested a single constant shift for the valley data. But that doesn't really seem very useful here.

[Teshan Rezel]

hi @the cyclist, I'm open to suggestions on this! I was looking to essentially bridge the gap between the two ends of the valley but without making it "dip". I've tried a gaussian smoothing but it still maintains the dip to a large degree.

[the cyclist]

We really need more context to help. Do the data in the valley carry any information at all, or are they completely spurious? Perhap you should choose an algorithm to simply identify what you consider to be the valley (which is probably a necessary step regardless), and delete them?

If your next step depends heavily on exactly how you handle the valley region, I would be worried how meaningful that next result is.

[Teshan Rezel]

Hi @the cyclist, the data in the valley isn't valuable. The problem is that the equipment that generates this data is being used for something it wasn't designed for, so the data is squiffy. The idea behind cleaning it is to make it look more presentable, but in a "meaningful" way. That is to say, I could simply copy the values from just before the valley appears across till the end of the valley, but it wouldn't look authentic. Does that make sense?

[the cyclist]

If the only thing you are doing is visualization, my approach would be to identify the bad points, and label them. (See below.) I would not try to make them look "authentic" -- this misleads your audience, and is a slippery slope. If you have a legitimate reason for not displaying these points at all, you can do that after you identify them.

If you have no need to do any further processing of these data, and can ignore them, then that's all you need.

r = readtable('https://www.mathworks.com/matlabcentral/answers/uploaded_files/652735/Results.xls','Sheet','Test Piece 1-BS 1','VariableNamingRule','preserve');

figure
hold on

isValley = (r.Temperature > 630) & (r.Temperature < 810); % Use a more meaningful identification algorithm here

plot(r.Temperature(~isValley),r.Height(~isValley),'.')
plot(r.Temperature(isValley),r.Height(isValley),'.')
legend({'good','bad'})

[Teshan Rezel]

thanks @the cyclist, I think I've gotten this far and used gaussian smoothening to get it looking a bit better. However, isolating where this valley occurs automatically is proving tricky and that's what I really need help with. I've tried to look at the max gradient and average gradient but to no avail...

[the cyclist]

The key is get a firm conceptual understanding of what defines the valley. In your original image, it looked simple, because it looked like a huge drop from one data point to the very next one. But in the dataset you uploaded that I plotted, it is not as clear.

So, before you think about how to code it, you need to make this conceptual understand that works in general for your datasets. Then I would recommend writing pseudocode for the actual algorithm steps, and then the actual MATLAB code. We can mostly be helpful in that last step.