CSV huge data and mesh grid.

Hi all,

I am quite old to Matlab, but have not used it for so many years (since 2007...) so a lot of rust out there.

I am getting some accelerometer data in a CSV file in the following format:

Time X Y Z Vector

( I combine the X,Y and Z components to find the final vector. )

There are 46 CSV files to read and each one of them creates a unique 2D line using the Time and Vector columns from each CSV file.

Here is the issue...

I want to create a 3D surface, basically by combining these 46 lines. The first 2-3 files do not have the same collumn length, but I can replace that with zeros if need be, or maybe discart them altogether and keep the ones that have the same length.

I have a 360000 x 5 double and I am using the 1st and 5th column. Meshgrid will not do the trick as it's a huge chunk of data.

So far all the data that I have seen to answers do not lead anywhere...

Your help is much appreciated,

I attach one of the files to see an example and here is the code that I have for the time being:

clc;
clear;
format loose;
file='DATA-003.csv';

data=readmatrix(file); %read file
A=data;
A(:,2)=A(:,2)./168.3; %x
A(:,3)=A(:,3)./168.3; %y
A(:,4)=A(:,4)./168.3; %z

for i=1:length(A) %find the vector 
A(i,5)=sqrt(A(i,2)^2+A(i,3)^2+A(i,4)^2);
end

x=A(:,1);
y=A(:,5);

figure(1)
% Trick surface into a 2-D plot
surface('XData', [x x],             ... 
        'YData', [y y],             ...
        'ZData', zeros(numel(x),2), ...
        'CData', [y y],             ...
        'FaceColor', 'none',        ...
        'EdgeColor', 'interp',      ...
        'Marker', 'none');

    
xlim([0 max(x)]);

Kindest regards,

Dimitri

Answer 1

[Star Strider]

I am not certain what result you want.

I would do somemthing like this:

T1 = readtable('https://www.mathworks.com/matlabcentral/answers/uploaded_files/712632/DATA-003.CSV', 'VariableNamingRule','preserve');
T1.Properties.VariableNames = {'Time','X','Y','Z'};
T1.Vector = sqrt(sum(T1{:,2:4}.^2,2))

T1 = 4888×5 table
      Time       X      Y      Z      Vector
    ________    ___    ___    ____    ______

    0.029046    -72     99    -167    207.06
    0.031694     -4     17    -163    163.93
    0.034342     12    -27    -197     199.2
    0.036989     32     51    -159    170.02
    0.039637     57     39    -188    200.28
    0.042284     17    -11    -240    240.85
    0.044933     21     24    -192    194.63
     0.04758     48     44    -167    179.25
     0.05008     55     91    -120    160.33
    0.052728    103     76    -120    175.46
    0.055375     92     60    -183    213.43
    0.058023     55    105    -140    183.44
    0.060671     33      5    -192    194.88
    0.063319     -4    -24    -277    278.07
    0.065966     85     83    -168    205.76
    0.068584    132     32    -197    239.28

One option might be:

VectorMtx = T1.Vector*ones(1,46);

figure
surf(VectorMtx)
xlabel('File Nr')
ylabel('Time [Units]')
zlabel('Column 5 [Units]')

Ideally, you would horizontally concatenate Column 5 from each .csv file to create the matrix, rather than repeating it as I have done here.

To do this in a loop, first find the longest Column 5 vector, then preallocate a (46xN) matrix where ‘N’ is that length, and write each Column 5 vector into it, allowing for differing lengths as:

for k = 1:46

Col5(k) = ...;

VectorMtx(1:numel(Col5(k)),k) = Col5(k);

end

.

[Dimitris Lassithiotakis]

First of all thanks for the great reply. I am heavily in debt sir.

Indeed this is the effect that I was trying to do. I am trying basically to do the following.

The image below represents a day of data. Lots of them, indeed, but they show you at least the situation that the vehicle went under. Moments that was stationary, big hits etc.

Then I want to create a gui (now obviously app designer... but gui brought back some nice memories...) so that I am able to choose the specific date/rally/event etc. Then I want to display the data as below (thanks to you!) and then I want to do an FFT on them to understand better the frequency. Is it possible to do an FFT on the single file for example to get the frequency? I mean, we have the data, but we have to find the equation for them right? Or is there another way to that puzzle?

Thanks once again,

D.

[Star Strider]

As always, my pleasure!

I have very llittle experience with appdesigner (or GUIDE) so I cannot help you with that part. It is likely best that you post that part as a new Question.

The fft should be straightforward. Some prototype code:

Fs = 5000;
Fn = Fs/2;
t = linspace(0, 1E+4, 1E+4)/Fs;
signal = sin(2*pi*100*t);

L = numel(signal);
time = linspace(0, L, L)/Fs;
N = 2^nextpow2(L);
FTs = fft(signal,N)/L;
Fv = linspace(0, 1, N/2+1)*Fn;
Iv = 1:numel(Fv);

figure
plot(Fv, abs(FTs(Iv))*2)
grid
xlabel('Frequency (Hz)')
ylabel('Amplitude')
xlim([0 200])                                   % Optional

In your application, ‘signal’ can be a matrix, however the time dimension must the rows and the individual signals are the columns. Since I am not certain what result you want, also consider the pspectrum function.

.

[Dimitris Lassithiotakis]

Thanks once again.

The data we uploaded for example. Let us say that you have the Vector and time. Of that individual file.

How it would be possible to do the fft on that particular data without knowing the graph equation? Ultimately I am looking to see the frequency of that data, the main one I mean, so that I use that for fatigue calculations.

Thanks once again.

[Star Strider]

I am not certain what you are asking, or about the ‘graph equation’. Taking the fft of ‘Vector’ is straightforward, however since the sampling intervals are not uniform, it is preferable to resample them to regualar sampling intervals that approximate the original sampling firquency:

T1 = readtable('https://www.mathworks.com/matlabcentral/answers/uploaded_files/712632/DATA-003.CSV', 'VariableNamingRule','preserve');
T1.Properties.VariableNames = {'Time','X','Y','Z'};
T1.Vector = sqrt(sum(T1{:,2:4}.^2,2))

T1 = 4888×5 table
      Time       X      Y      Z      Vector
    ________    ___    ___    ____    ______

    0.029046    -72     99    -167    207.06
    0.031694     -4     17    -163    163.93
    0.034342     12    -27    -197     199.2
    0.036989     32     51    -159    170.02
    0.039637     57     39    -188    200.28
    0.042284     17    -11    -240    240.85
    0.044933     21     24    -192    194.63
     0.04758     48     44    -167    179.25
     0.05008     55     91    -120    160.33
    0.052728    103     76    -120    175.46
    0.055375     92     60    -183    213.43
    0.058023     55    105    -140    183.44
    0.060671     33      5    -192    194.88
    0.063319     -4    -24    -277    278.07
    0.065966     85     83    -168    205.76
    0.068584    132     32    -197    239.28

Fs = 380;
[Vectorr,Timer] = resample(T1.Vector, T1.Time, Fs);    % Resample To 380 Hz

Fn = Fs/2;
t = Timer;
signal = Vectorr;

L = numel(signal);
time = linspace(0, L, L)/Fs;
N = 2^nextpow2(L);
signalm = signal-mean(signal);                      % Remove The D-C (Constant) Offset So The Other Peaks Are More Easily Visible
FTs = fft(signalm,N)/L;
Fv = linspace(0, 1, N/2+1)*Fn;
Iv = 1:numel(Fv);

figure
plot(Fv, abs(FTs(Iv))*2)
grid
xlabel('Frequency (Hz)')
ylabel('Amplitude')
set(gca, 'XScale','log')                            % Optional
set(gca, 'YScale','log')                            % Optional
xlim([0 50])                                        % Optional
ylim([0.01 max(ylim)])

There is not a lot of information here, however that may be different for other data sets.

If you wanted to take the fft of ‘X’, ‘Y’, and ‘Z’:

Fs = 380;
[XYZ,Timer] = resample(T1{:,[2 3 4]}, T1.Time, Fs);     % Resample To 380 Hz

Fn = Fs/2;
t = Timer;
signal = XYZ;

L = numel(signal);
time = linspace(0, L, L)/Fs;
N = 2^nextpow2(L);
signalm = signal-mean(signal);                          % Remove The D-C (Constant) Offset So The Other Peaks Are More Easily Visible
FTs = fft(signalm,N)/L;
Fv = linspace(0, 1, N/2+1)*Fn;
Iv = 1:numel(Fv);

figure
plot(Fv, abs(FTs(Iv,:))*2)
grid
xlabel('Frequency (Hz)')
ylabel('Amplitude')
legend('X', 'Y', 'Z', 'Location','best')
set(gca, 'XScale','log')                                % Optional
set(gca, 'YScale','log')                                % Optional
xlim([0 50])                                            % Optional
ylim([0.01 max(ylim)])

The same approach applies to any other matrix.

.

[Dimitris Lassithiotakis]

Again, your help is much appreciated sir!

I think that is what I would like to see.

May be it is the nature of the Data, either way the larger files that I did the trick, came out like this:

From my understanding there is something that we could call a peak in the 80Hz area, but I'd say that randomness or low end frequency is our friend. (thoughts?) - Or there is another way to analyse these and I am missing it?

[Star Strider]

As always, my pleasure!

The signals exhibit broadband noise, so frequency-selective filters will fail. The only ways to deal with that are either wavelet denoising, or if you do not have the Wavelet Toolbox, the Savitzky-Golay filter (either sgolayfilt or smoothdata with the 'sgolay' option). I usually use 3 for the order or degree, and then vary the window length until I get the result I want. The peaks at about 10 Hz and 80 Hz should remain, however they will be less noisy and more reliable to analyse. The signal length after Savitzky-Golay filtering does not change.

.

[Dimitris Lassithiotakis]

Hi again and many thanks for that. Seems that the data are scattered all over the place indeed, but valuable information can be withdrawn once we have many more to play with, or play in a lab environment with known frequencies to correlate these data.

I managed to write the software to read all the Excel files from a directory, count them, discard some of them based on size criteria and display certain information as well as getting 5 figures and graphs on the screen.

The problem I am facing is that although the calculations have finished, surfing the figures takes some time due to the huge amount of data. My question is: Is there any way to measure that and tell the user to wait? I use the drawnow function, but still I'd like to tell the user that Matlab is doing something in the background.

Once again your help is much appreciated,

D.

[Star Strider]

As always, my pleasure!

I am not certain. The drawnow call could slow the code, so perhaps eliminating that could help. I have no idea how to estimate the time it would take to draw the surf plot. I cannot find any information on that. Perhaps telling the user to please wait for the plots to be drawn or something similar is the only option.

.