Avoiding load and/or for loop - speed up

Hi everyone!

I have a directory containing mat-files. they all contain vectors A and B.

I would like to find the maximum value A.*B out of all the mat files. However loading all the .mat files and checking if the maximum value from all the last files is crossed is very timeconsuming:

for i = first:last
    load(filename{i})    
    U = lowpass(A,wpass)*U_scale;                  % [V] Lowpass-filtered & scaled voltage data
    I = lowpass(B,wpass)*I_scale;                  % [A] Lowpass-filtered & scaled current data
    E = abs(U.*I)*t_S;
    E_quant = quantile(E,0.9);
    if E_max < max(E_quant)
        E_max = max(E_quant);
    end
end

How do I do this efficiently?

Thanks a lot!

Ann

[Rik]

You should load the variables to a struct:

S=load(filename{i});A=S.A;B=S.B;

Otherwise I'm not sure there is a lot you can do. Disks (even SSDs) are relatively slow. I'm not sure using parfor would help, I doubt it but it may be worth a try.

[Ann Gerber]

Thanks a lot, decreased the runtime already by half!

I will later on use the variables A and B from the mat files again (in a loop). would it then make sense to save the data from the mat files to varables A1, B1, A2, B2,.... (or a struct S.A1, S.B1, S.A2, S.B2,...)?

[Rik]

Do not use numbered variables. You can load to a struct array (or save your loaded variable into such an array). Then you can use indexing to reach all variable, instead of having to use more complex methods.

[Ann Gerber]

ah yes of course, i just wanted to indicate that i would load it to consecutive "rows" of the struct. I just wasn't sure if doing so is inefficient since i will create structs with huge ammount of data...

[Rik]

If it fills up your RAM, then it is probably faster to load it later on. Your disk will either be busy reading the files, or moving data between the swap on the disk and the actual RAM. This depends on the size of your data, the amount of RAM used by Matlab and other programs, and your total installed system memory.

[Mathieu NOE]

Hi

just my 2 cents

maybe you could reduce the size of data in your mat files

I see your doing low pass filtering, so why not make that (and decimate) in first place when mat files are being generated ?

[Ann Gerber]

I get the final data and have no influence on how they are generated unfortunately. Thanks anyway! When profiling the script I noticed that filtering is very timeconsuming...

[Mathieu NOE]

maybe you could decimate the data inside the loop before doing the low pass filtering

depends what is the original sampling rate and what bandwith must be kept for U and I ( for your computation E = abs(U.*I)*t_S;)

as decimation applies also low pass filtering , you have your lowpass effect plus you reduce the size of U and I.

=> help decimate