Read in files with loop

I have the code below which reads in the data to log200. The thing is, I also need to read in other data too.

I need log200, log225, log250, in increments of 25 up to log900

How can I take my current working script which reads in log200, and replace it with a loop so that I do not have to copy and paste dozens of times?

clear
clc

lastn=1;

log200=importdata('C:/Users/Ben/Desktop/log.200_equil', ' ', 20, 0);
log200_cell = log200.data(:,6);
log200_cell = log200_cell(1:end-lastn,:);
log200_density_average=mean(log200_cell)

[Stephen23]

Ugh. DO NOT do this!

Dynamically accessing or creating variables names is one way that beginners force themselves into writing slow, complex, buggy code, like your code attempts to do:

https://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval

Simpler code using indexing or fieldnames would be much more efficient, less buggy, and easier to debug.

DO NOT learn from other beginners who recommend using eval or assignin, this will only force you into writing bad code.

[Benjamin Cowen]

This seems to be loading in the files, but how do I actually access the data? How do I perform the above operations using the method below? If what I am trying below is also not what you mean, could you show me what you mean so I don't make the same mistake in the future? It would nice to have one good example to base future work on. All I want to do is load in my data quickly and calculate an average.

     lastn=1;
      clear
      clc

% Specify the folder where the files live.
myFolder = 'C:/Users/Ben/Desktop/';
% Check to make sure that folder actually exists.  Warn user if it doesn't.
if ~isdir(myFolder)
  errorMessage = sprintf('Error: The following folder does not exist:\n%s', myFolder);
  uiwait(warndlg(errorMessage));
  return;
end
% Get a list of all files in the folder with the desired file name pattern.
filePattern = fullfile(myFolder, '*_equil'); % Change to whatever pattern you need.
theFiles = dir(filePattern);
for k = 1 : length(theFiles)
  baseFileName = theFiles(k).name;
  fullFileName = fullfile(myFolder, baseFileName);
  fprintf(1, 'Now reading %s\n', fullFileName);

end

[Stephen23]

@Benjamin Cowen: how to read files in a loop is shown in the MATLAB documentation:

https://www.mathworks.com/help/matlab/import_export/process-a-sequence-of-files.html

Note the simple cell array and indexing used to store the data.

[Benjamin Cowen]

@Stephen Cobeldick Please show me one example. clearly this is wrong, but how do I put myfilename at the end of the path? All I want to do is load in my files, without having to manually do it. The example you sent me assumes the file is in my current directory right? Why does it not include a path? No matter what I try, I just "Unable to open file." Why is there not a more detailed message than that? No matter what I try, whether I am just putting the file names as file1.txt and using the code exactly as is, or whatever, I just get that error saying it cannot open file.

     lastn=1;
      clear
      clc

numfiles = 28;
mydata = cell(1, numfiles);

for k = 200:25:numfiles
  myfilename = sprintf('log%d_equil', k);
  mydata{k} = importdata('C:/Users/Ben/Desktop/myfilename', 20, 0))
end

[Stephen23]

@Benjamin Cowen: see my answer.

[Benjamin Cowen]

All I care about is the 6th column from each log file. I skip the first 20 lines of the file, and just want the average value of the 6th column from each log file. That way, I can just plot the average values at each corresponding X.

[Stephen23]

@Benjamin Cowen: sure. It would help a lot if you uploaded a sample file or two, by clicking the paperclip button.

[Benjamin Cowen]

There is a sample file in my original question. By the way, I tried your solution below, and it seems like you know what you are talking about. I appreciate the help. I have a data structure (I guess), labeled C. This is {1,29}. Each one of these has a matrix in it called data. First I want to delete the last line from each matrix, which my original code did, and then take the average of the last 100 points in the 6th column. I tried doing some of this with my original solution. How do I apply it here?

Answer 1

[Stephen23]

Simpler to use indexing and fullfile:

D = 'C:/Users/Ben/Desktop/';
S = dir(fullfile(D,'*_equil')); % where is the file extension?
N = numel(S);
C = cell(1,N);
M = nan(1,N);
for k = 1:N
    F = fullfile(D,S(k).name);
    fprintf('Now reading %s\n',F);
    T = importdata(F,' ',20);
    C{k} = T.data;
    M(k) = mean(T.data(1:end-1,6));
end

All data will be in cell array C, and M should be all of the mean data joined together into one numeric vector. You will need to check the options for importdata: I don't have this so I can't run this code!

"... then take the average of the last 100 points in the 6th column"

If you really want the last 100 points, then perhaps you will need this:

      D{k} = mean(T.data(end-100:end-1,6));

[Benjamin Cowen]

So looks like C is a structure. 1x29. each of these contains a matrix called data. How do I reference the 6th column from the matrix data in one of them? I guess this works: C{1}.data(:,6)

[Benjamin Cowen]

The last row needs to be deleted from each matrix. I have that in my original code, but I cannot seem to get it to work here. also, for your most recent code i get an error:

Error using fprintf
Function is not defined for 'cell' inputs.

Error in pract (line 14)
    fprintf('Now reading %s\n',F);

[Stephen23]

With my updated answer, yes D{1}.data(:,6) should give the sixth column of the first files's data. Note that you can easily loop over each cell of the cell array, or simply calculate the mean inside the same loop where you read the files (as my answer shows).

Error using fprintf
Function is not defined for 'cell' inputs.

Both inputs to fullfile need to be char vectors, like my answer shows. Check the class of D and S.name.

[Benjamin Cowen]

I don't know what the problem is. The code worked earlier. when i copy this code, it does not work. It gives that error. I have it copied exactly as you have it. D says 1x29 cell but they all appear to be empty

[Stephen23]

@Benjamin Cowen: you are right, I re-used the variable name D for one of the cell arrays. I changed it in my answer, please check it again. You will need to define the path D again (i.e. run the code from the start!).

[Benjamin Cowen]

I think you have helped enough. I got a good enough start on this problem. thanks for all the help!

[Stephen23]

@Benjamin Cowen: any time, it was a pleasure! It is nice working with people who show an interest, and who actually respond to our replies :)

Good luck with the code and come and ask us anything!

Answer 2

[Paul Shoemaker]

You could wrap it in a FOR loop and employ the "eval" function.

    lastn=1;

    clear
    clc
    for idx = 200:25:900
        temp = importdata(['C:/Users/Ben/Desktop/log.' num2str(idx) '_equil'], ' ', 20, 0); % Use num2str to replace "200" with the idx value
        temp = temp.data(:,6);
        temp = temp(1:end-lastn,:);
        eval(['log' num2str(idx) '_density_average=mean(temp);']); % Now evaluate to create desired output
    end

In the above, I'm assuming that only the "mean" calculation is desired and everything prior to that is just processing that you can throw away. If not, you can use "eval" on those as well.

Paul Shoemaker http://www.matlabinvesting.com

Answer 3

[Image Analyst]

Do not use eval(). To read in a bunch of files called "log.***" use one of the two code samples in the FAQ: https://matlab.wikia.com/wiki/FAQ#How_can_I_process_a_sequence_of_files.3F

[Benjamin Cowen]

How do I use this code, below, which reads in each file, and perform the above operations still?

     lastn=1;
      clear
      clc

% Specify the folder where the files live.
myFolder = 'C:/Users/Ben/Desktop/';
% Check to make sure that folder actually exists.  Warn user if it doesn't.
if ~isdir(myFolder)
  errorMessage = sprintf('Error: The following folder does not exist:\n%s', myFolder);
  uiwait(warndlg(errorMessage));
  return;
end
% Get a list of all files in the folder with the desired file name pattern.
filePattern = fullfile(myFolder, '*_equil'); % Change to whatever pattern you need.
theFiles = dir(filePattern);
for k = 1 : length(theFiles)
  baseFileName = theFiles(k).name;
  fullFileName = fullfile(myFolder, baseFileName);
  fprintf(1, 'Now reading %s\n', fullFileName);

end

[Image Analyst]

It doesn't look like your file pattern is correct, but it looks like you have a working answer from Stephen. However, if you're still interested in using the FAQ code...

I think the file pattern should have been 'log.*' will work for "log.300_equil" if the extension if of the form "nnn_equil".

And 'log*.txt' will work for "log.300_equil.txt" if your files have '.txt' extensions like the one you attached.