Need help explaining what my for loop is doing when reading in .dat files

I'm trying to learn and apply a for loop to call in multiple .dat files so that I can then plot them. I have a folder (myFolder) which has multiple days of 3 different types of data, so I specifically grab only the "Es" data files for each day. I searched a bunch of answers on here and finally found one that worked for me but it involves this extra step at the bottom collating all the data "allDataArray". If the "Append data" section is not in the code then "dataArray" seems to only give the last day of data in the folder and won't plot each day, even though it does read them all in. Can someone please explain why the "dataArray" is only showing the last day of data and explain how I can put a plotting line into the for loop itself, currently I have it running after with the "allDataArray" but I'd like to run it without the "Append data" section.

% Specify the folder where the files live.
myFolder = 'C:\Users\tbrob\OneDrive\SO298 (2023)\Data\MATLAB\WorkingFiles';
% Check to make sure that folder actually exists.  Warn user if it doesn't.
if ~isdir(myFolder)
  errorMessage = sprintf('Error: The following folder does not exist:\n%s', myFolder);
  uiwait(warndlg(errorMessage));
  return;
end
% Get a list of all files in the folder with the desired file name pattern.
filePattern = fullfile(myFolder, '*Es*.dat'); % The folder has 3 different files types I only want the Es data from each day.
theFiles = dir(filePattern);
for k = 1 : length(theFiles)
  baseFileName = theFiles(k).name;
  fullFileName = fullfile(myFolder, baseFileName);
  fprintf(1, 'Now reading %s\n', fullFileName);
  % Now do whatever you want with this file name,
  % such as reading it in, plotting it, using it, or whatever.
  dataArray = readtable(fullFileName);
  dataArray=table2array(dataArray);
  % Append data
  if k == 1
     allDataArray = dataArray;
  else
     allDataArray = [allDataArray; dataArray]; % Must have same number of columns
  end
end
%The allDataArray has rows with NaN values and then the variable names at
%the beggining of each day, which we dont want so just getting rid of them via the NaNs
rows = any(isnan(allDataArray),2);
allDataArray(rows,:) = [];
%plotting
x=(320:2:950);
y=allDataArray(:,7:322);
L=y(1:2:end,:);
R=y(2:2:end,:);
subplot(2,1,1);
plot(x,L);
subplot(2,1,2);
plot(x,R);

Answer 1

[Atithi]

The reason dataArray only shows the last day of data is because you overwrite its value in each iteration of the loop. In each iteration, you read a new data file and assign its contents to dataArray, effectively discarding the previous data.

To plot the data within the loop without the need for the "Append data" section, you can move the plotting code inside the loop.

% Specify the folder where the files live.
myFolder = 'C:\Users\tbrob\OneDrive\SO298 (2023)\Data\MATLAB\WorkingFiles';

% Check to make sure that folder actually exists. Warn user if it doesn't.
if ~isdir(myFolder)
    errorMessage = sprintf('Error: The following folder does not exist:\n%s', myFolder);
    uiwait(warndlg(errorMessage));
    return;
end

% Get a list of all files in the folder with the desired file name pattern.
filePattern = fullfile(myFolder, '*Es*.dat'); % The folder has 3 different file types; we only want the Es data from each day.
theFiles = dir(filePattern);

% Plotting setup
figure; % Create a new figure
subplot(2,1,1);
hold on; % Enable hold to overlay plots
subplot(2,1,2);
hold on; % Enable hold to overlay plots

% Process each file and plot the data
for k = 1 : length(theFiles)
    baseFileName = theFiles(k).name;
    fullFileName = fullfile(myFolder, baseFileName);
    fprintf(1, 'Now reading %s\n', fullFileName);
    
    % Read the data file
    dataArray = readtable(fullFileName);
    dataArray = table2array(dataArray);
    
    % Extract relevant data for plotting
    x = 320:2:950;
    y = dataArray(:, 7:322);
    L = y(1:2:end, :);
    R = y(2:2:end, :);
    
    % Plot the data
    subplot(2, 1, 1);
    plot(x, L);
    subplot(2, 1, 2);
    plot(x, R);
end

% Adjust plot labels, titles, etc. as needed

% The allDataArray has rows with NaN values and then the variable names at
% the beginning of each day, which we don't want, so just getting rid of them via the NaNs
rows = any(isnan(allDataArray), 2);
allDataArray(rows, :) = [];

% Plotting
x = 320:2:950;
y = allDataArray(:, 7:322);
L = y(1:2:end, :);
R = y(2:2:end, :);
subplot(2, 1, 1);
plot(x, L);
subplot(2, 1, 2);
plot(x, R);

With this updated code, each file's data will be plotted within the loop, removing the need for the "Append data" section. Note that I've added the necessary hold on statements to allow overlaying the plots on the same figure. You may need to adjust the plot labels, titles, etc., as per your requirements.

Please give a follow up if this is working, or if you need any more help.

[Brandy]

That worked perfect, thanks! and thank you for explaining what was happening, it really helps me understand for loops a lot better.