Read all files inside more than one directory

Hello, I have a data directory that has 2 more subdirectories which contain binary data in them. the binary data are chunks of 100Sa/s binary files of data taken over a period of time. I want to read all the data inside both directories and manipulate them. in addition, All data in each directory should construct a plot. so far this is what i have and I keep getting errors when i am trying to open the files.

    Files=dir('*.*');
    Arrayfile=[]
    for k=1:length(Files)
       FileNames=Files(k).name
       if Files(cnt).isdir, continue
        Arrayfile=[Arrayfile;FileNames]
        fid=fopen(filedir(cnt).name, 'rb', 'ieee-be')%file open inside
        timestamp = fread(fid,4,'uint32');
        dt = fread(fid,1,'float64');
        cols=fread(fid,1, 'uint32');%get the column size
        rows=fread(fid,1,'uint32');%get the row size
        data=fread(fid,[rows,cols],'double')% get the full data r x c
       end
      end

Answer 1

[Image Analyst]

See my attached demo that recurses into all subdirectories and lists files.

[Mini Me]

with my code above I can get it to display the names of files in the directory. It just that the two subdirectories . and .. comes up also. therefor making it hard for me to read. plus I'm having a hardtime with fopen since I get the file names like this: filename=file(k). name %% assuming k is a loop count integer then fid= fopen(filename,'rb') give me invalide fid error

[dpb]

As told you before, use an appropriate wildcard...don't use '*.*' to return files. Use something w/ the appropriate extension to match the data and then you won't have directories in the file names.

And, as IA shows in his function, build the array of directories to traverse first and separately from that dir call for each subdirectory in turn.

You're making this harder than it need be by not following the hints given...

Or, of course, if you insist on mixing the two, simply test that if the directory is one of the two don't want, skip it...

for i=1:length(d)
  if d(i).isdir
    if all(d(i).name=='.'),continue,end  % skip the two references
    d(i).name  % do the traverse in the subdirectory here
  end
end

[Mini Me]

@ dpb there seem to be a misunderstanding. One file inside the directory does not have all the information. think of it as big audio file that is chopped up in chunks. so Id need to read all the files and combine them all together in the right order to get the final harmony. the same application works for binary files full of data info.

[dpb]

So what's the problem? As noted, put the dir call for the files wanted in the subdirectory where I noted and simply displayed the subdirectory name as a placeholder above. What you do with the files at the point is up to you.

What's the extension of the files in question and how are they identified to know which is the proper order?

[Image Analyst]

How are you going to be sure you stitch all the files together in the correct order?

[Mini Me]

Here's the dir setup:

           directory

              subdirectory 1

                  file 1
                  file 2

                  file n

              subdirectory 2

                  file 1

                  file 2

                  file n

               subdirectory n

@ Image Analyst my goal is to read all the files inside directory 1 inside a loop and dumped in an array and append each time. Then have flexibility to plot the column and rows inside that array. Then do the same for the second directory. At the end i want to be able to play( add or subtract) with the final Array data of directory 1 and 2.

[dpb]

That's exactly the structure above...as said, just do a dir on the files in each directory at the point outlined above and do your thing.

However, are these stream files after, perhaps, the first one? If so, instead of reading and appending, I'd use the OS facilities and simply copy/append to one file and then read that one.

Also, again I ask--what is the file extension and how are they named precisely in order to know the sequencing?

[Image Analyst]

As dpb says, just make minor modifications, like use '*.wav' instead of '*.png' and you should be almost ready to go. I assume you already know how to stitch your audio files together. Not sure why you couldn't do it, so I did it for you. It's extraordinarily well commented so I'm pretty sure you should be able to understand it. Here, use this (also attached below code as a m-file test3.m):

% Start with a folder and get a list of all subfolders.
% Finds and prints names of all wav files in 
% that folder and all of its subfolders.
clc;    % Clear the command window.
workspace;  % Make sure the workspace panel is showing.
format longg;
format compact;

% Define a starting folder.
startingFolder = cd;
% Ask user to confirm or change.
topLevelFolder = uigetdir(startingFolder);
if topLevelFolder == 0
  return;
end
% Get list of all subfolders.
allSubFolders = genpath(topLevelFolder);
% Parse into a cell array.
remain = allSubFolders;
listOfFolderNames = {};
while true
  [singleSubFolder, remain] = strtok(remain, ';');
  if isempty(singleSubFolder)
    break;
  end
  listOfFolderNames = [listOfFolderNames singleSubFolder];
end
numberOfFolders = length(listOfFolderNames)

% Process all wav files in all those folders.
for k = 1 : numberOfFolders
  % Get this folder and print it out.
  thisFolder = listOfFolderNames{k};
  fprintf('Processing folder %s\n', thisFolder);
  
  % Get wav files in this one folder.
  filePattern = sprintf('%s/*.wav', thisFolder);
  baseFileNames = dir(filePattern);
  numberOfFiles = length(baseFileNames);
  % Now we have a list of all files in this one folder.
  
  if numberOfFiles >= 1
    % Go through all those files in this folder.
    for f = 1 : numberOfFiles
      fullFileName = fullfile(thisFolder, baseFileNames(f).name);
      fprintf('     Processing wav file %s\n', fullFileName);
      % Now do your stitching together or whatever you want to do.
    end
  else
    fprintf('     Folder %s has no wav files in it.\n', thisFolder);
  end
end

[Mini Me]

Thanks @ Image Analyst I will try it. The audio thing was an example. my files extension are .Bin data collected from an acquisition system.

[Mini Me]

@ Image Analyst I have one more question After I do the stitching, my plan is to add all the data inside each folder into an array. Meaning, for the first folder: each files contain data and each time the loop iterate those data are fed into the array and append until the loop ends. Now for the second folder, I want to create a new array to put those data into when the loop iterates the second time reading the files inside the 2nd folder; and the same for all the remaining folders you have any idea to go about that? thank you in advance

[Image Analyst]

in the inner loop over f, read in the file, add the data to some accumulation variable.

thisData = MyReadingFunction(fullFileName); % You write this
if k == 1
  allData = thisData;
else
  allData = allData + thisData;
end

[Mini Me]

I understand what you did there. I already have my data read and appended into the array for the the 1st folder. the problem is when it's goinig through the 2nd one, the new data are added to the the old data. I want each folder of data to have their own array. i.e each time the k loop iterates, it opens the folder, then the f loop read the 1st file and add the data into the array, read the second file and append it in the array and so on until it's done. then back in the k loop a second time for the 2nd folder, the f loop does the same thing but only this time create a different array to put the data in until it'd done reading all the file in the second folder. and so on

[Image Analyst]

Make all data a cell array or 3D array, such as

thisData = MyReadingFunction(fullFileName); % You write this
if f == 1
  % Create a new cell, the k'th one, for each new loop over f
  allData{k} = thisData; 
else
  allData{k} = allData{k} + thisData;
end

[Mini Me]

Thanks alot @ Image Analyst. That does the job for me to have control on the data set for each folder. but unfortunately, at each iteration it doesnt append the new data. It replaces it because I take the szie of allData{k} and it remains the same throughout all iteration. it doesnt increase.

[Image Analyst]

I don't understand what you said. At each iteration of f, the data gets added to the existing data. So when all iterations over f are done, you have the sum of all of them and that sum is in allData{k}. allData has a new cell appended at each iteration of k so the size of allData does not stay the same, it grows by one cell for each folder until in the end it has the same number of cells as there are folders.

What is your definition of "add"? Do you mean the "plus" operation as usual , or do perhaps you mean the less common "append" or "stitch" where the array on each iteration grows larger because you're stitching/appending on new elements? If you do mean append, you need to know which dimension you want to append new data onto.

[Mini Me]

no by add i mean append...meaning if the first file has 10 samples making the cell size 10 x 1 ; when the next file is read and added into the cell, the additional 10 samples will be added in that cell making the cell size to be 20 x 1. the 1st sample of the second file will be the 11 sample in that cell

[Image Analyst]

use either comma or semicolon depending if you want to append horizontally or vertically.

allData{k} = [allData{k}, thisFData]; % Stitch side by side  
allData{k} = [allData{k}; thisFData]; % Stitch vertically

[Mini Me]

Thanks @ Image Analyst so I decide to take the mean of allData{k}. Now the mean will be one value for each row. meaning a 1 x 8 matrix. I want to take each column of that matrix and feed inside a new array each time the loop read a folder of data files (k loop). meaning I want the average of all the files in folder one. take the 1st column and feed in an array, take the second and feed it in an other array. and so on. move on to the 2nd folder, it takes the 1st column and feed in the same array that contain the first column of the previous folder. then the same for the second folder and so on. If can help me it'd be much appreciated as I am stuck on this for a while now