How to put NaN when concatenating *.dat files in a single matfile
3 views (last 30 days)
Show older comments
I have a collection of (1 x 3) *.dat files that I need to combine into a single matfile (n x 3). I tried using this solution (by this user) which worked! However, I needed to preserve the row counts as it is imperative that I do so. I modified his code by addressing the size/dimension difference through:
if size(dataArray, 2) ~= 3
dataArray = nan(1,3);
end
However, I cannot seem to address .dat files that seem to be either corrupted or empty. When running the loop, I get this error:
Error using dlmread (line 147)
Empty format character vector is not supported at the end of a file.
Error in csvread (line 48)
m=dlmread(filename, ',', r, c);
Error in untitled (line 19)
dataArray = csvread(fullFileName); % Or whatever function you want.
And upon checking the .dat files that show this error, all of them looks like this (See screenshot attached).
As much as I want to skip those files, I need to put NaN in the rows that they need to be. I have attached the full code below:
% Specify the folder where the files live.
myFolder = 'something'; % Wherever...
% Check to make sure that folder actually exists. Warn user if it doesn't.
if ~isdir(myFolder)
errorMessage = sprintf('Error: The following folder does not exist:\n%s', myFolder);
uiwait(warndlg(errorMessage));
return;
end
% Get a list of all files in the folder with the desired file name pattern.
filePattern = fullfile(myFolder, '*.dat'); % Change to whatever pattern you need.
theFiles = dir(filePattern);
for k = 1: length(theFiles)
baseFileName = theFiles(k).name;
fullFileName = fullfile(myFolder, baseFileName);
fprintf(1, 'Now reading %s\n', fullFileName);
% Now do whatever you want with this file name,
% such as reading it in, plotting it, using it, or whatever.
dataArray = csvread(fullFileName); % Or whatever function you want.
if isempty(dataArray) || any(isnan(dataArray(:)))
continue; % Skip appending if this file is empty or has nans in it.
end
if size(dataArray, 2) ~= 3
dataArray = nan(1,3);
end
% Append data
if k == 1
allDataArray = dataArray;
else
allDataArray = [allDataArray; dataArray]; % Must have same number of columns
end
end
xlswrite(outputFilename, allDataArray, 'All Data', 'A1');
2 Comments
Image Analyst
on 13 Jan 2022
Make it easy for us to help you. Attach 3 files : two good ones and a bad one.
Answers (1)
Nihal
on 6 Nov 2023
Hello Jeffery,
I see that you are encountering difficulties when attempting to merge various data files. After reviewing your code, I noticed that the "csvread" function is generating errors for certain "BAD FILES." To address this issue, I recommend implementing a try-catch method to capture the error and insert "nan" rows for those specific files.
Here is the corrected code:
% Specify the folder where the files live.
myFolder = 'untitled folder'; % Wherever...
% Check to make sure that folder actually exists. Warn user if it doesn't.
if ~isdir(myFolder)
errorMessage = sprintf('Error: The following folder does not exist:\n%s', myFolder);
uiwait(warndlg(errorMessage));
return;
end
% Get a list of all files in the folder with the desired file name pattern.
filePattern = fullfile(myFolder, '*.dat'); % Change to whatever pattern you need.
theFiles = dir(filePattern);
for k = 1: length(theFiles)
baseFileName = theFiles(k).name;
fullFileName = fullfile(myFolder, baseFileName);
fprintf(1, 'Now reading %s\n', fullFileName);
% Now do whatever you want with this file name,
% such as reading it in, plotting it, using it, or whatever.
try
dataArray = csvread(fullFileName); % Or whatever function you want.
if isempty(dataArray) || any(isnan(dataArray(:)))
continue; % Skip appending if this file is empty or has nans in it.
end
catch err
dataArray = nan(1,3);
end
% Append data
if k == 1
allDataArray = dataArray;
else
allDataArray = [allDataArray; dataArray]; % Must have same number of columns
end
end
xlswrite("res", allDataArray, 'All Data', 'A1');
For a more comprehensive understanding, I suggest referring to the following documentation: https://www.mathworks.com/help/matlab/ref/try.html
I hope this information proves helpful in resolving your concern.
0 Comments
See Also
Categories
Find more on Startup and Shutdown in Help Center and File Exchange
Products
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!