matlab 'unique' is skipping rows with data
12 views (last 30 days)
Show older comments
I'm trying to put parse a large file to only contain data that has changed. First column is continuous time so I exclude that column. Next 3 columns are what I need to parse...with unique values. When reviewing final file (I called 12Dec2023Events.csv) with a file I parsed by hand, I noticed not all data was included.
346:17:37:18.6839209 9 23 3E
346:17:37:18.6939210 2 E 3E (data not identified by unique)
unique did not pick up the change until
346:17:37:52.2139008 2 E 3E
346:17:37:52.2239008 2 0 3E
t=readtable('Myfile.csv');
[~,ind]=unique(t(:,2:4),'stable');
t2=t(ind,:);
Please help the files are just to massive to parse by hand and having a file with data missing isn't a solution either
0 Comments
Answers (2)
Stephen23
on 2 Jan 2024
Edited: Stephen23
on 2 Jan 2024
The basic problem is that your file is large, and by default READTABLE checks a limited number of rows** before deciding what data type each column has***. Your data file has very different data at the start of those columns than it does further down those columns, e.g. some of them contain mostly numeric data at the start... but in fact you want columns 2, 3, & 4 imported as text (because they all contain alphanumeric characters****).
So you need to tell READMATRIX that, e.g.:
unzip Myfile.zip
fnm = 'Myfile.csv';
obj = detectImportOptions(fnm);
obj = setvartype(obj,2:4,'string');
tbl = readtable(fnm,obj)
[~,ind] = unique(tbl(:,2:4),'stable');
t2 = tbl(ind,:)
And there is your "missing" data:
idx = all(t2{:,2:4}==["2","E","3E"],2);
t2(idx,:)
Just to confirm, lets check its location in the imported table:
idy = ind(idx)
tbl(idy,:)
And checking that line in the original file (don't forget the header is also one line):
So far everything looks as expected.
"matlab 'unique' is skipping rows with data"
So far I don't see any problem with UNIQUE.
** Apparently fewer than 79825:
*** Because otherwise people complain that file importing takes too long. This is a good example of John Lydgate's aphorism about pleasing all people all of the time.
**** Look at your table t: numeric columns cannot contain alphabetic characters. That should be the big clue for you, that you need to modify the file importing.
1 Comment
Stephen23
on 2 Jan 2024
In comparison, look at the columns of your table, what classes do columns 3 & 4 have? (hint: numeric).
When debugging always look at your data!
unzip Myfile.zip
t = readtable('Myfile.csv')
Star Strider
on 2 Jan 2024
I am not certain what the problem is, however if you want to test for more than one value in a row, specify that with the 'rows' argument. It will consider all the elements in a row (columns 2 to 4 in this instance) in its determination.
Try this —
Uz = unzip('Myfile.zip')
t=readtable(Uz{2}, 'VariableNamingRule','preserve')
[~,ind]=unique(t(:,2:4),'stable','rows');
t2=t(ind,:)
If you intend something else or want a different result, please provide more details.
.
2 Comments
Dyuman Joshi
on 2 Jan 2024
The output for a table without the 'rows' option is the same with the 'rows' option specified (see the description - https://in.mathworks.com/help/releases/R2019b/matlab/ref/unique.html?s_tid=doc_ta ), so that would not make a difference.
Star Strider
on 2 Jan 2024
That’s not how I read the section on Unique Rows in Matrix, although in this instance it is considering everything in the last three columns.
See Also
Categories
Find more on Logical in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!