Loren's blog on datastore. How should I use counterSize and done?

2 views (last 30 days)
gujax
gujax on 28 Nov 2021
Answered: gujax on 29 Nov 2021
I am trying to use datastore in Matlab v2018b but it does not have 'partialread'.
I am trying to replicate Loren's code (https://blogs.mathworks.com/loren/2019/05/29/big-data-in-mat-files/ and the section 'Use a portion of all variables') for a large variable whose size exceeds memory.
Unfortunately its written kind of poorly with many variables thrown in and not explained. For example, I cannot figure out how to use 'done' and 'counterSize'.
I tried
[fds_partial, counter, done] = fileDatastore("smallVars*.mat", "ReadFcn", @partialReadFcn, "UniformRead", true);
But I get an error of 'too many output arguments'
I tried
[M, counter, done] = read(fds_partial) %with
fds_partial = fileDatastore("smallVars*.mat", "ReadFcn", @partialReadFcn, "UniformRead", true);
Again I get an error of too many output arguments.
So I end up reading only one block of file and the readfcn function does not proceed beyond one block.
So what gives?

Answers (2)

Adam Danz
Adam Danz on 28 Nov 2021
> I am trying to use datastore in Matlab v2018b but it does not have 'partialread'.
partialreadFcn is a function defined in Loren's blog post. No currently existing release of Matlab has a "partialread" function.
> I cannot figure out how to use 'done' and 'counterSize'.
done appears to be a flag (true|false) indicating whether there is more to read or not.
There is no variable named counteSize.
> But I get an error of 'too many output arguments'
fileDatastore only has 1 output.
doc fileDatastore

gujax
gujax on 29 Nov 2021
I think I figured out some of the answers...
I need later versions of Matlab to get 'partial read' enabled using 'read mode'.
Also though 'done' and 'readcounter' can used in the partialread function, I am still not sure where these two variables get outputed and how I should be using them.
So its kind of useless to go through filedatastore when one can do this simply by re-reading partial files. Not sure how all this helps.

Products


Release

R2018b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!