MATLAB Answers

parallel processing for readtable function

3 views (last 30 days)
ps = parallel.Settings;
ps.Pool.AutoCreate = false; %do not autocreate parpool when encountering a |parfor|
ps.Pool.IdleTimeout = Inf; %do not shutdown parpool after Inf idle time
parfor j=1:10
tCOD{j,:}=readtable(full_file_name(j,:),'FileType','text', ...
'headerlines',end_of_header_line(j),'readvariablenames',0);
tCOD{j,:}=[];
end
When using 10 files with a size of about 100mb, only a couple of second differences are observed between for and parfor. Is there something wrong with the above steps for parfor. Is there any more convenient parallel processing approach that can be applied?
If the multiple "full_file_name(j,:)" can be read simultaneously (using multiple core) rather than sequentially, the speed of readtable can be increased significantly.

Accepted Answer

Edric Ellis
Edric Ellis on 19 Aug 2021
You've manually disabled the AutoCreate for parallel pools - I presume you're manually creating a pool with a separate parpool statement.
Whether or not parfor can go faster than a serial for loop depends on your underlying hardware for this case. It might simply be that the limiting factor is your disk drive, and trying to read from it multiple times simultaneously does not allow any speedup.
You could try manually running multiple copies of MATLAB and calling readtable from each of them simultaneously to see if they slow down when there is contention for disk access. I.e. run something like this in multiple copies of MATLAB:
while true
t = tic();
readtable(args..);
toc(t)
end
I suspect that you will see that as you start more copies of MATLAB, the toc time will increase.
  1 Comment
Walter Roberson
Walter Roberson on 19 Aug 2021
As a generalization: if you are not using server-quality hardware components, then optimal utilzation is typically two processes per controller. Not per drive . Any more and you are typically filling the memory channel between the controller and the rest of the hardware.
If you have only one drive per controller, then optimal utilization is typically two processes. Controllers can often be moving drive heads to position for the next read at the same time that the previous data is being transferred, so you want commands from multiple processes to be queued... but not too many processes.
The considerations are notably different for SSD.

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!