Slow access of large composite element
1 view (last 30 days)
Show older comments
I am using spmd to run some parallel algorithms which periodically communicate. At the end, each process has a structure which I would like to pull back to the client for further analysis. Unfortunately, the size of the structure is large (but I don't think they are excessive), and trying to pull just one of the composite elements leads to a timeout of the cluster (30 min) and therefore fails. Is there any way to have the client as one of the spmd processes, or a more efficient way to transfer data from one or more of the workers back to the client in a more efficient way?
EDIT:
A clunky workaround is to have each process save their workspace to a directory. This seems to work pretty quickly, but it's obnoxious to have to import all the workspaces to do the diagnostics. I would appreciate more elegant solutions.
5 Comments
Walter Roberson
on 9 Feb 2018
Looking more closely, it is for "communicating jobs", of which spmd is one kind.
Accepted Answer
More Answers (0)
See Also
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!