Clear Filters
Clear Filters

Is there a utility like nvidia-smi within matlab to determine which gpus are in current use?

8 views (last 30 days)
10 GPUs, 4 users. Want to avoid usage conflict.

Accepted Answer

Walter Roberson
Walter Roberson on 10 Oct 2023
Edited: Walter Roberson on 10 Oct 2023
No, there is no utility like that built into MATLAB.
My understanding is that the order of GPUs in the gpu device list depends upon the order in which the GPUs answered the probes at boot time, and so is potentially variable. However, if I understand correctly it is consistent during any particular boot session.
Therefore, if you can get the cooperation of all users, you could use simple resource management techniques such as creating a binary file with 10 entries in it that record whether a program is requesting to use the given GPU. If you were to write process IDs into the file, then you could have a management layer that checked whether the recorded process still existed and if not then freed up the entry.
  2 Comments
Sam Marshalik
Sam Marshalik on 10 Oct 2023
In combination with Walter's response, here is an idea.
Big disclaimer: Keep in mind this is not a perfect solution, as the information you get from nvidia-smi or gpuDevice will be stale after you retrieve it and may not be utilized in that momment in time but is being used by someone.
You can use the gpuDevice command in MATLAB to view GPU resources. You can use view memory usage to see if a GPU is being used in that momment (snippit from gpuDevice)
TotalMemory: 2147287040 (2.15 GB)
AvailableMemory: 1615209678 (1.62 GB)
You can also use the system command to call nvidia-smi via MATLAB. You can then use MATLAB text parsing to programmatically look at usage, get a handle to that GPU (have not played with this, but assuming it is possible), and use that GPU.

Sign in to comment.

More Answers (1)

Joss Knight
Joss Knight on 11 Oct 2023
gpuDeviceTable is the utility for listing your devices and their properties. You can also see all your devices in the Parallel menu on the Home toolstrip.
The menu entry will also tell you when the device was last used by your MATLAB, so you can check whether your code is successfully accessing the correct GPU.
The simplest way to control GPU access to multiple users is to put the devices into Exclusive Process compute mode, which means that only one user will be able to select each device at a time. You can use a command like "nvidia-smi -i 0 -c 3" to set the compute mode.
For a more sophisticated automated management of which user has access to which GPU, you will need a more sophisticated scheduling or resource management system. This can use environment variables like CUDA_VISIBLE_DEVICES to ensure that each user starting a process on a resource accesses a particular GPU only.

Tags

Products


Release

R2023b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!