Repository navigation
Reset only GPUs with pending MIG mode changes - #525
aryangorwade wants to merge 1 commit into
Conversation
| } | ||
|
|
||
| func ResetAllGPUs() (string, error) { | ||
| func ResetGPUs(devicesToReset []int) (string, error) { |
There was a problem hiding this comment.
Instead of changing an exported function, keep the existing ResetAllGPUs and add a new ResetGPUs(devices []int) function.
There was a problem hiding this comment.
So we still want to keep it even though it is not being used anywhere else because it's exported.
There was a problem hiding this comment.
Essentially, yes. I doubt anyone is using functions from the cmd/ package, but we should avoid making breaking changes if we can.
There was a problem hiding this comment.
Sounds good. Added it back as a thin wrapper of ResetGPUs (since ResetAllGPUs's underlying methods were changed to take in a list as well)
Signed-off-by: Aryan Gorwade <agorwade@nvidia.com>
a949536 to
90e8b92
Compare
|
Thanks for the effort @aryangorwade! We don't want to do this because of reasons mentioned in #357 (comment). |
Addresses #357
ApplyMigModecurrently sets the MIG mode only on GPUs selected by the config, but then resets every GPU on the node if any mode change is pending. This interrupts workloads on GPUs the config never touched. This PR collects the indices of GPUs whose mode change is pending and resets only those: throughnvidia-smiwhen the nvidia module is loaded, or through a PCI reset when it is not.Verification
Customer sequence reproduced on two A100 PCIe 40 GB GPUs:
• GPU 0 configured with two 3g.20gb MIG instances.
• A live CUDA context running inside a GPU 0 MIG instance.
• GPU 1 initially MIG-disabled.
• Applied a GPU-1-only profile enabling MIG and creating two 3g.20gb instances.
Result: success. The GPU 0 workload remained alive, its MIG UUIDs were unchanged, and GPU 1 received the requested configuration.
Unit tests
The changes directly construct
nvml.New(),nvpci.New(), andexec.Command(....mock-nvmlcannot fully test this. Not sure if this can be adequately unit-tested. There were no existing test cases around this.