Skip to content

Reset only GPUs with pending MIG mode changes - #525

Closed
aryangorwade wants to merge 1 commit into
NVIDIA:mainfrom
aryangorwade:mig-manager-reset-affected-gpus
Closed

aryangorwade wants to merge 1 commit into
NVIDIA:mainfrom
aryangorwade:mig-manager-reset-affected-gpus

Conversation

@aryangorwade

@aryangorwade aryangorwade commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Addresses #357

ApplyMigMode currently sets the MIG mode only on GPUs selected by the config, but then resets every GPU on the node if any mode change is pending. This interrupts workloads on GPUs the config never touched. This PR collects the indices of GPUs whose mode change is pending and resets only those: through nvidia-smi when the nvidia module is loaded, or through a PCI reset when it is not.

Verification

Customer sequence reproduced on two A100 PCIe 40 GB GPUs:

• GPU 0 configured with two 3g.20gb MIG instances.
• A live CUDA context running inside a GPU 0 MIG instance.
• GPU 1 initially MIG-disabled.
• Applied a GPU-1-only profile enabling MIG and creating two 3g.20gb instances.

Result: success. The GPU 0 workload remained alive, its MIG UUIDs were unchanged, and GPU 1 received the requested configuration.

Unit tests

The changes directly construct nvml.New(), nvpci.New(), and exec.Command(.... mock-nvml cannot fully test this. Not sure if this can be adequately unit-tested. There were no existing test cases around this.

}

func ResetAllGPUs() (string, error) {
func ResetGPUs(devicesToReset []int) (string, error) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of changing an exported function, keep the existing ResetAllGPUs and add a new ResetGPUs(devices []int) function.

@aryangorwade aryangorwade Sep 29, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we still want to keep it even though it is not being used anywhere else because it's exported.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Essentially, yes. I doubt anyone is using functions from the cmd/ package, but we should avoid making breaking changes if we can.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Added it back as a thin wrapper of ResetGPUs (since ResetAllGPUs's underlying methods were changed to take in a list as well)

Signed-off-by: Aryan Gorwade <agorwade@nvidia.com>
@aryangorwade
aryangorwade force-pushed the mig-manager-reset-affected-gpus branch from a949536 to 90e8b92 Compare September 29, 2026 21:49
@aryangorwade
aryangorwade marked this pull request as draft October 6, 2026 21:20
@rajathagasthya

Copy link
Copy Markdown
Contributor

Thanks for the effort @aryangorwade! We don't want to do this because of reasons mentioned in #357 (comment).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants