Skip to content

[Bug]: assert fails after apply when a profile is written with its compute slice count (1c.1g.5gb) #587

Closed as not planned

Description

@0z5a

Motivation

This was found by reading the code with an AI assistant (Claude Code, claude-fable-5-2) and confirmed with the NVML mock the unit tests use (go-nvml dgxa100). It was not hit on real hardware in this investigation; it is a consistency problem between what validation accepts and what assert compares. The repo itself uses the long spelling (types.MigProfile doc comment, pkg/mig/config/known_configs.go).

Check here if you would like to work on this issue

  • I would like to work on this issue.

Describe the bug

1c.1g.5gb and 1g.5gb name the same MIG profile (so do 2c.2g.10gb / 2g.10gb, 7c.7g.40gb / 7g.40gb, with or without +me). Both pass config validation, and apply creates the right devices for either spelling. The doc comment of types.MigProfile lists "1c.1g.5gb+me" as an example, and pkg/mig/config/known_configs.go uses the long spelling for the A100 device types.

The current MIG config is always reported under the short name, and MigConfig.Equals compares the names as strings. So for a config that uses the long spelling:

  • nvidia-mig-parted assert fails right after a successful apply of the same config;
  • every later apply clears and re-creates the MIG devices instead of skipping the GPU, and runs the pre-apply-config hook again.

From reading pkg/mig/reconfigure/reconfigure.go (isConfigAlreadyApplied calls assert), mig-manager would also reconfigure such a GPU every time it handles the label. That part was not run.

To Reproduce

On main at 80044e1, with the NVML mock the unit tests use (no GPU needed). Save as _repro/main.go in a checkout and run go run ./_repro:

package main

import (
	"fmt"

	"github.com/NVIDIA/go-nvml/pkg/nvml"
	"github.com/NVIDIA/go-nvml/pkg/nvml/mock/dgxa100"

	"github.com/NVIDIA/mig-parted/pkg/mig/config"
	"github.com/NVIDIA/mig-parted/pkg/types"
)

func main() {
	types.SetMockNVdevlib()
	server := dgxa100.New() // A100-SXM4-40GB
	server.Devices[0].SetMigMode(nvml.DEVICE_MIG_ENABLE)
	manager := config.NewNvmlMigConfigManager(server)

	desired := types.MigConfig{"1c.1g.5gb": 2, "2c.2g.10gb": 1}
	fmt.Println("valid format:", desired.AssertValidFormat())
	fmt.Println("SetMigConfig error:", manager.SetMigConfig(0, desired))

	current, _ := manager.GetMigConfig(0)
	fmt.Println("current:", current)
	// The comparison made by 'assert' and by 'apply'.
	fmt.Println("current.Equals(desired):", current.Equals(desired))
}

Output:

valid format: <nil>
SetMigConfig error: <nil>
current: map[1g.5gb:2 2g.10gb:1]
current.Equals(desired): false

The equivalent config file entry is:

    mig-devices:
      "1c.1g.5gb": 2
      "2c.2g.10gb": 1

Expected behavior

current.Equals(desired) is true, so assert passes after apply and a second apply leaves the GPU alone.

A tested patch exists that compares the two configs by canonical profile name (the form go-nvlib's MigProfileInfo.String() prints), similar in spirit to #489. The alternative would be to reject the long spelling in AssertValidFormat, which would contradict the MigProfile doc comment and known_configs.go. Which do you prefer?

Environment (please provide the following information):

  • mig-parted version: main at 80044e1 (after v0.15.1)
  • Host OS: reproduced with the Go unit-test mock on macOS (arm64) and Linux (x86_64); no real GPU involved
  • Kernel Version: n/a (mock)
  • NVIDIA Driver Version: n/a (mock)
  • GPU Model(s): go-nvml dgxa100 mock (A100-SXM4-40GB)

This report was prepared with an AI coding assistant (Claude Code, claude-fable-5-2); the reproducer was run with the go-nvml mock as shown above.

Activity

  1. rajathagasthya commented on Nov 17, 2026

    @rajathagasthya
    Contributor

    Thanks for the report. We do not plan to change this behavior. When the compute slice count equals the GPU instance slice count, the canonical profile name is the short form (1g.5gb, 2g.10gb). assert and apply compare against that form, so please use it in your config. Profiles with fewer compute slices than the GPU instance (e.g. 1c.2g.10gb) are not a supported configuration in the GPU Operator and we don't plan to add support for them. Closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions