Sudo nvidia-smi -lgc -lmc

Accelerated Computing CUDA CUDA Programming and Performance
1

Translation:

I set nvidia-smi -lmc 9751 and nvidia-smi -lgc 900, but when actually running:

liyanru@longjing-2:~$ nvidia-smi -q -d CLOCK -i 0

==============NVSMI LOG==============

Timestamp                                 : Thu Feb 29 13:56:22 2024
Driver Version                            : 535.161.07
CUDA Version                              : 12.2

Attached GPUs                             : 4
GPU 00000000:18:00.0
    Clocks
        Graphics                          : 900 MHz
        SM                                : 900 MHz
        Memory                            : 9501 MHz
        Video                             : 1470 MHz
    Applications Clocks
        Graphics                          : N/A
        Memory                            : N/A
    Default Applications Clocks
        Graphics                          : N/A
        Memory                            : N/A
    Deferred Clocks
        Memory                            : N/A
    Max Clocks
        Graphics                          : 2100 MHz
        SM                                : 2100 MHz
        Memory                            : 9751 MHz
        Video                             : 1950 MHz
    Max Customer Boost Clocks
        Graphics                          : N/A
    SM Clock Samples
        Duration                          : Not Found
        Number of Samples                 : Not Found
        Max                               : Not Found
        Min                               : Not Found
        Avg                               : Not Found
    Memory Clock Samples
        Duration                          : Not Found
        Number of Samples                 : Not Found
        Max                               : Not Found
        Min                               : Not Found
        Avg                               : Not Found
    Clock Policy
        Auto Boost                        : N/A
        Auto Boost Default                : N/A

I checked nvidia-smi -q -d SUPPORTED_CLOCKS:

GPU 00000000:18:00.0
    Supported Clocks
        Memory                            : 9751 MHz
            Graphics                      : 2100 MHz
            ...
            Graphics                      : 900 MHz
        Memory                            : 9501 MHz

It seems to be supported.

2

The (abbreviated) output with SUPPORTED_CLOCKS seems to suggest that the combination {m=9751, g=900} should work, as you note. But you observe that this is not the case.

You would want to set graphics and memory clocks in a single invocation of nvidia-smi to make absolutely sure that only valid clock combinations are requested.

In addition, there is always a possibility that the output SUPPORTED_CLOCKS is not entirely correct. Setting one clock very high and the other very low would seem to be a combination that is more difficult to support (= make to work reliably).

Can you successfully set memory clock to 9751 in conjunction with the highest supported graphics clock? If so, try reducing the graphics clock one step at a time, and see how low you can set it. If the memory clock cannot be set to 9751 with any graphics clock setting, try the next lower memory clock.