Nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing

Infrastructure & Networking Software And Drivers RDMA Software For GPU
1

Hello,

On my HGX GPU cluster I have the following error which occurs when the trainings start to run.
This causes problems with the reliability of the AI models.

Have you ever had this error? Do you have any ideas?

Thanks for help.

Best

[25033.266922] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[25033.280589] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[25033.300821] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[25033.320988] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[25033.342081] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[25033.360507] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[25033.380740] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[25033.400553] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[25033.420777] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[25033.440911] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[25033.461063] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[25033.481198] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[25033.501350] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing

modinfo nvidia_peermem
filename:       /lib/modules/5.15.0-1060-nvidia/updates/dkms/nvidia-peermem.ko
version:        535.183.01
license:        Linux-OpenIB
description:    NVIDIA GPU memory plug-in
author:         Yishai Hadas
srcversion:     14719BA3A7992087B56C615
depends:        nvidia,ib_core
retpoline:      Y
name:           nvidia_peermem
vermagic:       5.15.0-1060-nvidia SMP mod_unload modversions 
sig_id:         PKCS#7
signer:         cloudvm Secure Boot Module Signature key
sig_key:        51:5C:6A:D4:65:84:FC:3D:1F:16:36:3B:C0:BE:11:38:40:66:2B:40
sig_hashalgo:   sha512
signature:      AB:28:07:0D:59:AC:6C:F5:34:44:57:F7:AC:83:62:48:99:B7:90:71:
                A4:E8:1E:D7:21:94:6D:80:69:6B:19:00:D3:2A:D5:A8:CA:B4:CE:3C:
                54:FC:CD:05:85:DA:39:58:E7:47:45:3E:88:F6:9B:8C:FF:62:75:7E:
                0E:70:D1:0D:7B:28:AF:9B:FC:CE:8E:5A:69:63:55:5E:F2:69:A6:8B:
                8C:7D:7D:E5:EA:0C:1F:3C:A7:F6:6E:DA:8B:D0:EA:4D:65:CE:24:2B:
                31:5B:00:32:A6:D2:8C:C9:AB:40:42:EF:82:B3:A7:9F:93:7C:2E:9A:
                3F:C9:43:B6:99:B8:F6:11:62:C5:70:C9:BC:7B:5B:E7:9E:38:9F:91:
                8C:A4:91:B9:7A:31:2B:4D:76:3B:94:47:3D:9A:08:12:54:A8:7A:B8:
                AE:A7:E2:E6:29:DA:3F:30:5F:CA:79:6C:E1:95:82:EF:E4:B9:48:ED:
                AF:6E:62:94:F3:F6:93:73:C2:11:72:E0:A2:9A:CF:38:81:28:57:F6:
                B2:CB:E0:FF:A2:20:4B:7B:16:30:0F:7F:4B:51:91:D2:2E:53:D7:71:
                66:C0:4A:0B:D6:E6:BC:DC:04:51:6D:6E:92:DA:4B:F1:EA:D2:37:9A:
                AE:3C:F1:83:AC:7B:C1:35:2D:EF:E7:2A:92:E7:AB:D7
parm:           peerdirect_support:Set level of support for Peer-direct, 0 [default] or 1 [legacy, for example MLNX_OFED 4.9 LTS] (int)

nvidia-smi 
Fri Aug 30 16:55:07 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA H100 80GB HBM3          Off | 00000000:0A:00.0 Off |                    0 |
| N/A   30C    P0              71W / 700W |      0MiB / 81559MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA H100 80GB HBM3          Off | 00000000:18:00.0 Off |                    0 |
| N/A   26C    P0              71W / 700W |      0MiB / 81559MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA H100 80GB HBM3          Off | 00000000:23:00.0 Off |                    0 |
| N/A   25C    P0              70W / 700W |      0MiB / 81559MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA H100 80GB HBM3          Off | 00000000:2C:00.0 Off |                    0 |
| N/A   30C    P0              71W / 700W |      0MiB / 81559MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   4  NVIDIA H100 80GB HBM3          Off | 00000000:87:00.0 Off |                    0 |
| N/A   30C    P0              70W / 700W |      0MiB / 81559MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   5  NVIDIA H100 80GB HBM3          Off | 00000000:90:00.0 Off |                    0 |
| N/A   25C    P0              70W / 700W |      0MiB / 81559MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   6  NVIDIA H100 80GB HBM3          Off | 00000000:B8:00.0 Off |                    0 |
| N/A   25C    P0              71W / 700W |      0MiB / 81559MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   7  NVIDIA H100 80GB HBM3          Off | 00000000:C1:00.0 Off |                    0 |
| N/A   30C    P0              72W / 700W |      0MiB / 81559MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

Operating System: Ubuntu 22.04.4 LTS              
Kernel: Linux 5.15.0-1060-nvidia
ofed_info -s
MLNX_OFED_LINUX-23.10-1.1.9.0
2

Hi hme_ses,

Thank you for posting your query on NVIDIA community.

The issue posted requires troubleshooting and potential internal escalation. A valid NVAIE entitlement will be required for the same.

If there an active entitlement/support contract in place, please do not hesitate to open a support ticket by emailing enterprisesupport@nvidia.com

For contracts, please reach out to Networking-Contracts@nvidia.com

Thanks,
Namrata.

3

hi @namrata1,
Thanks for your reply.
yes you are right I’ll open a case directly on NVIDIA Support.
Best regards

4

we have a small cluster of 32 gpu (4 nodes) and are getting the same eerror - any updates?

5

Those kernel logs are harmless. See nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing · Issue #120 · Mellanox/nv_peer_memory · GitHub

6

Hello @drossetti,
thanks for your answer.

We’ve had NVIDIA driver crashes, and we saw a lot of these kernel logs before the crash.

And the second point is the quality of the results computed by the GPUs, with so many errors (sometimes as many as 20k messages in the logs) we also wondered if this altered the quality of the generated models.

Anyway, we’ll try an update again

I’ll keep you posted

thanks again

7

Hi, I recently encountered a similar issue, but my driver version is higher than 535; it is 550.90.07, and my CUDA version is 12.4.

nvidia-smi
Fri Aug  5 11:29:08 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H200                    On  |   00000000:18:00.0 Off |                    0 |
| N/A   59C    P0            642W /  700W |   75084MiB / 143771MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H200                    On  |   00000000:3A:00.0 Off |                    0 |
| N/A   67C    P0            646W /  700W |   75334MiB / 143771MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA H200                    On  |   00000000:4B:00.0 Off |                    0 |
| N/A   66C    P0            647W /  700W |   74884MiB / 143771MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA H200                    On  |   00000000:5C:00.0 Off |                    0 |
| N/A   56C    P0            644W /  700W |   75116MiB / 143771MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA H200                    On  |   00000000:9A:00.0 Off |                    0 |
| N/A   54C    P0            646W /  700W |   74984MiB / 143771MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA H200                    On  |   00000000:BA:00.0 Off |                    0 |
| N/A   64C    P0            630W /  700W |   75094MiB / 143771MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA H200                    On  |   00000000:CA:00.0 Off |                    0 |
| N/A   65C    P0            638W /  700W |   74926MiB / 143771MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA H200                    On  |   00000000:DA:00.0 Off |                    0 |
| N/A   59C    P0            628W /  700W |   75176MiB / 143771MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A    185474      C   /usr/bin/python                             75064MiB |
|    1   N/A  N/A    185475      C   /usr/bin/python                             75314MiB |
|    2   N/A  N/A    185476      C   /usr/bin/python                             74864MiB |
|    3   N/A  N/A    185477      C   /usr/bin/python                             75096MiB |
|    4   N/A  N/A    185478      C   /usr/bin/python                             74964MiB |
|    5   N/A  N/A    185479      C   /usr/bin/python                             75074MiB |
|    6   N/A  N/A    185480      C   /usr/bin/python                             74906MiB |
|    7   N/A  N/A    185481      C   /usr/bin/python                             75156MiB |
+-----------------------------------------------------------------------------------------+

modinfo nvidia_peermem
filename:       /lib/modules/5.14.0-284.25.1.el9_2.x86_64/kernel/drivers/video/nvidia-peermem.ko
version:        550.90.07
license:        Linux-OpenIB
description:    NVIDIA GPU memory plug-in
author:         Yishai Hadas
rhelversion:    9.2
srcversion:     4F8B460B3801C5451579324
depends:        nvidia,ib_core
retpoline:      Y
name:           nvidia_peermem
vermagic:       5.14.0-284.25.1.el9_2.x86_64 SMP preempt mod_unload modversions 
parm:           peerdirect_support:Set level of support for Peer-direct, 0 [default] or 1 [legacy, for example MLNX_OFED 4.9 LTS] (int)


[alice@rocky ~]$ cat /etc/os-release 
NAME="Rocky Linux"
VERSION="9.2 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.2"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.2 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.2"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.2"

[alice@rocky ~]$ uname -a
Linux rocky 5.14.0-284.25.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 2 14:53:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

During the training workload, the process is aborted. Currently, I have collected the following dmesg information, and it is accompanied by a use-after-free error:

[4382693.454116] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4382693.454221] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4382693.454364] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4382693.454468] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4382693.454568] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4382693.454711] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4382693.454888] BUG: KFENCE: use-after-free read in nv_get_p2p_free_callback+0x2b/0xc0 [nvidia_peermem]
[4382693.455176]  nv_get_p2p_free_callback+0x2b/0xc0 [nvidia_peermem]
[4382693.455295]  nv_p2p_mem_info_free_callback+0x12/0x30 [nvidia]
[4382693.455722]  _nv000083rm+0x59/0x130 [nvidia]
[4382693.456676]  nvidia_ioctl.isra.0+0x82/0x7d0 [nvidia]
[4382693.457033]  nvidia_unlocked_ioctl+0x1e/0x30 [nvidia]
[4382693.458491]  nvidia_ioctl.isra.0+0x13c/0x7d0 [nvidia]
[4382693.458869]  nvidia_unlocked_ioctl+0x1e/0x30 [nvidia]
[4382693.461482] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4382693.461866] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4382693.462308] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4382693.462648] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4382693.462995] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4382693.463382] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4382693.463724] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing

It should be noted that the point where the error occurs is not fixed and can be like the following:

[4034758.972940] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4034758.973029] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4034758.973159] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4034758.973255] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4034758.973352] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4034758.973473] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4034758.973570] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4034758.973657] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4034758.973786] ==================================================================
[4034758.973856] BUG: KFENCE: use-after-free read in nv_get_p2p_free_callback+0x2b/0xc0 [nvidia_peermem]

[4034758.974011] Use-after-free read at 0x00000000623db960 (in kfence-#182):
[4034758.974101]  nv_get_p2p_free_callback+0x2b/0xc0 [nvidia_peermem]
[4034758.974198]  nv_p2p_mem_info_free_callback+0x12/0x30 [nvidia]
[4034758.974680]  _nv000083rm+0x59/0x130 [nvidia]

[4034758.975175] kfence-#182: 0x0000000032269aec-0x00000000cf82b3e2, size=56, cache=kmalloc-64

[4034758.975461] allocated by task 3888320 on cpu 50 at 4034751.211360s:
[4034758.975626]  mlx5_alloc_cmd_msg+0x3d/0x1d0 [mlx5_core]
[4034758.975860]  cmd_exec+0x398/0x420 [mlx5_core]
[4034758.976075]  mlx5_cmd_do+0x1e/0x40 [mlx5_core]
[4034758.976282]  mlx5_cmd_exec+0x18/0x30 [mlx5_core]
[4034758.976493]  mlx5_core_destroy_qp+0xe1/0x110 [mlx5_ib]
[4034758.976696]  destroy_qp_common+0x294/0x3e0 [mlx5_ib]
[4034758.976892]  mlx5_ib_destroy_qp+0x28/0x40 [mlx5_ib]
[4034758.977089]  ib_destroy_qp_user+0x8d/0x1b0 [ib_core]
[4034758.977305]  uverbs_free_qp+0x35/0x90 [ib_uverbs]
[4034758.977513]  destroy_hw_idr_uobject+0x1b/0x50 [ib_uverbs]
[4034758.977725]  uverbs_destroy_uobject+0x34/0x1c0 [ib_uverbs]
[4034758.977941]  __uverbs_cleanup_ufile+0xbe/0x130 [ib_uverbs]
[4034758.978160]  uverbs_destroy_ufile_hw+0x38/0xf0 [ib_uverbs]
[4034758.978384]  ib_uverbs_close+0x1f/0xb0 [ib_uverbs]
[4034758.978612]  __fput+0x91/0x250
[4034758.978843]  task_work_run+0x59/0x90
[4034758.979078]  do_exit+0x262/0x4a0
[4034758.979314]  do_group_exit+0x2d/0x90
[4034758.979553]  get_signal+0x9d3/0xa00
[4034758.979796]  arch_do_signal_or_restart+0x25/0x100
[4034758.980046]  exit_to_user_mode_loop+0x9c/0x130
[4034758.980306]  exit_to_user_mode_prepare+0xb6/0x100
[4034758.980574]  syscall_exit_to_user_mode+0x12/0x30
[4034758.980848]  do_syscall_64+0x69/0x90
[4034758.981133]  entry_SYSCALL_64_after_hwframe+0x63/0xcd

[4034758.981679] freed by task 3888320 on cpu 50 at 4034751.211888s:
[4034758.982082]  cmd_exec+0x2f3/0x420 [mlx5_core]
[4034758.982396]  mlx5_cmd_do+0x1e/0x40 [mlx5_core]
[4034758.982739]  mlx5_cmd_exec+0x18/0x30 [mlx5_core]
[4034758.983064]  mlx5_core_destroy_qp+0xe1/0x110 [mlx5_ib]
[4034758.983377]  destroy_qp_common+0x294/0x3e0 [mlx5_ib]
[4034758.983691]  mlx5_ib_destroy_qp+0x28/0x40 [mlx5_ib]
[4034758.984009]  ib_destroy_qp_user+0x8d/0x1b0 [ib_core]
[4034758.984335]  uverbs_free_qp+0x35/0x90 [ib_uverbs]
[4034758.984665]  destroy_hw_idr_uobject+0x1b/0x50 [ib_uverbs]
[4034758.985012]  uverbs_destroy_uobject+0x34/0x1c0 [ib_uverbs]
[4034758.985350]  __uverbs_cleanup_ufile+0xbe/0x130 [ib_uverbs]
[4034758.985670]  uverbs_destroy_ufile_hw+0x38/0xf0 [ib_uverbs]
[4034758.985977]  ib_uverbs_close+0x1f/0xb0 [ib_uverbs]
[4034758.986282]  __fput+0x91/0x250
[4034758.986578]  task_work_run+0x59/0x90
[4034758.986874]  do_exit+0x262/0x4a0
[4034758.987165]  do_group_exit+0x2d/0x90
[4034758.987454]  get_signal+0x9d3/0xa00
[4034758.987736]  arch_do_signal_or_restart+0x25/0x100
[4034758.988015]  exit_to_user_mode_loop+0x9c/0x130
[4034758.988292]  exit_to_user_mode_prepare+0xb6/0x100
[4034758.988564]  syscall_exit_to_user_mode+0x12/0x30
[4034758.988834]  do_syscall_64+0x69/0x90
[4034758.989113]  entry_SYSCALL_64_after_hwframe+0x63/0xcd

[4034758.989690] CPU: 31 PID: 3888245 Comm: pt_data_worker Kdump: loaded Tainted: P    B      OE    --------  ---  5.14.0-284.25.1.el9_2.x86_64 #1
[4034758.990273] Hardware name: ASUSTeK COMPUTER INC. ESC N8-E11V/Z13PN-D32 Series, BIOS 2602 10/25/2024
[4034758.990581] ==================================================================
[4034758.990902] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4034758.991273] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4034758.991608] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4034758.991975] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4034758.992305] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4034758.992610] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing
[4034758.992964] nvidia-peermem nv_get_p2p_free_callback:127 ERROR detected invalid context, skipping further processing