AMP samples are not starting "an illegal memory access was encountered"

hey! I can not run any AMP samples in IsaacGymEnvs.
After folloving the commands from rl-examles docs, non of the samples working
all giving the same issue

[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: …/…/…/source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 4202
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: …/…/…/source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 4210
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: …/…/…/source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3480
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: …/…/…/source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3535
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: …/…/…/source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 5993
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: …/…/…/source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 937
[Error] [carb.gym.plugin] Failed to fill root state tensor

I found the main issue. This is because my gpu’s 6gb ram is not enough for most of the large gpu memory allocation.
from HumanoidAMP.yaml decreased
#numEnvs: ${resolve_default:4096,${…num_envs}}
numEnvs: ${resolve_default:200,${…num_envs}}

from HumanoidAMPPPO.yaml decreased
#minibatch_size: 32768
minibatch_size: 32

#amp_minibatch_size: 4096
amp_minibatch_size: 16

now they are working without any crash and error

releted topics :
1)Gym cuda error: running out of memory - #3 by jimingre
2)AssertionError

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.