Hi guys,
I faced the followed error. Could you help me to find where the problems come from? I use personal-defined five-finger hand for the task of in-hand manipulation.
I think the errors come from the usd model, but didn’t figure out it.
I changed the minibatch_size to 1 and num_envs to 1
2022-11-25 13:05:10 [10,052ms] [Warning] [omni.hydra] Mesh /World/envs/env_0/_Phadist/visuals_xform/visuals has corrupted data in primvar st_9: buffer size 643 doesn't match expected size 10000 in faceVarying primvars
... lots of similar warning
2022-11-25 13:05:10 [10,107ms] [Warning] [omni.physx.plugin] ParseFEMClothMaterial: UsdPrim doesn't have a PhysxSchemaPhysxDeformableSurfaceMaterialAPI
PxgCudaDeviceMemoryAllocator fail to allocate memory 2147483648 bytes!! Result = 2
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU cudaMainGjkEpa or prepareLostFoundPairs kernel fail!
700, FILE /buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 2165
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: Patch buffer overflow detected, please increase its size to at least 0 in the scene desc!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 2179
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 54
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 60
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: memcpy failed fail!
700, FILE /buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 2408
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 54
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 60
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 54
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 60
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 80
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 80
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU solveContactParallel fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1492
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU solveContactParallel fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1529
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1697
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 859
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 879
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 907
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1784
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1697
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 859
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 879
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 907
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1784
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1697
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 859
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 879
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 907
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1784
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1697
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,226ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 859
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 879
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 907
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1784
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1697
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 859
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 879
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 907
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1784
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1697
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 859
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 879
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 907
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1784
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1697
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 859
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 879
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 907
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1784
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1697
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 859
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 879
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 907
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1784
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU solveContactParallel fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1858
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 80
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: GPU integrateCoreParallel fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 2406
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 80
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 80
2022-11-25 13:05:11 [11,227ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,427ms] [Error] [omni.physx.plugin] PhysX error: PhysX Internal CUDA error. Simulation can not continue!, FILE /buildAgent/work/99bede84aa0a52c2/source/physx/src/NpScene.cpp, LINE 3509
2022-11-25 13:05:11 [11,427ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 146
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA context validation failed
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 193
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] Failed to initialize GPU simulation data
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 457
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 458
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 459
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 460
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 461
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 462
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 463
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 464
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 465
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 466
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 467
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 468
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 469
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 471
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 472
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 474
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 475
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 476
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 478
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 479
2022-11-25 13:05:11 [11,428ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 480
Error executing job with overrides: ['task=, 'num_envs=1']
Traceback (most recent call last):
File "./scripts/rlgames_train.py", line 138, in <module>
parse_hydra_configs()
File "/home/lz/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/python/lib/python3.7/site-packages/hydra/main.py", line 52, in decorated_main
config_name=config_name,
File "/home/lz/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/python/lib/python3.7/site-packages/hydra/_internal/utils.py", line 378, in _run_hydra
lambda: hydra.run(
File "/home/lz/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/python/lib/python3.7/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
raise ex
File "/home/lz/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/python/lib/python3.7/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
return func()
File "/home/lz/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/python/lib/python3.7/site-packages/hydra/_internal/utils.py", line 381, in <lambda>
overrides=args.overrides,
File "/home/lz/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/python/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 111, in run
_ = ret.return_value
File "/home/lz/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/python/lib/python3.7/site-packages/hydra/core/utils.py", line 233, in return_value
raise self._return_value
File "/home/lz/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/python/lib/python3.7/site-packages/hydra/core/utils.py", line 160, in run_job
ret.return_value = task_function(task_cfg)
File "./scripts/rlgames_train.py", line 104, in parse_hydra_configs
task = initialize_task(cfg_dict, env)
File "/home/lz/Documents/Code/omni_isaac_gym_repos/OmniIsaacGymEnvs/omniisaacgymenvs/utils/task_util.py", line 72, in initialize_task
env.set_task(task=task, sim_params=sim_config.get_physics_params(), backend="torch", init_sim=init_sim)
File "/home/lz/Documents/Code/omni_isaac_gym_repos/OmniIsaacGymEnvs/omniisaacgymenvs/envs/vec_env_rlgames.py", line 51, in set_task
super().set_task(task, backend, sim_params, init_sim)
File "/home/lz/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py", line 82, in set_task
self._world.reset()
File "/home/lz/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 281, in reset
SimulationContext.reset(self, soft=soft)
File "/home/lz/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py", line 423, in reset
SimulationContext.initialize_physics(self)
File "/home/lz/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py", line 403, in initialize_physics
self._physics_sim_view = omni.physics.tensors.create_simulation_view(self.backend)
File "/home/lz/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/extsPhysics/omni.physics.tensors-1.4.15-5.1/omni/physics/tensors/impl/api.py", line 11, in create_simulation_view
raise Exception("Failed to create simulation view backend")
Exception: Failed to create simulation view backend
/home/lz/.local/share/ov/pkg/isaac_sim-2022.1.1/python.sh: line 40: 3129 Segmentation fault (core dumped) $python_exe "$@" $args
There was an error running python