2023-08-25 04:17:24 [1,008,976ms] [Error] [omni.physx.plugin] PhysX error: memcpy failed fail! 2 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 1849
2023-08-25 04:17:24 [1,008,976ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,976ms] [Error] [omni.physx.plugin] PhysX error: GPU cudaMainGjkEpa or prepareLostFoundPairs kernel fail!
700, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 1859
2023-08-25 04:17:24 [1,008,976ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,976ms] [Error] [omni.physx.plugin] PhysX error: memcpy failed fail! 3 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 1739
2023-08-25 04:17:24 [1,008,976ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,977ms] [Error] [omni.physx.plugin] PhysX error: memcpy failed fail! 3 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 1761
2023-08-25 04:17:24 [1,008,977ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,977ms] [Error] [omni.physx.plugin] PhysX error: memcpy failed fail! 3 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 1739
2023-08-25 04:17:24 [1,008,977ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,977ms] [Error] [omni.physx.plugin] PhysX error: memcpy failed fail! 3 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 1761
2023-08-25 04:17:24 [1,008,977ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,977ms] [Error] [omni.physx.plugin] PhysX error: memcpy failed fail! 3 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 1739
2023-08-25 04:17:24 [1,008,977ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,978ms] [Error] [omni.physx.plugin] PhysX error: memcpy failed fail! 3 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 1761
2023-08-25 04:17:24 [1,008,978ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,978ms] [Error] [omni.physx.plugin] PhysX error: memcpy failed fail! 3 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 1739
2023-08-25 04:17:24 [1,008,978ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,978ms] [Error] [omni.physx.plugin] PhysX error: memcpy failed fail! 3 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 1761
2023-08-25 04:17:24 [1,008,978ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,979ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpucommon/include/PxgCudaUtils.h, LINE 53
2023-08-25 04:17:24 [1,008,979ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,979ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpucommon/include/PxgCudaUtils.h, LINE 59
2023-08-25 04:17:24 [1,008,979ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,979ms] [Error] [omni.physx.plugin] PhysX error: memcpy failed fail!
700, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 2077
2023-08-25 04:17:24 [1,008,979ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,980ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpucommon/include/PxgCudaUtils.h, LINE 53
2023-08-25 04:17:24 [1,008,980ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,980ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpucommon/include/PxgCudaUtils.h, LINE 59
2023-08-25 04:17:24 [1,008,980ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,980ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpucommon/include/PxgCudaUtils.h, LINE 53
2023-08-25 04:17:24 [1,008,980ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,981ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpucommon/include/PxgCudaUtils.h, LINE 59
2023-08-25 04:17:24 [1,008,981ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,981ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpucommon/include/PxgCudaUtils.h, LINE 75
2023-08-25 04:17:24 [1,008,981ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,981ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpucommon/include/PxgCudaUtils.h, LINE 81
2023-08-25 04:17:24 [1,008,981ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,982ms] [Error] [omni.physx.plugin] PhysX error: GPU artiContactConstraintPrepare fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1169
2023-08-25 04:17:24 [1,008,982ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,982ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpucommon/include/PxgCudaUtils.h, LINE 75
2023-08-25 04:17:24 [1,008,982ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,983ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpucommon/include/PxgCudaUtils.h, LINE 81
2023-08-25 04:17:24 [1,008,983ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,983ms] [Error] [omni.physx.plugin] PhysX error: GPU solveContactParallel fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1462
2023-08-25 04:17:24 [1,008,983ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,983ms] [Error] [omni.physx.plugin] PhysX error: GPU solveContactParallel fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1499
2023-08-25 04:17:24 [1,008,983ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,984ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,984ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,984ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,984ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,984ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,984ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,984ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,984ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,985ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,985ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,985ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,985ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,985ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,985ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,985ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,985ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,986ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,986ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,986ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,986ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,986ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,986ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,987ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,987ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,987ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,987ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,987ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,987ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,987ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,987ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,988ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,988ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,988ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,988ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,988ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,988ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,988ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,989ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,989ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,989ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,989ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,989ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,989ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,989ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,990ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,990ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,990ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,990ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,990ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1695
2023-08-25 04:17:24 [1,008,990ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,991ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 845
2023-08-25 04:17:24 [1,008,991ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,991ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 865
2023-08-25 04:17:24 [1,008,991ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,991ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 893
2023-08-25 04:17:24 [1,008,991ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,991ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1782
2023-08-25 04:17:24 [1,008,991ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,991ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,991ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,991ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,991ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,991ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,991ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,992ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,992ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,992ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,992ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,992ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,992ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,992ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,992ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,992ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,992ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,992ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,992ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,992ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,992ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,993ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,993ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,993ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,993ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,993ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,993ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,993ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,993ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,993ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,993ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,993ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,993ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,993ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,993ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1695
2023-08-25 04:17:24 [1,008,994ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,995ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 845
2023-08-25 04:17:24 [1,008,995ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,995ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 865
2023-08-25 04:17:24 [1,008,995ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,995ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 893
2023-08-25 04:17:24 [1,008,995ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,995ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1782
2023-08-25 04:17:24 [1,008,995ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,995ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,995ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,995ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,995ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,995ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,995ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,996ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,996ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,996ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,996ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,996ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,996ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,996ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,996ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,996ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,996ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,996ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,996ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,996ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,996ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,997ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,997ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,997ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,997ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,997ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,997ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,997ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,997ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,997ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,997ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,997ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,997ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,997ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,997ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,998ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,998ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,998ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,998ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,998ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,998ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,998ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,998ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,998ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,998ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,998ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,998ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,998ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,008,998ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,999ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1695
2023-08-25 04:17:24 [1,008,999ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,999ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 845
2023-08-25 04:17:24 [1,008,999ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,999ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 865
2023-08-25 04:17:24 [1,008,999ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,999ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 893
2023-08-25 04:17:24 [1,008,999ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,999ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1782
2023-08-25 04:17:24 [1,008,999ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,999ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,008,999ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,008,999ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,008,999ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,000ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,009,000ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,000ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,009,000ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,000ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,009,000ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,000ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,009,000ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,000ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,009,000ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,000ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,009,000ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,000ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,009,000ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,001ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,009,001ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,001ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,009,001ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,001ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,009,001ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,001ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,009,001ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,001ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,009,001ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,001ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,009,001ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,001ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,009,001ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,002ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,009,002ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,002ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,009,002ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,002ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,009,002ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,002ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,009,002ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,002ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,009,002ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,002ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,009,002ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,002ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,009,002ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,003ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,009,003ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,003ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1695
2023-08-25 04:17:24 [1,009,003ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,003ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 845
2023-08-25 04:17:24 [1,009,003ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,003ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 865
2023-08-25 04:17:24 [1,009,003ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,003ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 893
2023-08-25 04:17:24 [1,009,003ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,003ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1782
2023-08-25 04:17:24 [1,009,003ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,003ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,009,003ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,004ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,009,004ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,004ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,009,004ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,004ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,009,004ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,004ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,009,004ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,004ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,009,004ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,004ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,009,004ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,004ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,009,004ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,005ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,009,005ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,005ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,009,005ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,005ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,009,005ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,005ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,009,005ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,005ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,009,005ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,005ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,009,005ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,005ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,009,005ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,006ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1050
2023-08-25 04:17:24 [1,009,006ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,006ms] [Error] [omni.physx.plugin] PhysX error: GPU averageDeltaV fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 1060
2023-08-25 04:17:24 [1,009,006ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,006ms] [Error] [omni.physx.plugin] PhysX error: GPU mArtiSolvePartitionFunction fail to launch kernel!!
, FILE /buildAgent/work/f25a4639a4b1bdc1/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1609
2023-08-25 04:17:24 [1,009,006ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-08-25 04:17:24 [1,009,352ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuRigidBodyView.cpp: 190
2023-08-25 04:17:24 [1,009,352ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuRigidBodyView.cpp: 191
2023-08-25 04:17:24 [1,009,352ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuRigidBodyView.cpp: 188
2023-08-25 04:17:24 [1,009,352ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuRigidBodyView.cpp: 189
2023-08-25 04:17:24 [1,009,352ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuRigidBodyView.cpp: 190
2023-08-25 04:17:24 [1,009,352ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuRigidBodyView.cpp: 191
2023-08-25 04:17:24 [1,009,370ms] [Error] [omni.physx.flatcache.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.flatcache/plugins/DirectGpuHelper.cpp: 487
2023-08-25 04:17:24 [1,009,370ms] [Error] [omni.physx.flatcache.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.flatcache/plugins/DirectGpuHelper.cpp: 488
2023-08-25 04:17:24 [1,009,370ms] [Error] [omni.physx.flatcache.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.flatcache/plugins/DirectGpuHelper.cpp: 492
2023-08-25 04:17:24 [1,009,370ms] [Error] [omni.physx.flatcache.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.flatcache/plugins/DirectGpuHelper.cpp: 493
2023-08-25 04:17:24 [1,009,370ms] [Error] [omni.physx.flatcache.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.flatcache/plugins/DirectGpuHelper.cpp: 497
2023-08-25 04:17:24 [1,009,370ms] [Error] [omni.physx.flatcache.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.flatcache/plugins/DirectGpuHelper.cpp: 498
2023-08-25 04:17:24 [1,009,539ms] [Warning] [omni.isaac.orbit_envs.isaac_env] Simulation is stopped. Please exit the simulator...
Traceback (most recent call last):
File "sb3/train.py", line 127, in <module>
main()
File "sb3/train.py", line 117, in main
agent.learn(total_timesteps=n_timesteps, callback=checkpoint_callback)
File "/home/isaac/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/python/lib/python3.7/site-packages/stable_baselines3/ppo/ppo.py", line 327, in learn
progress_bar=progress_bar,
File "/home/isaac/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/python/lib/python3.7/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 262, in learn
continue_training = self.collect_rollouts(self.env, callback, self.rollout_buffer, n_rollout_steps=self.n_steps)
File "/home/isaac/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/python/lib/python3.7/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 181, in collect_rollouts
new_obs, rewards, dones, infos = env.step(clipped_actions)
File "/home/isaac/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/python/lib/python3.7/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 162, in step
return self.step_wait()
File "/home/isaac/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/python/lib/python3.7/site-packages/stable_baselines3/common/vec_env/vec_normalize.py", line 149, in step_wait
obs, rewards, dones, infos = self.venv.step_wait()
File "/home/isaac/Research/Omniverse/Orbit/source/extensions/omni.isaac.orbit_envs/omni/isaac/orbit_envs/utils/wrappers/sb3.py", line 145, in step_wait
return self.step(self._async_actions)
File "/home/isaac/Research/Omniverse/Orbit/source/extensions/omni.isaac.orbit_envs/omni/isaac/orbit_envs/utils/wrappers/sb3.py", line 121, in step
reset_ids = (dones > 0).nonzero(as_tuple=False)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Exception ignored in: <function _make_registry.<locals>._Registry.__del__ at 0x7f476a5315f0>
Traceback (most recent call last):
File "/home/isaac/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.viewport.registry/omni/kit/viewport/registry/registry.py", line 103, in __del__
File "/home/isaac/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.viewport.registry/omni/kit/viewport/registry/registry.py", line 98, in destroy
TypeError: 'NoneType' object is not callable
Exception ignored in: <function _make_registry.<locals>._Registry.__del__ at 0x7f476a5315f0>
Traceback (most recent call last):
File "/home/isaac/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.viewport.registry/omni/kit/viewport/registry/registry.py", line 103, in __del__
File "/home/isaac/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.viewport.registry/omni/kit/viewport/registry/registry.py", line 98, in destroy
TypeError: 'NoneType' object is not callable
Exception ignored in: <function SettingChangeSubscription.__del__ at 0x7f492d1f75f0>
Traceback (most recent call last):
File "/home/isaac/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/kernel/py/omni/kit/app/_impl/__init__.py", line 114, in __del__
AttributeError: 'NoneType' object has no attribute 'get_settings'
Exception ignored in: <function RegisteredActions.__del__ at 0x7f3ac6451cb0>
Traceback (most recent call last):
File "/home/isaac/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.kit.viewport.menubar.lighting-104.0.8/omni/kit/viewport/menubar/lighting/actions.py", line 345, in __del__
File "/home/isaac/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.kit.viewport.menubar.lighting-104.0.8/omni/kit/viewport/menubar/lighting/actions.py", line 350, in destroy
TypeError: 'NoneType' object is not callable
/home/isaac/.local/share/ov/pkg/isaac_sim-2022.2.0/python.sh: line 41: 7128 Segmentation fault (core dumped) $python_exe "$@" $args
There was an error running python
I got this error while running RL agent in Isaac Sim, using stable baselines3