Point Cloud conversion consumes all GPU memory

  • Hardware & Software Version

    • Computer OS: Ubuntu 20.01.4 LTS
    • GPU: 2 * RTX 3090 with 24576 MB
    • GPU Driver: 510.73.05
    • CUDA: 11.6
    • Kaolin: 2021.2.0
    • Kit: 102.1.1
  • My Operation:

It would work that using data generator let all dataset (22 usd files) convert into RGB. But when I would like to apply whole dataset (22 usd files) for the conversion of “Point Cloud”, it would happen memory leak. Even if I convert it one by one, the first operation does work. However, the second one failed. The Kaolin App would be automatically shutted down.

When executing the “Point Colud” conversion for the second usd file, both of my GPU is out of memory as shown in the figure.

  • Console Logs

2022-07-07 06:45:37 [136,816ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2022-07-07 06:45:37 [136,816ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2022-07-07 06:45:37 [136,816ms] [Error] [gpu.foundation.plugin] Texture creation failed for the device: 0.
2022-07-07 06:45:37 [136,816ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp 25
2022-07-07 06:45:37 [136,816ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp Synthetic Image Data RT. Will not execute this or subsequent RenderGraph operations. Aborting RenderGraph execution
2022-07-07 06:45:37 [136,816ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to execute RenderGraph on device 0. Error Code: 2
2022-07-07 06:45:37 [136,819ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2022-07-07 06:45:37 [136,819ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2022-07-07 06:45:37 [136,819ms] [Error] [gpu.foundation.plugin] Texture creation failed for the device: 0.
2022-07-07 06:45:37 [136,821ms] [Error] [omni.usd] SceneRenderer failed to render view. Aborting this frame for this view.
2022-07-07 06:45:37 [136,823ms] [Error] [gpu.foundation.plugin] No RpResource was allocated, getResourceBuffer failed
Fatal Python error: Segmentation fault

Current thread 0x00007fa7bb774740 (most recent call first):
2022-07-07 06:45:38 [137,446ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.events.plugin.so!carbOnPluginStartup
2022-07-07 06:45:38 [137,451ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::Scheduler::)(unsigned int, int, carb::cpp20::latch), carb::tasking::Scheduler*, unsigned int, int, carb::cpp20::latch*> > >::_M_run()
2022-07-07 06:45:38 [137,455ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpthread.so.0!funlockfile
2022-07-07 06:45:38 [137,459ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpthread.so.0!raise
2022-07-07 06:45:38 [137,463ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpthread.so.0!funlockfile
2022-07-07 06:45:38 [137,466ms] [Fatal] [carb.crashreporter-breakpad.plugin] libgpu.foundation.so!void std::this_thread::sleep_for<long, std::ratio<1l, 1000000l> >(std::chrono::duration<long, std::ratio<1l, 1000000l> > const&)
2022-07-07 06:45:38 [137,467ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.scenedb.plugin.so!void std::vector<unsigned long*, std::allocator<unsigned long*> >::_M_realloc_insert<unsigned long*>(__gnu_cxx::__normal_iterator<unsigned long**, std::vector<unsigned long*, std::allocator<unsigned long*> > >, unsigned long*&&)
2022-07-07 06:45:38 [137,470ms] [Fatal] [carb.crashreporter-breakpad.plugin] libgpu.foundation.so!std::vector<unsigned int, std::allocator >::_M_default_append(unsigned long)
2022-07-07 06:45:38 [137,471ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.scenerenderer-rtx.plugin.so!_init
2022-07-07 06:45:38 [137,472ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.scenerenderer-rtx.plugin.so!_init
2022-07-07 06:45:38 [137,473ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.scenerenderer-rtx.plugin.so!_init
2022-07-07 06:45:38 [137,476ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.scenerenderer-mux.plugin.so!carbOnPluginShutdown
2022-07-07 06:45:38 [137,479ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.hydra.so!std::__detail::_Map_base<unsigned short, std::pair<unsigned short const, std::string>, std::allocator<std::pair<unsigned short const, std::string> >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator(unsigned short const&)
2022-07-07 06:45:38 [137,482ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.hydra.so!std::__detail::_Map_base<unsigned short, std::pair<unsigned short const, std::string>, std::allocator<std::pair<unsigned short const, std::string> >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator(unsigned short const&)
2022-07-07 06:45:38 [137,485ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<omni::usd::UsdContext::Impl::closeAndOpenStage(std::string const&, bool, bool)::{lambda()#1}> >, void> >::_M_invoke(std::_Any_data const&)
2022-07-07 06:45:38 [137,488ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<omni::usd::UsdContext::Impl::closeAndOpenStage(std::string const&, bool, bool)::{lambda()#1}> >, void> >::_M_invoke(std::_Any_data const&)
2022-07-07 06:45:38 [137,491ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!omni::usd::UsdManager::drawViewports(float)
2022-07-07 06:45:38 [137,492ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext-viewport_window.plugin.so!_init
2022-07-07 06:45:38 [137,496ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.events.plugin.so!void std::this_thread::sleep_for<long, std::ratio<1l, 1000000l> >(std::chrono::duration<long, std::ratio<1l, 1000000l> > const&)
2022-07-07 06:45:38 [137,498ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!void std::vector<unsigned long, std::allocator >::_M_realloc_insert<unsigned long const&>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator > >, unsigned long const&)
2022-07-07 06:45:38 [137,501ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!carbOnPluginRegisterEx
2022-07-07 06:45:38 [137,504ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!std::__future_base::_State_baseV2::_M_is_deferred_future() const
2022-07-07 06:45:38 [137,508ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.events.plugin.so!void std::this_thread::sleep_for<long, std::ratio<1l, 1000000l> >(std::chrono::duration<long, std::ratio<1l, 1000000l> > const&)
2022-07-07 06:45:38 [137,511ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.loop-default.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tupleomni::kit::RunLoopThread::run():{lambda()#1} > >::~_State_impl()
2022-07-07 06:45:38 [137,514ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!_init
2022-07-07 06:45:38 [137,518ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!carbOnPluginPostShutdown
2022-07-07 06:45:38 [137,518ms] [Fatal] [carb.crashreporter-breakpad.plugin] kit!_init
2022-07-07 06:45:38 [137,522ms] [Fatal] [carb.crashreporter-breakpad.plugin] libc.so.6!__libc_start_main
2022-07-07 06:45:38 [137,522ms] [Fatal] [carb.crashreporter-breakpad.plugin] kit!_init
Segmentation fault (core dumped)

  • My Questions

    • Is it normal that converting into point cloud consumes both of my GPU memory approximately 48 GB?
    • Or is it the technical bug in Kaolin APP?

Thanks.

Hello @aaec3155! Thanks for reporting this! I’ve reached out to the Kaolin team for further assistance!

Hello @aaec3155, thank you for reporting this. It appears that generating point clouds does result in a memory leak on multi-GPU systems. We’re currently working on developing a new point cloud conversion extension and will ensure that this leak is addressed. In the mean time, please try the following:

  1. Open the terminal using the launcher panel (Open in Terminal)
    image
  2. Launch kaolin using ./kaolin.sh --/renderer/multiGpu/enabled=false.

If the problem persists, please let us know and we’ll continue to help find a solution. Thanks again for bringing this to our attention!

Hello @jlafleche ,

It does work.

I could input the dataset folder to convert into point cloud one at one time.

When operating, it could be converted successfully.

Thanks a lot.