How to execute the kaolin app in the script way?

Hello!

I would like to convert multiple 3D object.usd file into the point_cloud.usd format.

However, when applying the whole dataset, it would be out of the memory. So I would like execute in the script way by loop.

My original thinking is that given the input of “root dir”, “output dir”, “enable point cloud”, and “click run” in the script by loop in each usd file.

After reading your technical document and API on Github, the former only taught you how to execute it on the GUI platform. The latter one could not convert it into point cloud.

By the third solution related to directly apply the code in kaolin folder at the system, there are some extension packge not included problem.

Do you have any tutorials or resources related to my development for the reference?

Thanks.

Hello @aaec3155! I notified the development team for help in answering your question!

Hello @aaec3155, thanks for reaching out!

Just to confirm I understand your question, you’re looking to convert multiple 3D assets into point clouds (in usd format), but have been running into memory limit issues. “when applying the whole dataset” → Do you mean you run out of memory when trying to convert to pointclouds all assets loaded in at once? Or are you still running out of memory using the GUI as you described above looping through each asset one by one?

If it’s the latter case, you can try reducing the Resolution parameter. This will reduce the density of the pointcloud but will also reduce the VRAM consumption. Please reach out if this doesn’t address your issue and I’m happy to provide more support.

Hello @jlafleche

Thanks for your reply

I follow your solutions

  • Apply the dataset with 23 usd files with default resolution value 224: Run out of memory, and kaolin is shutted down

  • Apply the single usd file one by one with low resolution from the default value 224 to 50: Same situation as the former

Unfortunately, it may not work. QAQ

Until now, the way I could use in manually is

  • Open kaolin → Asset Conversion into point cloud format (Default Resolution) → Close kaolin
  • Repeat this operation until I convert all usd assets in the dataset.

I also posted another article in this forum.

  • Hardware & Software Version

    • Computer OS: Ubuntu 20.01.4 LTS
    • GPU: 2 * RTX 3090 with 24576 MB
    • Hardware & Software Version
    • Computer OS: Ubuntu 20.01.4 LTS
    • GPU: 2 * RTX 3090 with 24576 MB
    • GPU Driver: 210.73.05
      • GPU Driver: 510.73.05
    • CUDA: 11.6
    • Kaolin: 2021.2.0
    • Kit: 102.1.1
  • My Operation:

It would work that using data generator let all dataset (22 usd files) convert into RGB. But when I would like to apply whole dataset (22 usd files) for the conversion of “Point Cloud”, it would happen memory leak. Even if I convert it one by one, the first operation does work. However, the second one failed. The Kaolin App would be automatically shutted down.

When executing the “Point Colud” conversion for the second usd file, both of my GPU is out of memory as shown in the figure.

  • Console Logs

2022-07-07 06:45:37 [136,816ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2022-07-07 06:45:37 [136,816ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2022-07-07 06:45:37 [136,816ms] [Error] [gpu.foundation.plugin] Texture creation failed for the device: 0.
2022-07-07 06:45:37 [136,816ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp 25
2022-07-07 06:45:37 [136,816ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp Synthetic Image Data RT. Will not execute this or subsequent RenderGraph operations. Aborting RenderGraph execution
2022-07-07 06:45:37 [136,816ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to execute RenderGraph on device 0. Error Code: 2
2022-07-07 06:45:37 [136,819ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2022-07-07 06:45:37 [136,819ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2022-07-07 06:45:37 [136,819ms] [Error] [gpu.foundation.plugin] Texture creation failed for the device: 0.
2022-07-07 06:45:37 [136,821ms] [Error] [omni.usd] SceneRenderer failed to render view. Aborting this frame for this view.
2022-07-07 06:45:37 [136,823ms] [Error] [gpu.foundation.plugin] No RpResource was allocated, getResourceBuffer failed
Fatal Python error: Segmentation fault

Current thread 0x00007fa7bb774740 (most recent call first):
2022-07-07 06:45:38 [137,446ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.events.plugin.so!carbOnPluginStartup
2022-07-07 06:45:38 [137,451ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::Scheduler::)(unsigned int, int, carb::cpp20::latch), carb::tasking::Scheduler*, unsigned int, int, carb::cpp20::latch*> > >::_M_run()
2022-07-07 06:45:38 [137,455ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpthread.so.0!funlockfile
2022-07-07 06:45:38 [137,459ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpthread.so.0!raise
2022-07-07 06:45:38 [137,463ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpthread.so.0!funlockfile
2022-07-07 06:45:38 [137,466ms] [Fatal] [carb.crashreporter-breakpad.plugin] libgpu.foundation.so!void std::this_thread::sleep_for<long, std::ratio<1l, 1000000l> >(std::chrono::duration<long, std::ratio<1l, 1000000l> > const&)
2022-07-07 06:45:38 [137,467ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.scenedb.plugin.so!void std::vector<unsigned long*, std::allocator<unsigned long*> >::_M_realloc_insert<unsigned long*>(__gnu_cxx::__normal_iterator<unsigned long**, std::vector<unsigned long*, std::allocator<unsigned long*> > >, unsigned long*&&)
2022-07-07 06:45:38 [137,470ms] [Fatal] [carb.crashreporter-breakpad.plugin] libgpu.foundation.so!std::vector<unsigned int, std::allocator >::_M_default_append(unsigned long)
2022-07-07 06:45:38 [137,471ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.scenerenderer-rtx.plugin.so!_init
2022-07-07 06:45:38 [137,472ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.scenerenderer-rtx.plugin.so!_init
2022-07-07 06:45:38 [137,473ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.scenerenderer-rtx.plugin.so!_init
2022-07-07 06:45:38 [137,476ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.scenerenderer-mux.plugin.so!carbOnPluginShutdown
2022-07-07 06:45:38 [137,479ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.hydra.so!std::__detail::_Map_base<unsigned short, std::pair<unsigned short const, std::string>, std::allocator<std::pair<unsigned short const, std::string> >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator(unsigned short const&)
2022-07-07 06:45:38 [137,482ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.hydra.so!std::__detail::_Map_base<unsigned short, std::pair<unsigned short const, std::string>, std::allocator<std::pair<unsigned short const, std::string> >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator(unsigned short const&)
2022-07-07 06:45:38 [137,485ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<omni::usd::UsdContext::Impl::closeAndOpenStage(std::string const&, bool, bool)::{lambda()#1}> >, void> >::_M_invoke(std::_Any_data const&)
2022-07-07 06:45:38 [137,488ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<omni::usd::UsdContext::Impl::closeAndOpenStage(std::string const&, bool, bool)::{lambda()#1}> >, void> >::_M_invoke(std::_Any_data const&)
2022-07-07 06:45:38 [137,491ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!omni::usd::UsdManager::drawViewports(float)
2022-07-07 06:45:38 [137,492ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext-viewport_window.plugin.so!_init
2022-07-07 06:45:38 [137,496ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.events.plugin.so!void std::this_thread::sleep_for<long, std::ratio<1l, 1000000l> >(std::chrono::duration<long, std::ratio<1l, 1000000l> > const&)
2022-07-07 06:45:38 [137,498ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!void std::vector<unsigned long, std::allocator >::_M_realloc_insert<unsigned long const&>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator > >, unsigned long const&)
2022-07-07 06:45:38 [137,501ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!carbOnPluginRegisterEx
2022-07-07 06:45:38 [137,504ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!std::__future_base::_State_baseV2::_M_is_deferred_future() const
2022-07-07 06:45:38 [137,508ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.events.plugin.so!void std::this_thread::sleep_for<long, std::ratio<1l, 1000000l> >(std::chrono::duration<long, std::ratio<1l, 1000000l> > const&)
2022-07-07 06:45:38 [137,511ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.loop-default.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tupleomni::kit::RunLoopThread::run():{lambda()#1} > >::~_State_impl()
2022-07-07 06:45:38 [137,514ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!_init
2022-07-07 06:45:38 [137,518ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!carbOnPluginPostShutdown
2022-07-07 06:45:38 [137,518ms] [Fatal] [carb.crashreporter-breakpad.plugin] kit!_init
2022-07-07 06:45:38 [137,522ms] [Fatal] [carb.crashreporter-breakpad.plugin] libc.so.6!__libc_start_main
2022-07-07 06:45:38 [137,522ms] [Fatal] [carb.crashreporter-breakpad.plugin] kit!_init
Segmentation fault (core dumped)

  • My Questions

    • Is it normal that converting into point cloud consumes both of my GPU memory approximately 48 GB?
    • Or is it the technical bug in Kaolin APP?