Isaac Sim Log Level / To many log messages

marvin.wiedemann · October 2, 2024, 1:42pm

Hi all,
In our Python application, we utilize containerized Isaac Sim instances to generate synthetic image data. However, we are encountering issues with logging, as Isaac Sim produces an overwhelming number of messages. Here’s the situation:

A bash script starts an Isaac Sim container using Docker, and a python script is executed using python.sh (tested with Isaac Sim 4.0.0 up to 4.2.0).
Based on forum discussions, we tried to set the log level via flags within the python.sh command (How to stop printing warning level messages to console for Isaac sim?). Unfortunately, this did not work.
We also tried using carb settings (Deactivate warnings - #7 by hijimasa), but that did not work for us as well.
The only successful method we found was modifying the log level in the apps/omni.isaac.sim.python.kit file, as suggested in How to stop printing warning level messages to console for Isaac sim? - #3 by Hammad_M. This stops much of the log output. While this reduces much of the log output, the initial log messages during app startup remain.

Given this context, I have two questions:

Are there alternative methods to change the log level without modifying the apps/omni.isaac.sim.python.kit file? This method is not ideal for a containerized solution.
Is there a way to completely eliminate the Isaac startup logs?

Thanks in advance!
Marvin

marvin.wiedemann · October 2, 2024, 1:53pm

The messages we are able to disable via the omni.isaac.sim.python.kit file are all warnings. This is great but I would love to do this via the flags or carb settings.

The logs which are still remaining are the standard Isaac Sim startup logs. As we start multiple Isaac Sim instances in our approach, this is flooding our terminal.

[2.563s] [ext: omni.convexdecomposition-106.1.9] startup
[2.568s] [ext: omni.graph.nodes-1.146.1] startup
[2.579s] [ext: omni.physx.cooking-106.1.9] startup
[2.592s] [ext: omni.kit.primitive.mesh-1.0.17] startup
[2.597s] [ext: omni.physx-106.1.9] startup
[2.611s] [ext: omni.kit.stage_templates-1.2.5] startup
[2.617s] [ext: omni.isaac.version-1.1.0] startup
[2.620s] [ext: omni.kit.widget.material_preview-1.0.16] startup
[2.624s] [ext: omni.isaac.nucleus-0.3.1] startup
[2.627s] [ext: omni.physics.tensors-106.1.9] startup
[2.637s] [ext: omni.physx.stageupdate-106.1.9] startup
[2.641s] [ext: omni.graph.action_core-1.1.6] startup
[2.646s] [ext: omni.warp.core-1.2.1] startup
[2.747s] [ext: omni.physx.tensors-106.1.9] startup
[2.753s] [ext: omni.graph.scriptnode-1.20.1] startup
[2.754s] [ext: omni.kit.manipulator.viewport-107.0.0] startup
[2.761s] [ext: omni.graph.action_nodes-1.24.0] startup
[2.771s] [ext: omni.kit.ui_test-1.3.2] startup
[2.780s] [ext: omni.isaac.core-3.19.5] startup
[3.200s] [ext: omni.graph.visualization.nodes-2.1.1] startup
[3.213s] [ext: omni.graph.action-1.102.1] startup
[3.217s] [ext: omni.kit.window.cursor-1.1.2] startup
[3.224s] [ext: omni.isaac.ui-0.16.0] startup
[3.256s] [ext: omni.kit.viewport.menubar.core-106.1.0] startup
[3.296s] [ext: omni.kit.graph.delegate.default-1.2.2] startup
[3.300s] [ext: omni.syntheticdata-0.6.9] startup
[3.331s] [ext: omni.kit.graph.usd.commands-1.3.1] startup
[3.337s] [ext: omni.kit.graph.editor.core-1.5.3] startup
[3.343s] [ext: omni.kit.numpy.common-0.1.2] startup
[3.346s] [ext: omni.warp-1.2.1] startup
[3.359s] [ext: omni.kit.window.material_graph-1.8.18] startup
[3.469s] [ext: omni.isaac.dynamic_control-1.3.8] startup
[3.501s] [ext: omni.replicator.core-1.11.20] startup
Warp 1.2.1 initialized:
CUDA Toolkit 11.8, Driver 12.2
Devices:
“cpu” : “x86_64”
“cuda:0” : “NVIDIA A40-48Q” (48 GiB, sm_86, mempool not supported)
Kernel cache:
/root/.cache/warp/1.2.1
Warp UserWarning: Support for CUDA memory pools was not detected on devices [‘cuda:0’]. This prevents memory allocations in CUDA graphs and may result in poor performance. Is the UVM driver enabled?
[3.729s] [ext: omni.isaac.lula-3.0.1] startup
[3.744s] [ext: omni.isaac.surface_gripper-1.0.1] startup
[3.753s] [ext: omni.isaac.core_nodes-1.16.3] startup
[3.786s] [ext: omni.isaac.motion_generation-7.1.0] startup
[3.814s] [ext: omni.isaac.manipulators-2.1.0] startup
[3.819s] [ext: omni.isaac.franka-0.4.1] startup
[3.824s] [ext: omni.kit.graph.widget.variables-2.1.0] startup
[3.842s] [ext: omni.kit.graph.delegate.modern-1.10.6] startup
[3.850s] [ext: omni.isaac.cortex-0.3.9] startup
[3.852s] [ext: omni.graph.window.core-1.113.1] startup
[3.892s] [ext: omni.isaac.wheeled_robots-2.3.3] startup
[3.918s] [ext: omni.isaac.cortex.sample_behaviors-1.0.6] startup
[3.921s] [ext: omni.graph.window.generic-1.26.0] startup
[3.955s] [ext: omni.isaac.menu-0.7.3] startup
[3.970s] [ext: omni.kit.widget.live-2.1.8] startup
[3.985s] [ext: omni.kit.actions.window-1.1.1] startup
[4.025s] [ext: omni.kit.widget.cache_indicator-2.0.10] startup
[4.216s] [ext: omni.kit.hotkeys.window-1.4.5] startup
[4.244s] [ext: omni.kit.ui.actions-1.0.2] startup
[4.247s] [ext: omni.kit.widget.stage_icons-1.0.5] startup
[4.251s] [ext: omni.kit.menu.common-1.1.7] startup
[4.257s] [ext: omni.kit.selection-0.1.4] startup
[4.261s] [ext: omni.isaac.kit-1.13.2] startup
[4.262s] [ext: omni.isaac.block_world-1.0.0] startup
[4.275s] [ext: omni.kit.window.stage-2.5.10] startup
[4.299s] [ext: omni.isaac.debug_draw-1.1.0] startup
[4.309s] [ext: omni.kit.menu.edit-1.1.24] startup
[4.323s] [ext: omni.kit.menu.file-1.1.14] startup
[4.327s] [ext: omni.kit.profiler.window-2.2.3] startup
[4.362s] [ext: omni.kit.menu.stage-1.2.5] startup
[4.366s] [ext: omni.isaac.occupancy_map-1.0.3] startup
[4.389s] [ext: omni.importer.mjcf-1.1.1] startup
[4.404s] [ext: omni.graph.window.action-1.28.0] startup
[4.417s] [ext: omni.kit.property.camera-1.0.8] startup
[4.426s] [ext: omni.kit.property.light-1.0.10] startup
[4.430s] [ext: omni.kit.property.geometry-1.3.1] startup
[4.437s] [ext: omni.hydra.scene_api-0.1.2] startup
[4.444s] [ext: omni.kit.property.audio-1.0.14] startup
[4.457s] [ext: omni.kit.property.render-1.1.2] startup
[4.459s] [ext: omni.kit.property.transform-1.5.9] startup
[4.464s] [ext: omni.kit.property.material-1.10.8] startup
[4.472s] [ext: omni.kit.widget.layers-1.8.0] startup
[4.490s] [ext: omni.kit.property.isaac-0.2.3] startup
[4.497s] [ext: omni.kit.property.bundle-1.3.1] startup
[4.498s] [ext: omni.sensors.nv.common-1.2.2-isaac-1] startup
[4.513s] [ext: omni.kit.property.layer-1.1.8] startup
[4.517s] [ext: omni.kit.stage_column.payload-2.0.0] startup
[4.519s] [ext: omni.isaac.scene_blox-0.1.2] startup
[4.523s] [ext: omni.sensors.nv.materials-1.2.1-isaac-1] startup
[4.526s] [ext: omni.sensors.nv.wpm-1.2.1-isaac-1] startup
[4.528s] [ext: omni.kit.viewport.actions-106.0.2] startup
[4.534s] [ext: omni.sensors.nv.lidar-1.2.2-isaac-1] startup
[4.541s] [ext: omni.sensors.nv.radar-1.2.1-isaac-1] startup
[4.549s] [ext: omni.kit.viewport.menubar.display-107.0.2] startup
[4.553s] [ext: omni.kit.manipulator.transform-104.7.4] startup
[4.557s] [ext: omni.kit.widget.toolbar-1.7.2] startup
[4.569s] [ext: omni.sensors.tiled-0.0.6] startup
[4.572s] [ext: omni.sensors.nv.ids-1.1.0-isaac-1] startup
[4.578s] [ext: omni.isaac.range_sensor-3.1.2] startup
[4.596s] [ext: omni.usdphysics.ui-106.1.9] startup
[4.629s] [ext: omni.physx.commands-106.1.9] startup
[4.634s] [ext: omni.kit.manipulator.tool.snap-1.5.11] startup
[4.644s] [ext: omni.isaac.sensor-12.9.1] startup
[4.739s] [ext: omni.kit.manipulator.selector-1.1.1] startup
[4.745s] [ext: omni.physx.ui-106.1.9] startup
[4.811s] [ext: omni.isaac.quadruped-2.0.1] startup
[4.824s] [ext: omni.kit.viewport.manipulator.transform-107.0.2] startup
[4.835s] [ext: omni.physx.demos-106.1.9] startup
[4.867s] [ext: omni.kit.property.physx-106.1.9] startup
[4.974s] [ext: omni.kit.widget.calendar-1.0.8] startup
[4.978s] [ext: omni.kit.manipulator.prim.core-107.0.4] startup
[4.989s] [ext: omni.fabric.commands-1.1.5] startup
[4.993s] [ext: omni.physx.vehicle-106.1.9] startup
[5.018s] [ext: omni.kit.widget.extended_searchfield-1.0.28] startup
[5.035s] [ext: omni.kit.manipulator.prim.fabric-107.0.3] startup
[5.040s] [ext: omni.kit.manipulator.prim.usd-107.0.2] startup
[5.041s] [ext: omni.kit.widget.timeline-105.0.1] startup
[5.048s] [ext: omni.kit.window.commands-0.2.6] startup
[5.050s] [ext: omni.physx.camera-106.1.9] startup
[5.065s] [ext: omni.kit.manipulator.camera-105.0.5] startup
[5.075s] [ext: omni.kit.viewport.menubar.camera-107.0.2] startup
[5.082s] [ext: omni.kit.manipulator.prim-107.0.0] startup
[5.083s] [ext: omni.kit.manipulator.selection-106.0.1] startup
[5.085s] [ext: omni.kit.window.toolbar-1.6.1] startup
[5.089s] [ext: omni.kit.stage_column.variant-1.0.13] startup
[5.092s] [ext: omni.kit.viewport.menubar.render-106.1.1] startup
[5.097s] [ext: omni.kit.viewport.menubar.settings-107.0.3] startup
[5.109s] [ext: omni.physx.cct-106.1.9] startup
[5.126s] [ext: omni.physx.graph-106.1.9] startup
[5.161s] [ext: omni.physx.supportui-106.1.9] startup
[5.190s] [ext: omni.physx.telemetry-106.1.9] startup
[5.198s] [ext: omni.graph.ui_nodes-1.26.0] startup
[5.206s] [ext: omni.kit.viewport.bundle-104.0.1] startup
[5.207s] [ext: omni.isaac.universal_robots-0.3.5] startup
[5.213s] [ext: omni.kit.window.console-0.2.13] startup
[5.229s] [ext: omni.kit.window.script_editor-1.7.6] startup
[5.237s] [ext: omni.kit.menu.create-1.0.16] startup
[5.246s] [ext: omni.graph.bundle.action-2.4.1] startup
[5.249s] [ext: omni.physx.bundle-106.1.9] startup
[5.251s] [ext: omni.rtx.window.settings-0.6.17] startup
[5.259s] [ext: omni.kit.window.status_bar-0.1.7] startup
[5.267s] [ext: omni.kit.window.title-1.1.5] startup
[5.269s] [ext: omni.replicator.isaac-1.15.2] startup
[5.316s] [ext: omni.replicator.replicator_yaml-2.0.6] startup
[5.330s] [ext: omni.rtx.settings.core-0.6.3] startup
[5.354s] [ext: omni.kit.viewport.menubar.lighting-106.0.2] startup
[5.370s] [ext: semantics.schema.editor-0.3.8] startup
[5.382s] [ext: semantics.schema.property-1.0.4] startup
[5.392s] [ext: omni.kit.viewport.rtx-104.0.1] startup
[5.393s] [ext: omni.isaac.utils-1.0.1] startup
[5.400s] [ext: omni.isaac.cloner-0.8.1] startup
[5.402s] [ext: omni.importer.urdf-1.14.1] startup
[5.462s] [ext: omni.kit.window.stats-0.1.6] startup
[5.470s] [ext: omni.isaac.sim.python-4.2.0] startup
[5.473s] Simulation App Starting
[7.501s] app ready
[8.360s] Simulation App Startup Complete

ahaidu · October 2, 2024, 2:39pm

Hi Marvin,

here are some possible relevant carb settings, can you check if any of them solve what you are trying to achieve?

# App
import carb.settings
carb.settings.get_settings().set("/app/enableDeveloperWarnings", False)
carb.settings.get_settings().set("/app/scripting/ignoreWarningDialog", True)

# UI Console
import carb.settings
carb.settings.get_settings().set("/exts/omni.kit.window.console/logFilter/verbose", False)
carb.settings.get_settings().set("/exts/omni.kit.window.console/logFilter/info", False)
carb.settings.get_settings().set("/exts/omni.kit.window.console/logFilter/warning", False)
carb.settings.get_settings().set("/exts/omni.kit.window.console/logFilter/error", False)
carb.settings.get_settings().set("/exts/omni.kit.window.console/logFilter/fatal", False)

# Log 
import carb.settings
carb.settings.get_settings().set("/log/debugConsoleLevel", "Fatal")  # verbose"|"info"|"warning"|"error"|"fatal"
carb.settings.get_settings().set("/log/enabled", False)
carb.settings.get_settings().set("/log/outputStreamLevel", "Error")
carb.settings.get_settings().set("/log/fileLogLevel", "Error")

marvin.wiedemann · October 2, 2024, 2:45pm

Thanks Adrei!

I added all of them after creating my Isaac Sim instance:

# start isaac
kit = SimulationApp(launch_config=isaac_config_data["ISAAC_CONFIG"])

# App
import carb.settings
carb.settings.get_settings().set("/app/enableDeveloperWarnings", False)
carb.settings.get_settings().set("/app/scripting/ignoreWarningDialog", True)

# UI Console
import carb.settings
carb.settings.get_settings().set("/exts/omni.kit.window.console/logFilter/verbose", False)
carb.settings.get_settings().set("/exts/omni.kit.window.console/logFilter/info", False)
carb.settings.get_settings().set("/exts/omni.kit.window.console/logFilter/warning", False)
carb.settings.get_settings().set("/exts/omni.kit.window.console/logFilter/error", False)
carb.settings.get_settings().set("/exts/omni.kit.window.console/logFilter/fatal", False)

# Log 
import carb.settings
carb.settings.get_settings().set("/log/debugConsoleLevel", "Fatal")  # verbose"|"info"|"warning"|"error"|"fatal"
carb.settings.get_settings().set("/log/enabled", False)
carb.settings.get_settings().set("/log/outputStreamLevel", "Error")
carb.settings.get_settings().set("/log/fileLogLevel", "Error")

But once I remove my modifications in the python.kit file I again get all warnings, such as

2024-10-02 14:42:29 [28,635ms] [Warning] [omni.replicator.core.ogn.python._impl.nodes.OgnGetPrims] No prims found with given inputs:

or

2024-10-02 14:42:16 [15,794ms] [Warning] [omni.usd] Warning: in AppendProperty at line 858 of /builds/omniverse/usd-ci/USD/pxr/usd/sdf/path.cpp – Can only append a property ‘png’ to a prim path ()