[Error] [omni.syntheticdata.plugin] CUDA error 801: cudaErrorNotSupported - operation not supported)

Omniverse Isaac Sim
1

Hi, I tried the sample code of Replicator from the tutorials and got this CUDA error. I checked my OS and hardware driver versions. They should meet the requirements indicated in the tutorials.

My environment is as below.
Software: Isacc SIm 2022.1.0
OS: Ubuntu 20.04
Displayer card:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.05    Driver Version: 510.73.05    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:17:00.0 Off |                  N/A |
| 24%   32C    P8    32W / 260W |     11MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:65:00.0  On |                  N/A |
| 34%   42C    P8    34W / 260W |    549MiB / 11264MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1420      G   /usr/lib/xorg/Xorg                  4MiB |
|    0   N/A  N/A      2169      G   /usr/lib/xorg/Xorg                  4MiB |
|    1   N/A  N/A      1420      G   /usr/lib/xorg/Xorg                 53MiB |
|    1   N/A  N/A      2169      G   /usr/lib/xorg/Xorg                278MiB |
|    1   N/A  N/A      2299      G   /usr/bin/gnome-shell               61MiB |
|    1   N/A  N/A      3981      G   ...785781292877507092,131072      126MiB |
|    1   N/A  N/A     14324      G   ...RendererForSitePerProcess       14MiB |
+-----------------------------------------------------------------------------+

Cuda

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0

Log file attached.
kit_20220616_162206.log (1.1 MB)

Sample code

import omni.replicator.core as rep

with rep.new_layer():
    camera = rep.create.camera(position=(0, 0, 1000))
    render_product = rep.create.render_product(camera, (1024, 1024))

    def move_shapes():
        #shapes = rep.get.prims(path_pattern='[^M_]Cube$|[^M_]Cone$|[^M_]Sphere$')
        shapes = rep.get.prims(semantics=[('class', 'cube'), ('class', 'cone')])

        with shapes:
            rep.modify.pose(
                position=rep.distribution.uniform((-500, 50, -500), (500, 50, 500)),
                rotation=rep.distribution.uniform((0,-180, 0), (0, 180, 0)),
            )
        return shapes.node

    rep.randomizer.register(move_shapes)


    # Setup randomization
    with rep.trigger.on_frame(num_frames=100):
        rep.randomizer.move_shapes()

    writer = rep.WriterRegistry.get("BasicWriter")
    writer.initialize(output_dir="_output", rgb=True,   bounding_box_2d_tight=True)
    writer.attach([render_product])

Should I change my Cuda version? Can you advise? Thanks.

3

I tried with a new OS and different versions of cuda. It still doesn’t work.

Whenever I want to access the image data, this error occurs.

Can anyone help with this? Thank you!

4

I can confirm I get the same error with 2022.1.0, when trying to use ROS1CameraHelper

1 Like
5

When I tried to access the image data, this error occurs.
I tried to change the params of “multi_gpu” and “active_gpu”, it still doesn’t work.

6

Updates.

Now I am able to run ISAAC with Replicator under headless mode and using one specific GPU.

The standalone app python codes and log files are provided below for the develop team testing.

The synthetic data can only generated with headless = true, active_gpu = 1 (log file 4).

offline_generation-test.py (3.9 KB)

1.headless=false&active_gpu=0.log|attachment (145.3 KB)
2.headless=false&active_gpu=1.log|attachment (4.4 KB)
3.headless=true&active_gpu=0.log|attachment (145.3 KB)
4.headless=true&active_gpu=1.log|attachment (42.6 KB)

It seems that GPU order from nvidia-smi cmd is different from the order required for ISAAC. For example, when I set active_gpu = 1, nvidia-smi cmd reports ISAAC runs on GPU-0.

7

Thanks for reporting @zhengzj!
The replicator team is still taking a look and trying to isolate the issue.

10

Hi @zhengzj.

There’s a chance this is addressed in the most recent release, but its most likely the fix won’t make it in, until a future release. I’ll follow up when I have more concrete info.

11

At first glance, it seems that in 2022.1.1 the error doesn’t happen with ROS1 Camera Helper anymore.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.