No module named nvml using tlt-dataset-convert

Ubuntu 18.04 latest, RTX2080ti, tlt-streamanalytics:v2.0_dp_py2 container

I receive the error below (no module named nvml) when I run the tlt-dataset-convert command in the detectnet_v2 example. Deepstream 5 works fine on this box outside the container.

print(“Converting Tfrecords for kitti trainval dataset”)
!tlt-dataset-convert -d $SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt
-o $DATA_DOWNLOAD_DIR/tfrecords/kitti_trainval/kitti_trainval

************Output **************

Creating a new directory for the output tfrecords dump.

print(“Converting Tfrecords for kitti trainval dataset”)

!tlt-dataset-convert -d $SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt \

                 -o $DATA_DOWNLOAD_DIR/tfrecords/kitti_trainval/kitti_trainval

Converting Tfrecords for kitti trainval dataset
Using TensorFlow backend.
Traceback (most recent call last):
File “/usr/local/bin/tlt-dataset-convert”, line 5, in
from iva.detectnet_v2.scripts.dataset_convert import main
File “./detectnet_v2/scripts/dataset_convert.py”, line 14, in
File “./detectnet_v2/dataio/build_converter.py”, line 13, in
File “./detectnet_v2/dataio/kitti_converter_lib.py”, line 22, in
File “./detectnet_v2/dataio/dataset_converter_lib.py”, line 20, in
File “./detectnet_v2/dataloader/utilities.py”, line 15, in
File “/usr/local/lib/python2.7/dist-packages/modulus/init.py”, line 8, in
from modulus import blocks
File “/usr/local/lib/python2.7/dist-packages/modulus/blocks/init.py”, line 22, in
from modulus.blocks import data_loaders
File “/usr/local/lib/python2.7/dist-packages/modulus/blocks/data_loaders/init.py”, line 9, in
from modulus.blocks.data_loaders.sqlite_dataloader import SQLiteDataLoader
File “./modulus/blocks/data_loaders/sqlite_dataloader.py”, line 10, in
File “/usr/local/lib/python2.7/dist-packages/modulus/dataloader/init.py”, line 8, in
from modulus.dataloader import humanloop
File “./modulus/dataloader/humanloop.py”, line 16, in
File “/usr/local/lib/python2.7/dist-packages/modulus/processors/init.py”, line 26, in
from modulus.processors.buffers import NamedTupleStagingArea
File “./modulus/processors/buffers.py”, line 10, in
File “/usr/local/lib/python2.7/dist-packages/modulus/hooks/init.py”, line 9, in
from modulus.hooks.hooks import KerasCheckpointListener
File “./modulus/hooks/hooks.py”, line 26, in
ImportError: No module named nvml

Thanks,
Patrick

Please paste your
$SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt

Thanks.

Please run below and paste the result too.
$ nvidia-smi

Archive: /opt/ngccli/ngccli_reg_linux.zip
inflating: /opt/ngccli/ngc
extracting: /opt/ngccli/ngc.md5
root@ece58f118dd7:/workspace# nvidia-smi
Tue Jun 16 14:49:35 2020
±----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 208… On | 00000000:01:00.0 On | N/A |
| 35% 26C P8 23W / 260W | 323MiB / 11016MiB | 2% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
±----------------------------------------------------------------------------+
root@ece58f118dd7:/workspace#

kitti_config {
root_directory_path: “/workspace/tlt-experiments/data/training”
image_dir_name: “image_2”
label_dir_name: “label_2”
image_extension: “.png”
partition_mode: “random”
num_partitions: 2
val_split: 14
num_shards: 10
}
image_directory_path: “/workspace/tlt-experiments/data/training”

Looks nvidia-smi runs smoothly.
This is the first time to see “ImportError: No module named nvml” in TLT. Other users do not meet this issue.

How did you trigger the docker? Is it below?
docker run --runtime=nvidia -it nvcr.io/nvidia/tlt-streamanalytics:v2.0_dp_py2 /bin/bash

Also please double check below.

Software Requirements

Ubuntu 18.04 LTS
NVIDIA GPU Cloud account and API key - https://ngc.nvidia.com/
docker-ce installed, https://docs.docker.com/install/linux/docker-ce/ubuntu/
nvidia-docker2 installed, instructions: https://github.com/nvidia/nvidia-docker/wiki/Installation-(version-2.0)
NVIDIA GPU driver v410.xx or above

Note: DeepStream 5.0 - NVIDIA SDK for IVA inference https://developer.nvidia.com/deepstream-sdk is recommended.

Installation Prerequisites

Install Docker. See: https://www.docker.com/.
NVIDIA GPU driver v410.xx or above. Download from https://www.nvidia.com/Download/index.aspx?lang=en-us.
Install NVIDIA Docker 2 from: https://github.com/NVIDIA/nvidia-docker.