Jetson-containers does not build correctly

kenluebbert · January 8, 2025, 3:11pm

When following this tutorial: NanoDB - NVIDIA Jetson AI Lab

Step 4:
git clone GitHub - dusty-nv/jetson-containers: Machine Learning Containers for NVIDIA Jetson and JetPack-L4T
bash jetson-containers/install.sh

Fails with this error log:
nanodb_r36.4.2-python.txt (7.8 KB)

resulting in an incomplete container:

sudo docker image ls
REPOSITORY       TAG                       IMAGE ID       CREATED         SIZE
<none>           <none>                    fb4900030dde   16 hours ago    6.91GB
nanodb           r36.4.2-cudnn_9.4         4d73a710a23c   16 hours ago    6.91GB
nanodb           r36.4.2-cuda_12.6         57f6b38b7dd0   16 hours ago    5.91GB
nanodb           r36.4.2-build-essential   be8f81cd8a0f   16 hours ago    723MB
nanodb           r36.4.2-pip_cache_cu126   efca951b50f6   16 hours ago    723MB
ubuntu           22.04                     981912c48e9a   3 months ago    69.2MB
dustynv/nanodb   r36.2.0                   8118ef4e5cab   10 months ago   16GB

From the error log:

e[0mLooking in indexes: http://jetson.webredirect.org/jp5/cu126
e[91mERROR: Could not find a version that satisfies the requirement pip (from versions: none)
e[0me[91mERROR: No matching distribution found for pip

This URL does not exist:
http://jetson.webredirect.org/jp5/cu126
and might need to be updated to:
http://jetson.webredirect.org/jp6/cu126

AastaLLL · January 9, 2025, 5:14am

Hi,

Could you share how you setup the environment?

When you build the container, what JetPack, L4T, and CUDA version is recognized?
Below is the output from our side for your reference:

$ jetson-containers run $(autotag nanodb)
Namespace(packages=['nanodb'], prefer=['local', 'registry', 'build'], disable=[''], user='dustynv', output='/tmp/autotag', quiet=False, verbose=False)
-- L4T_VERSION=36.4.2  JETPACK_VERSION=6.1.1  CUDA_VERSION=12.6
-- Finding compatible container image for ['nanodb']

Couldn't find a compatible container for nanodb, would you like to build it? [y/N] y
-- Building containers  ['build-essential', 'pip_cache:cu126', 'cuda:12.6', 'cudnn:9.4', 'python', 'numpy', 'cmake', 'onnx', 'pytorch:2.5', 'cuda-python', 'faiss', 'faiss_lite', 'torchvision', 'huggingface_hub', 'rust', 'transformers', 'tensorrt', 'torch2trt', 'clip_trt', 'nanodb']
-- Building container nanodb:r36.4.2-build-essential

Thanks.

kenluebbert · January 9, 2025, 2:21pm

Hi,

Host Environment setup:
a) intel i3 x86_64 laptop > install Ubuntu 22.04.5 LTS > fully functional
b) install SDK Manager 2.2.0.12028

Jetson Orin Nano Super setup:
a) unbox > insert microSD loaded with JetPack 6.1> power-up > fully functional
b) install 500 GB NVMe > install jumper pins 9-10 > connect via USB to Host computer
c) sdkmanager > flash JetPack 6.1 to NVMe success > power-down
d) remove jumper > power-up > boot from NVMe > fully functional including MAXN
e) ollama works
f) start install of NanoDB jetson-containers > build not successful
g) jetson-containers run $(autotag nanodb)…

-- L4T_VERSION=36.4.2  JETPACK_VERSION=5.1  CUDA_VERSION=12.6
-- Finding compatible container image for ['nanodb']
nanodb:r36.4.2-cudnn_9.4
bash: $: command not found
bash: syntax error near unexpected token `packages=['nanodb'],'

h> exit container > sudo apt-cache show nvidia-jetpack

Package: nvidia-jetpack
Source: nvidia-jetpack (6.1)
Version: 6.1+b123
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-jetpack-runtime (= 6.1+b123), nvidia-jetpack-dev (= 6.1+b123)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_6.1+b123_arm64.deb
Size: 29312
SHA256: b6475a6108aeabc5b16af7c102162b7c46c36361239fef6293535d05ee2c2929
SHA1: f0984a6272c8f3a70ae14cb2ca6716b8c1a09543
MD5sum: a167745e1d88a8d7597454c8003fa9a4
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

I suspect jetson-containers/jetson_containers/l4t_version.py
get_jetpack_version routine needs to be updated with one more row for 36.4.2:

NVIDIA_JETPACK = {
        # -------- JP6 --------
        "36.4.2": "6.1.1",

kenluebbert · January 10, 2025, 9:14pm

I updated jetson-containers/jetson_containers/l4t_version.py with this entry in the NVIDIA_JETPACK lookup: “36.4.2”: “6.1.1”,

and the container build process does not end with an error, but the container does not contain nanodb in PYTHONPATH or nanodb.py in the /opt folder.

This results in this error when trying to run nanodb in the container:

/usr/local/bin/python3: No module named nanodb

sudo docker image ls

REPOSITORY       TAG                       IMAGE ID       CREATED         SIZE
nanodb           r36.4.2-faiss_lite        8202946adfc6   6 hours ago     9.95GB
nanodb           r36.4.2-faiss             0d3ba7e39cbe   6 hours ago     9.93GB
nanodb           r36.4.2-cuda-python       7f7fd3f11bad   6 hours ago     8.2GB
nanodb           r36.4.2-pytorch_2.5       7f137914f4bd   6 hours ago     8.06GB
nanodb           r36.4.2-onnx              0ba3f2a4559a   6 hours ago     7.14GB
nanodb           r36.4.2-cmake             46b48ff725ac   6 hours ago     7.11GB
nanodb           r36.4.2-numpy             470679f443f6   6 hours ago     7.05GB
nanodb           r36.4.2-python            4e9b47c13510   6 hours ago     6.99GB
nanodb           r36.4.2-cudnn_9.4         c416a92ae506   6 hours ago     6.92GB
nanodb           r36.4.2-cuda_12.6         96aad89635d7   6 hours ago     5.92GB
nanodb           r36.4.2-pip_cache_cu126   db5d38d31e3e   7 hours ago     734MB
nanodb           r36.4.2-build-essential   4ee0e57bf1c1   7 hours ago     734MB
ubuntu           22.04                     981912c48e9a   4 months ago    69.2MB
dustynv/nanodb   r36.2.0                   8118ef4e5cab   10 months ago   16GB

Any suggestions would be helpful.

Thanks

kenluebbert · January 11, 2025, 1:05am

Was running out of available memory during torchvision step, and needed to disable the Desktop GUI via sudo init 3 and the build completed:

REPOSITORY       TAG                       IMAGE ID       CREATED          SIZE
nanodb           r36.4.2                   aa8a9828aa11   5 minutes ago    15.1GB
nanodb           r36.4.2-nanodb            aa8a9828aa11   5 minutes ago    15.1GB
nanodb           r36.4.2-clip_trt          5d4f8b469172   6 minutes ago    15GB
nanodb           r36.4.2-torch2trt         226d87235e85   7 minutes ago    15GB
nanodb           r36.4.2-tensorrt          c8704e6c43c8   32 minutes ago   14.9GB
nanodb           r36.4.2-transformers      3202ddfb58e1   34 minutes ago   11.9GB
nanodb           r36.4.2-rust              0277cac7c464   36 minutes ago   11.4GB
nanodb           r36.4.2-huggingface_hub   861716bd61a7   37 minutes ago   9.99GB
nanodb           r36.4.2-torchvision       65b1dcd32bb7   38 minutes ago   9.98GB
nanodb           r36.4.2-faiss_lite        8202946adfc6   10 hours ago     9.95GB
nanodb           r36.4.2-faiss             0d3ba7e39cbe   10 hours ago     9.93GB
nanodb           r36.4.2-cuda-python       7f7fd3f11bad   11 hours ago     8.2GB
nanodb           r36.4.2-pytorch_2.5       7f137914f4bd   11 hours ago     8.06GB
nanodb           r36.4.2-onnx              0ba3f2a4559a   11 hours ago     7.14GB
nanodb           r36.4.2-cmake             46b48ff725ac   11 hours ago     7.11GB
nanodb           r36.4.2-numpy             470679f443f6   11 hours ago     7.05GB
nanodb           r36.4.2-python            4e9b47c13510   11 hours ago     6.99GB
nanodb           r36.4.2-cudnn_9.4         c416a92ae506   11 hours ago     6.92GB
nanodb           r36.4.2-cuda_12.6         96aad89635d7   11 hours ago     5.92GB
nanodb           r36.4.2-pip_cache_cu126   db5d38d31e3e   11 hours ago     734MB
nanodb           r36.4.2-build-essential   4ee0e57bf1c1   11 hours ago     734MB
ubuntu           22.04                     981912c48e9a   4 months ago     69.2MB
dustynv/nanodb   r36.2.0                   8118ef4e5cab   10 months ago    16GB

Advice: read the System Setup document before starting:

github.com

dusty-nv/jetson-containers/blob/master/docs/setup.md

# System Setup

Install the latest version of JetPack 4 on Nano/TX1/TX2, JetPack 5 on Xavier, or JetPack 6 on Orin.  The following versions are supported:

* JetPack 4.6.1+ (>= L4T R32.7.1)
* JetPack 5.1+  (>= L4T R35.2.1)
* JetPack 6.0 DP (L4T R36.2.0)
> [!NOTE]  
> <sup>- Building on/for x86 platforms isn't supported at this time (one can typically install/run packages the upstream way there)</sup><br>
> <sup>- The below steps are optional for [pulling/running](/docs/run.md) existing container images from registry, but recommended for building containers locally.</sup>

## Clone the Repo

This will download and install the jetson-containers utilities:

```bash
git clone https://github.com/dusty-nv/jetson-containers
bash jetson-containers/install.sh
```

This file has been truncated. show original

AastaLLL · January 13, 2025, 7:12am

Hi,

It looks like you have successfully built the container.
To optimize memory usage, you can also check the below document for guidance.

Thanks.

kenluebbert · January 14, 2025, 2:30pm

There remains an issue with the r36.4.2 indexing process where the validation step returns a warning for each image being validated:

22:23:21 | INFO | validating index with k=8
22:23:22 | WARNING | incorrect or duplicate index [0]  indexes=[-1 -1 -1 -1 -1 -1 -1 -1]  distances=[-3.4028235e+38 -3.4028235e+38 -3.4028235e+38 -3.4028235e+38
 -3.4028235e+38 -3.4028235e+38 -3.4028235e+38 -3.4028235e+38]

Search queries using this index return an incorrect list of images all with similarity values of 0.0237

IMPORTANT: using the r36.2.0 container pulled from:
https://hub.docker.com/r/dustynv/nanodb/tags
results an indexing process that runs 4x faster (than 36.4.2) and generates an index that is validated with few warnings or errors. The index query results are correct.

Not sure if I have correctly built the 36.4.2 container or if the 36.4.2 container has a bug in nanodb per: Nanodb Image Scan Not working

AastaLLL · January 15, 2025, 5:40am

Hi,

About the performance, do you run the identical test for the r36.2.0 and r36.4.2 containers?

Thanks.

kenluebbert · January 15, 2025, 2:26pm

Yes: both containers used the same set of files from:
wget http://images.cocodataset.org/zips/train2017.zip
wget http://images.cocodataset.org/zips/val2017.zip
wget http://images.cocodataset.org/zips/unlabeled2017.zip

From the logs, the indexing rates of my Jetson Orin Nano Super at MAXN, :
2.5 images per second using container r36.4.2 (elapsed time 28 hours)
10.9 images per second using container r36.2.0 (elapsed time 7 hours)

AastaLLL · January 16, 2025, 7:56am

Hi,

Not sure if this is related to the SDCard image.
Could you try to setup the JetPack 6.2 with NVMe SSD to see if it helps?

Thanks.

kenluebbert · January 16, 2025, 3:00pm

These tests were all run from JetPack 6.1 flashed to NVMe using sdkmanager, booted from NVMe. The SDCard was removed.

AastaLLL · January 20, 2025, 8:38am

Hi,

How about the performance?
Do you get the similar performance as r36.2.0?

Thanks.

Topic		Replies	Views
JetPack 4.6 Production Release with L4T 32.6.1 Jetson Nano	47	11949	March 10, 2022
NVIDIA Jetson Nano 2GB Developer Kit available now Jetson Nano	79	6304	March 10, 2022
Jetson container package torch2trt on nano orin 8gb Developement kit Jetson Orin Nano containers	2	641	September 6, 2023
Using my Jetson Nano for Preparing a Jetson Developer Kit for Use Jetson Nano	17	1380	October 14, 2021
Please provide more detailed guidance to flash Jetson Nano with Windows 11 PC Jetson Nano	10	2087	December 12, 2022
Error building l4t-jetpack docker image with runtime libraries Jetson AGX Orin docker	3	198	July 10, 2024
No module named nanodb AI Foundation Models and Endpoints jetson-inference	9	66	January 19, 2025
Nvidia Jetson Orin shows 6.3GB of RAM Jetson Orin Nano	36	4541	June 1, 2023
Jetpack 6.1 - docker cuda version mismatch Jetson Orin Nano docker	15	340	December 19, 2024
Jetson Nano with CUDA 11 and JetPack (TensorFlow2 compatibility) Jetson Nano jetpack , tensorflow , jetson-nano	10	1453	September 11, 2023

Jetson-containers does not build correctly

Related topics