DS app running on SD, but not on eMMC (image problem?)

Maybe you can skip reading. Seems, I had an incomplete flash this run. Read on below, please

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
Jetson Nano eMMC
• DeepStream Version
5.1
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I’m having a very weird problem. My DS application runs very well from SD. I was using the latest 5.1 SD image from the Nvidia side so far, installed GStreamer and DS SDK manually after having updated/upgraded the system.

Today I got an eMMC module. Mount and flash OK. I flashed using the Nvidia SDK manager 1.5 on Ubuntu and completed the installation using monitor and keyboard.

I tried on different ways make my app run, to no avail so far.

  1. Using the SDK manager to create an image containing system, JetPack and DS SDK, all latest
  2. Using the SDK manager to create an image as above, but not containing DS in order to install it on the device later on.

My basic setup after initial flash on device (after being up and running) is:

sudo apt update && sudo apt upgrade -y
sudo apt install -y nano python3-pip
sudo apt install -y libssl1.0.0 libgstreamer1.0-0 \
  gstreamer1.0-tools gstreamer1.0-plugins-good \
  gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly \
  gstreamer1.0-plugins-base gstreamer1.0-libav \
  libgstrtspserver-1.0-0 libjansson4=2.11-1
sudo apt install deepstream-5.1 -y
sudo nvpmodel -m 0 && sudo jetson_clocks
sudo apt autoremove -y

As said: I have done this 100 times from SD image, but now from eMMC I cannot make my app run (I know this sounds weird, but it is what it is)

My app traces this at startup, already giving a hint, what’s going wrong:

2021-04-29 18:59:02,142 inference.py    INFO    : JetPack SDK version: # R32 (release), REVISION: 5.1, GCID: 26202423, BOARD: t210ref, EABI: aarch64, DATE: Fri Feb 19 16:45:52 UTC 2021 
2021-04-29 18:59:02,143 inference.py    INFO    : DeepStream SDK version: Version: 5.1 GCID: 26140461 EABI:  DATE: Tue Feb 16 09:54:48 UTC 2021 
2021-04-29 18:59:02,195 ranging.py      INFO    : Configured model: 'resnet10-caffemodel'
2021-04-29 18:59:02,197 ranging.py      INFO    : Configured object heights: {'person': {'abbr': 'p', 'height': 1700}}
2021-04-29 18:59:02,197 ranging.py      INFO    : Configured objects: ['person']
Traceback (most recent call last):
  File "inference.py", line 647, in <module>
    inference = Inference(args, labels, log)
  File "inference.py", line 163, in __init__
    pgie.set_property('config-file-path', args.config["pgie-config-file"])
AttributeError: 'NoneType' object has no attribute 'set_property'
2021-04-29 18:59:02,508 inference.py    ERROR   : Exception: 'NoneType' object has no attribute 'set_property'
2021-04-29 18:59:02,510 inference.py    INFO    : Terminating...

The error points to a pgie object being None.

I tried to reproduce in a Python console on a fresh booted system, and indeed: pgie is None after this sequence:

import sys
sys.path.append('/opt/nvidia/deepstream/deepstream/lib')
import gi
gi.require_version('Gst', '1.0')
from gi.repository import GObject, Gst

GObject.threads_init()
Gst.init(None)
[]

pgie = Gst.ElementFactory.make('nvinfer', 'primary-inference')
print(pgie)
None

I currently cannot compare this with the results from SD, but I’m already having doubts with the [] as result of Gst.init(None), so I cannot even initialize GStreamer…

If you compare the initialization sequence with the official Python sample and also the creation of pgie, you will note similarities:

Imports:

Gst-Init:

PGIE object creation:

So what might be the problem? I suppose it does have nothing to do with the fact, that I’m now using eMMC instead of an SD, but since I’m using a default image for the SD and a “self-made” for the eMMC maybe there is a difference?

Here BTW a screenshot of the USB cam sample app of the DS Python samples:

Sorry for the long post :/

I think I missed the moment, when the Jetson restarted in the middle of the installation and a login screen appeared on my Ubuntu. Doing it again right now.

BTW: Is there any way around this “splendid” SDK Manager? I have read about “flash.sh”, would that work instead? Using the “Manager” to stitch together the deployment and flash it using good old console commands?

The flash is so lame… Installation takes an hour already and is at 51 %, this is not normal, IMHO…

OK, now it is stuck. Hangs at 51.65 %, my Ubuntu tells me every now and then, that the “Activation of the network connection failed”… Oh, boy…what to do w/o damaging it all?

Paused and resumed now. But - I can literally watch the bits walking over the line… 0.01 % per second… What a mess…

OK, having abandoned this “experience” now after my Nano told me, that he only has 500 MBs left…

df -H on the Nano said 14.5 GB full…

During installation. Check this out…

Maybe someone could please comment on these two videos: Is this the right behaviour of flashing a 16 GB eMMC to a Jetson Nano Dev Kit via USB?

Video 1: Shows the progress: 0.01% upload progress per second…

The second video is way more impressive: It shows the box hanging at 99 % flashing CUDA AI (NOTE: I’m just flashing the essentials, not even DS) while the box is saying, that it only has 400 MB left.

Please comment and advise.

Hi,
We have seen storage shortage in running DS SDK on Xaviern NX emmc, and it is listed in

DeepStream cannot be installed on the current 16 GB Xavier NX production modules since Jetpack software takes the entire 16 GB emmc memory space. We recommend using Xavier NX developer kits with 32 GB SD card.

Jotson Nano is not listed, but for more complicated usecases, it is still possible to hit certain issues. We would like to suggest move rootfs to external storage. This would need to use custom board instead of Jetson Nano devkit.
For booting from external device, please refer to
Jetson/L4T/Boot From External Device - eLinux.org

Wait, what? To be honest - you must be kidding, do you?

To recap the entire sad story: I needed > 100 posts in a thread with one of the experts here just to figure out, that it is NOT possible to make the SD card read-only by using an overlay ramfs (SD card damage protection - #5 by linuxdev). Then the suggestion of one of you team mates was, eMMC would be the solution to use for having a protected system (Make Jetson Nano SD /dev/root read-only - #3 by JerryChang) and now this after all? It is not possible at all to achieve a protection of a Jetson Nano, because it is impossible to flash the eMMC???

I have no words…

Hi,
For now security feature is on emmc and DeepStream SDK requires larger free space. If you need read-only rootfs + DeepStream SDK, it may not work in current L4T releases. We would check to reduce memory/storage usage of DeepStream SDK in future release.

Meanwhile I have serious doubts that the security fuse would work at all. I recall that one of your startup scripts (nvidia_init.sh?) was attempting to create sym links at boot - this was one of the reasons why I abandoned all attempts to prevent your system from using the rootfs at all. How do you solve this problem on a fused eMMC?

I also would like to point out, that I didn’t make something “wild”: I was using your SDK manager, I was selecting a combination, which was offered me as valid (Nano dev kit/eMMC, option 3 in target selection dialog), but it didn’t work at all.

I also tried to break the installation process after having installed and booted the base image and proceed with instructions found here How to Install JetPack :: NVIDIA JetPack Documentation to no avail.

A well developed, well working application ran against the wall this night just because of this unexpected and unresolvable problem.

Log of my attempts to solve the problem not by using an overlay fs, but by preventing components to write into rootfs but to ramdisk instead - didn’t work either.

https://gist.github.com/neilyoung/5a680052dbdce6258183754031c023e9

…and I hope you all realize that you have rendered the Nano to be a useless device for productive use cases. And this in situations in which it could be a real competitor to a Raspberry PI 4. If this wasn’t your intention at all, then rest satisfied.

Just to remind you of the poor outcome of a lot of hours of useless work on this:

  • it is currently not possible to protect the rootfs of a Nano from being corrupted by applying common mechanisms like initramfs. That this has been possible up to end of last year can be seen on various sources on the web

  • it is currently not possible to flash the combination of base system, JetPack 4.5 and DeepStream 5.1 to a 16 GB eMMC in order to use the protection mechanism of the eMMC, because JetPack has become FatPack during the pandemic as all of us.

OK, 185 MB left on the 16 GB eMMC after installing root system and JetPack SDK.

What can I do to get some more free space? E.g. I don’t need the entire Computer Vision stuff. What can I de-install? I also don’t need Multimedia API. I most likely would also not need TensorRT because I don’t intend to train on the device.

Please advice.

My removal list so far, w/o knowing, if this all will work

All prefixed with sudo apt remove...

chromium*
firefox*
thunderbird*
libreoffice*
libnvinfer-samples
cuda-documentation-10-2
cuda-samples-10-2
libvisionworks-samples
libopencv-samples
man-db

At least I have 2.9 GB free after having installed JP/DS and the system.

What else? Hints?

So now back to the initial problem: I think I should have all required components installed, but the problem as reported above, remains:

2021-04-30 12:15:48,499 ranging.py      INFO    : Configured objects: ['person']
Traceback (most recent call last):
  File "inference.py", line 647, in <module>
    inference = Inference(args, labels, log)
  File "inference.py", line 163, in __init__
    pgie.set_property('config-file-path', args.config["pgie-config-file"])
AttributeError: 'NoneType' object has no attribute 'set_property'

What is this, guys??

EDIT: No, it was just an illusion. JetPack wasn’t truly installed. I suppose, already the first warning on my system (“500 MB left”) was the point, at which the SDK manager was unable to install the things completely. At least he told me, it was fine. It wasn’t.

It is a mess, really…:(

Verified and double-checked. There is a problem with whatsoever. Even if I - at least to all my knowledge - having installed all required components, I’m still not able to instantiate GST and create a PGIE object.

I will open up another thread, since this is pretty much polluted and burned.