Imagenet-camera with webcam

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

Hi,
i’m trying to run imagenet-camera demo with Logitech Webcam C525.i am getting complete blank screen with following

[gstreamer] initialized gstreamer, version 1.14.1.0
[gstreamer] gstCamera attempting to initialize with GST_SOURCE_NVCAMERA
[gstreamer] gstCamera pipeline string:
v4l2src device=/dev/video0 ! video/x-raw, width=(int)1280, height=(int)720, format=RGB ! videoconvert ! video/x-raw, format=RGB ! videoconvert !appsink name=mysink
[gstreamer] gstCamera successfully initialized with GST_SOURCE_V4L2

imagenet-camera:  successfully initialized video device
    width:  1280
   height:  720
    depth:  24 (bpp)

imageNet -- loading classification network model from:
         -- prototxt     networks/googlenet.prototxt
         -- model        networks/bvlc_googlenet.caffemodel
         -- class_labels networks/ilsvrc12_synset_words.txt
         -- input_blob   'data'
         -- output_blob  'prob'
         -- batch_size   2

[TRT]  TensorRT version 5.0.3
[TRT]  desired precision specified for GPU: FASTEST
[TRT]  requested fasted precision for device GPU without providing valid calibrator, disabling INT8
[TRT]  native precisions detected for GPU:  FP32, FP16, INT8
[TRT]  selecting fastest native precision for GPU:  FP16
[TRT]  attempting to open engine cache file networks/bvlc_googlenet.caffemodel.2.1.GPU.FP16.engine
[TRT]  loading network profile from engine cache... networks/bvlc_googlenet.caffemodel.2.1.GPU.FP16.engine
[TRT]  device GPU, networks/bvlc_googlenet.caffemodel loaded
[TRT]  device GPU, CUDA engine context initialized with 2 bindings
[TRT]  networks/bvlc_googlenet.caffemodel input  binding index:  0
[TRT]  networks/bvlc_googlenet.caffemodel input  dims (b=2 c=3 h=224 w=224) size=1204224
[cuda]  cudaAllocMapped 1204224 bytes, CPU 0x21b996000 GPU 0x21b996000
[TRT]  networks/bvlc_googlenet.caffemodel output 0 prob  binding index:  1
[TRT]  networks/bvlc_googlenet.caffemodel output 0 prob  dims (b=2 c=1000 h=1 w=1) size=8000
[cuda]  cudaAllocMapped 8000 bytes, CPU 0x21bb96000 GPU 0x21bb96000
device GPU, networks/bvlc_googlenet.caffemodel initialized.
[TRT]  networks/bvlc_googlenet.caffemodel loaded
imageNet -- loaded 1000 class info entries
networks/bvlc_googlenet.caffemodel initialized.
default X screen 0:   1920 x 1080
[OpenGL]  glDisplay display window initialized
[OpenGL]   creating 1280x720 texture
loaded image  fontmapA.png  (256 x 512)  2097152 bytes
[cuda]  cudaAllocMapped 2097152 bytes, CPU 0x21bd96000 GPU 0x21bd96000
[cuda]  cudaAllocMapped 8192 bytes, CPU 0x21bb98000 GPU 0x21bb98000
[gstreamer] gstreamer transitioning pipeline to GST_STATE_PLAYING
[gstreamer] gstreamer changed state from NULL to READY ==> mysink
[gstreamer] gstreamer changed state from NULL to READY ==> videoconvert1
[gstreamer] gstreamer changed state from NULL to READY ==> capsfilter1
[gstreamer] gstreamer changed state from NULL to READY ==> videoconvert0
[gstreamer] gstreamer changed state from NULL to READY ==> capsfilter0
[gstreamer] gstreamer changed state from NULL to READY ==> v4l2src0
[gstreamer] gstreamer changed state from NULL to READY ==> pipeline0
[gstreamer] gstreamer changed state from READY to PAUSED ==> videoconvert1
[gstreamer] gstreamer changed state from READY to PAUSED ==> capsfilter1
[gstreamer] gstreamer changed state from READY to PAUSED ==> videoconvert0
[gstreamer] gstreamer changed state from READY to PAUSED ==> capsfilter0
[gstreamer] gstreamer stream status CREATE ==> src
[gstreamer] gstreamer changed state from READY to PAUSED ==> v4l2src0
[gstreamer] gstreamer changed state from READY to PAUSED ==> pipeline0
[gstreamer] gstreamer msg new-clock ==> pipeline0
[gstreamer] gstreamer stream status ENTER ==> src
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> videoconvert1
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> capsfilter1
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> videoconvert0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> capsfilter0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> v4l2src0
[gstreamer] gstreamer msg stream-start ==> pipeline0
[gstreamer] gstCamera onEOS
[gstreamer] gstreamer v4l2src0 ERROR Internal data stream error.
[gstreamer] gstreamer Debugging info: gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
[gstreamer] gstreamer changed state from READY to PAUSED ==> mysink

imagenet-camera:  camera open for streaming

imagenet-camera:  failed to capture frame
imagenet-camera:  failed to convert from NV12 to RGBA
[TRT]  imageNet::Classify( 0x(nil), 1280, 720 ) -> invalid parameters
[cuda]   cudaNormalizeRGBA((float4*)imgRGBA, make_float2(0.0f, 255.0f), (float4*)imgRGBA, make_float2(0.0f, 1.0f), camera->GetWidth(), camera->GetHeight())
[cuda]      invalid device pointer (error 17) (hex 0x11)
[cuda]      /home/nvidia/Documents/jetson-inference/imagenet-camera/imagenet-camera.cpp:193
[cuda]   registered 14745600 byte openGL texture for interop access (1280x720)

imagenet-camera:  failed to capture frame
imagenet-camera:  failed to convert from NV12 to RGBA
[TRT]  imageNet::Classify( 0x(nil), 1280, 720 ) -> invalid parameters
[cuda]   cudaNormalizeRGBA((float4*)imgRGBA, make_float2(0.0f, 255.0f), (float4*)imgRGBA, make_float2(0.0f, 1.0f), camera->GetWidth(), camera->GetHeight())
[cuda]      invalid device pointer (error 17) (hex 0x11)
[cuda]      /home/nvidia/Documents/jetson-inference/imagenet-camera/imagenet-camera.cpp:193

v4l2-ctl -V output is

Format Video Capture:
	Width/Height      : 640/480
	Pixel Format      : 'YUYV'
	Field             : None
	Bytes per Line    : 1280
	Size Image        : 614400
	Colorspace        : sRGB
	Transfer Function : Default (maps to sRGB)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Limited Range)
	Flags             :

lsusb output is

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 006: ID 046d:0826 Logitech, Inc. HD Webcam C525
Bus 001 Device 004: ID 0461:4d22 Primax Electronics, Ltd 
Bus 001 Device 003: ID 413c:2003 Dell Computer Corp. Keyboard
Bus 001 Device 002: ID 1a40:0101 Terminus Technology Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

why am i getting blank screen ? is webcam not suitable (I already changed DEFAULT_CAMERA value) ?
if its suitable how to get past that.

2

Hi anilkunchalaece, can you try applying this patch for supporting YUY2 format?

[url]Jetson Xavier + Logitech C920 Issue · Issue #267 · dusty-nv/jetson-inference · GitHub

3

Thanks, it solved the issue. isn’t better to add the same to two days a demo doc ?

4

There is a compatibility issue that needs to be resolved, where on TX1/TX2 kernel version the previous RGB format works, but on Xavier kernel version YUY2 format is required. It’s on my todo list to explore a way of detecting this at runtime.

5

Glad to know that.
Btw how to extend that demo ?
Can you suggest me few references on how to extend that demo ?

Ideally i like to interface it with some micro-controller.

6

The output from the image classifier is provided in the img_class and confidence variables returned here: https://github.com/dusty-nv/jetson-inference/blob/9181361b1fecb4a969b997ac13d01fe44b233e30/imagenet-camera/imagenet-camera.cpp#L158

Take these results, and provide them to your micro-controller interface or whatnot.
Note that to get a string representation of the image class from the provided class index, use the imageNet::GetClassDesc() function like so:

const char* class_description = net->GetClassDesc(img_class);
7

OK, a patch has been checked into master which automatically switches to YUY2 format for USB cameras on L4T R31 / JetPack 4. See commit 9be689

8

Thanks.