All the input arrays must have same number of dimensions, but the array at index 0 has 2 dimension(s) and the array at index 2 has 3 dimension(s)

shehjadishan211 · July 29, 2024, 6:18am

I am trying to run nano_llm video.py on my ubuntu machine(not jetson). I have compiled the jetson_utils without libsoup. Loaded the model as hugging face model. This error has occured.

[gstreamer] gstEncoder -- codec not specified, defaulting to H.264
[gstreamer] gstEncoder -- pipeline launch string:
[gstreamer] appsrc name=mysource is-live=true do-timestamp=true format=3 ! x264enc name=encoder bitrate=4000 speed-preset=ultrafast tune=zerolatency ! video/x-h264 ! h264parse ! qtmux ! filesink location=./data/my_output.mp4 
[video]  created gstEncoder from file:///home/sigmind/VideoDownloader/nano_llm_try/./data/my_output.mp4
------------------------------------------------
gstEncoder video options:
------------------------------------------------
  -- URI: file:///home/sigmind/VideoDownloader/nano_llm_try/./data/my_output.mp4
     - protocol:  file
     - location:  ./data/my_output.mp4
     - extension: mp4
  -- deviceType: file
  -- ioType:     output
  -- codec:      H264
  -- codecType:  cpu
  -- frameRate:  30
  -- bitRate:    4000000
  -- numBuffers: 4
  -- zeroCopy:   true
------------------------------------------------
[gstreamer] gstDecoder -- onPreroll()
[gstreamer] gstreamer message stream-start ==> pipeline0
[gstreamer] gstreamer message duration-changed ==> mpeg4vparse1
[gstreamer] gstreamer mysink taglist, video-codec=(string)"MPEG-4\ video", maximum-bitrate=(uint)259793, bitrate=(uint)259793, language-code=(string)en;
[gstreamer] gstreamer mysink taglist, title=(string)"Sakri\ Dengarours\ Road\ Accident\ Live\ CCTV\ Footage", artist=(string)rahulssharma00, comment=(string)"https://www.youtube.com/watch\?v\=N28T8z4PN7s", description=(string)"Sakri\ Dengarours\ Road\ Accident\ Live\ CCTV\ Footage\,\012live\ road\ accidents\ in\ hyderabad\,\ live\ road\ accident\ youtube\,\ srs\ travels\ live\ road\ accident\,\ road\ accident\ live\ in\ kerala\,\ truck\ road\ accident\ live\,\ ghat\ road\ accident\ live\,\ live\ road\ bus\ accident\,\ \012ktm\ Bike\ road\ accident\ live\,\012train\ compilation\ india\,\012train\ compilation\ 1\,\012train\ compilation\ youtube\,\012train\ to\ busan\ compilation\,\012fastest\ train\ compilation\,\012train\ falls\ compilation\,\012train\ hits\ car\ compilation\,\012local\ train\ compilation\,\012train\ collisions\ -\ mega\ compilation\ 2013\,\012train\ snow\ plow\ compilation\,\012train\ passing\ compilation\,\012road\ train\ compilation\,\012why\ we\ train\ compilation", datetime=(datetime)2019, encoder=(string)Lavf59.34.102, private-qt-tag=(sample)0000028f6c6465730000028764617461000000010000000053616b72692044656e6761726f75727320526f6164204163636964656e74204c697665204343545620466f6f746167652c0a6c69766520726f6164206163636964656e747320696e206879646572616261642c206c69766520726f6164206163636964656e7420796f75747562652c207372732074726176656c73206c69766520726f6164206163636964656e742c20726f6164206163636964656e74206c69766520696e206b6572616c612c20747275636b20726f6164206163636964656e74206c6976652c206768617420726f6164206163636964656e74206c6976652c206c69766520726f616420627573206163636964656e742c200a6b746d2042696b6520726f6164206163636964656e74206c6976652c0a747261696e20636f6d70696c6174696f6e20696e6469612c0a747261696e20636f6d70696c6174696f6e20312c0a747261696e20636f6d70696c6174696f6e20796f75747562652c0a747261696e20746f20627573616e20636f6d70696c6174696f6e2c0a6661737465737420747261696e20636f6d70696c6174696f6e2c0a747261696e2066616c6c7320636f6d70696c6174696f6e2c0a747261696e20686974732063617220636f6d70696c6174696f6e2c0a6c6f63616c20747261696e20636f6d70696c6174696f6e2c0a747261696e20636f6c6c6973696f6e73202d206d65676120636f6d70696c6174696f6e20323031332c0a747261696e20736e6f7720706c6f7720636f6d70696c6174696f6e2c0a747261696e2070617373696e6720636f6d70696c6174696f6e2c0a726f616420747261696e20636f6d70696c6174696f6e2c0a77687920776520747261696e20636f6d70696c6174696f6e:None:R3N0U2VnbWVudCwgZmxhZ3M9KEdzdFNlZ21lbnRGbGFncylHU1RfU0VHTUVOVF9GTEFHX05PTkUsIHJhdGU9KGRvdWJsZSkxLCBhcHBsaWVkLXJhdGU9KGRvdWJsZSkxLCBmb3JtYXQ9KEdzdEZvcm1hdClHU1RfRk9STUFUX1RJTUUsIGJhc2U9KGd1aW50NjQpMCwgb2Zmc2V0PShndWludDY0KTAsIHN0YXJ0PShndWludDY0KTAsIHN0b3A9KGd1aW50NjQpMTg0NDY3NDQwNzM3MDk1NTE2MTUsIHRpbWU9KGd1aW50NjQpMCwgcG9zaXRpb249KGd1aW50NjQpMCwgZHVyYXRpb249KGd1aW50NjQpMTg0NDY3NDQwNzM3MDk1NTE2MTU7AA__:YXBwbGljYXRpb24veC1nc3QtcXQtbGRlcy10YWcsIHN0eWxlPShzdHJpbmcpaXR1bmVzOwA_, container-format=(string)"ISO\ MP4/M4A";
[gstreamer] gstreamer mysink taglist, video-codec=(string)"MPEG-4\ Video\ \(Simple\ Profile\)", maximum-bitrate=(uint)259793, bitrate=(uint)259793, language-code=(string)en;
[gstreamer] gstBufferManager -- map buffer size was less than max size (337920 vs 337951)
[gstreamer] gstBufferManager recieve caps:  video/x-raw, format=(string)I420, width=(int)640, height=(int)352, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)0/1
[gstreamer] gstBufferManager -- recieved first frame, codec=MPEG4 format=i420 width=640 height=352 size=337951
[cuda]   allocated 4 ring buffers (337951 bytes each, 1351804 bytes total)
[cuda]   allocated 4 ring buffers (8 bytes each, 32 bytes total)
[gstreamer] gstreamer changed state from READY to PAUSED ==> mysink
[gstreamer] gstreamer changed state from READY to PAUSED ==> pipeline0
[gstreamer] gstreamer message async-done ==> pipeline0
[gstreamer] gstreamer message new-clock ==> pipeline0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> mysink
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> capsfilter0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> decoder
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> mpeg4vparse1
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> queue0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> qtdemux1
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> filesrc0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> pipeline0
[cuda]   allocated 4 ring buffers (675840 bytes each, 2703360 bytes total)
[cuda]   allocated 2 ring buffers (337920 bytes each, 675840 bytes total)
[gstreamer] gstEncoder -- starting pipeline, transitioning to GST_STATE_PLAYING
[gstreamer] gstreamer changed state from NULL to READY ==> filesink0
[gstreamer] gstreamer changed state from NULL to READY ==> qtmux0
[gstreamer] gstreamer changed state from NULL to READY ==> h264parse0
[gstreamer] gstreamer changed state from NULL to READY ==> capsfilter1
[gstreamer] gstreamer changed state from NULL to READY ==> encoder
[gstreamer] gstreamer changed state from NULL to READY ==> mysource
[gstreamer] gstreamer changed state from NULL to READY ==> pipeline1
[gstreamer] gstreamer changed state from READY to PAUSED ==> qtmux0
[gstreamer] gstreamer changed state from READY to PAUSED ==> h264parse0
[gstreamer] gstreamer changed state from READY to PAUSED ==> capsfilter1
[gstreamer] gstreamer changed state from READY to PAUSED ==> encoder
[gstreamer] gstreamer stream status CREATE ==> src
[gstreamer] gstreamer changed state from READY to PAUSED ==> mysource
[gstreamer] gstreamer changed state from READY to PAUSED ==> pipeline1
[gstreamer] gstreamer message new-clock ==> pipeline1
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> qtmux0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> h264parse0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> capsfilter1
[gstreamer] gstreamer stream status ENTER ==> src
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> encoder
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> mysource
[gstreamer] gstEncoder -- new caps: video/x-raw, width=640, height=352, format=(string)I420, framerate=30/1
[gstreamer] gstreamer message latency ==> encoder
[gstreamer] gstreamer message stream-start ==> pipeline1
[gstreamer] gstreamer changed state from READY to PAUSED ==> filesink0
[gstreamer] gstreamer message async-done ==> pipeline1
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> filesink0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> pipeline1
Enbedding shape:(1, 32)
Enbedding shape:(1, 7)
Enbedding shape:(1, 196, 2560)
Enbedding shape:(1, 13)
Traceback (most recent call last):
  File "/home/sigmind/VideoDownloader/nano_llm_try/video.py", line 105, in <module>
    embedding, _ = chat_history.embed_chat()
  File "/home/sigmind/VideoDownloader/nano_llm_try/nano_llm/chat/history.py", line 385, in embed_chat
    embeddings = np.concatenate(embeddings, axis=1) #, position
  File "<__array_function__ internals>", line 180, in concatenate
ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 2 dimension(s) and the array at index 2 has 3 dimension(s)

kesong · July 29, 2024, 6:29am

Can you share more on your project? Are you running on x86 dGPU platform? Can you share the reproduce steps of the issue?

shehjadishan211 · July 29, 2024, 6:40am

Sorry. Yes, I am on a x86 dGPU platform. I have compiled the jetson_utils from the source. In video.py model(api=hf) thats all the changes I have made into the code structure and pipeline. Maybe enough to reproduce.

shehjadishan211 · July 30, 2024, 6:22am

@kesong Kindly, let me know if it was reproducible.

dusty_nv · July 30, 2024, 4:48pm

@shehjadishan211 as mentioned on GitHub, I have not ported this project to x86 so YMMV - I would start running a text LLM first and see if you can get that working. Then can try VLM. Also on x86 you can just run the upstream Llava repo (or VILA, Moondream, ect)

shehjadishan211 · July 31, 2024, 7:13am

@dusty_nv I need to analyze real time video feed like video.py(nanoLLM) offers but on a x86 machine. It would be helpful if you could refer to a repo for that.

dusty_nv · August 1, 2024, 4:16am

How I would do it on x86…you presumably have more compute available, and could just install/run the original VLM repos like Llava, VILA, Prismatic, ect (which those are based on top of HuggingFace Transformers), or quantized with llama.cpp for those available in GGUF. And then for the camera you could use cv2.VideoCapture

shehjadishan211 · August 1, 2024, 6:25am

TYVM. Got the Idea!

system · August 15, 2024, 6:25am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Problem with C++ gstreamer pipeline Jetson Nano gstreamer	7	3300	September 19, 2021
Three stream GStreamer pipeline stuck Jetson TX2	7	4825	October 18, 2021
Gstreamer Pipeline issue : PosixMemMap:84 mmap failed Jetson Orin Nano gstreamer	15	690	April 25, 2024
H.264 decoding failure Jetson Nano gstreamer	12	3765	October 18, 2021
Gstreamer pipeline produces greyscale video stream (using NV12 and nvvvidconv) Jetson Nano gstreamer	27	1854	November 20, 2023
Unable to rotate video stream in Jetson Inference Jetson Nano camera	8	76	August 13, 2024
Problem with using openmax in tegra k1 (driver R19.3) Jetson TK1	21	7729	August 16, 2015
Multiple output resolutions Gstreamer Jetson Nano gstreamer	9	3371	October 18, 2021
Video scaling with GStreamer element 'videoscale' and 'nvdrmvideosink set-mode=1' Jetson TX2 gstreamer , video	18	8881	March 5, 2022
./video-viewer /dev/video0 from github Hello AI World failing Jetson Nano camera , jetson-inference , gstreamer	16	3129	October 15, 2021

All the input arrays must have same number of dimensions, but the array at index 0 has 2 dimension(s) and the array at index 2 has 3 dimension(s)

Related topics