No space left on device

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.0.10816
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Hi,

I am trying to copy tensorrt to the device according to the guide. But I incur in some memory issues, in particular when trying to transfer I get the following output:

sample_uff_plugin_v2_ext_debug                                                                                      100% 7420KB  11.2MB/s   00:00    
sample_mnist_debug                                                                                                  100% 7363KB  13.8MB/s   00:00    
sample_mnist_api                                                                                                    100% 2278KB  17.0MB/s   00:00    
sample_uff_mnist_debug                                                                                              100% 7327KB  15.6MB/s   00:00    
sample_uff_faster_rcnn_debug                                                                                        100% 7833KB  16.7MB/s   00:00    
sample_io_formats_debug                                                                                             100% 7330KB  20.4MB/s   00:00    
sample_int8                                                                                                         100% 2245KB  14.4MB/s   00:00    
sample_io_formats                                                                                                   100% 2203KB  15.4MB/s   00:00    
sample_onnx_mnist_coord_conv_ac_debug                                                                               100% 7350KB  12.5MB/s   00:00    
sample_algorithm_selector_debug                                                                                     100% 7606KB  12.4MB/s   00:00    
sample_dynamic_reshape                                                                                              100% 2174KB  13.4MB/s   00:00    
sample_int8_debug                                                                                                   100% 7500KB  20.1MB/s   00:00    
sample_uff_faster_rcnn                                                                                              100% 2388KB  18.7MB/s   00:00    
sample_fasterRCNN_debug                                                                                             100% 7676KB  14.2MB/s   00:00    
sample_googlenet_debug                                                                                              100% 7298KB  14.6MB/s   00:00    
sample_uff_mask_rcnn_debug                                                                                          100% 7549KB  14.6MB/s   00:00    
sample_uff_mask_rcnn                                                                                                100% 2268KB  18.1MB/s   00:00    
sample_onnx_mnist_coord_conv_ac                                                                                     100% 2213KB  10.9MB/s   00:00    
trtexec                                                                                                             100%  504KB   7.4MB/s   00:00    
sample_fasterRCNN                                                                                                   100% 2300KB  12.5MB/s   00:00    
sample_mnist                                                                                                        100% 2211KB  14.9MB/s   00:00    
sample_uff_ssd                                                                                                      100% 2253KB  13.0MB/s   00:00    
sample_mnist_api_debug                                                                                              100% 7585KB  12.2MB/s   00:00    
sample_onnx_mnist                                                                                                   100% 2189KB  13.9MB/s   00:00    
sample_char_rnn_debug                                                                                                 0%    0     0.0KB/s   --:-- ETAscp: /home/orin-1/tensorrt/bin/sample_char_rnn_debug: No space left on device
sample_char_rnn_debug                                                                                               100% 7962KB  16.0MB/s   00:00    
sample_uff_mnist                                                                                                      0%    0     0.0KB/s   --:-- ETAscp: /home/orin-1/tensorrt/bin/sample_uff_mnist: No space left on device
sample_uff_mnist                                                                                                    100% 2188KB  25.2MB/s   00:00    
scp: /home/orin-1/tensorrt/bin/chobj: No space left on device
sample_uff_ssd_debug                                                                                                  0%    0     0.0KB/s   --:-- ETAscp: /home/orin-1/tensorrt/bin/sample_uff_ssd_debug: No space left on device
sample_uff_ssd_debug                                                                                                100% 7494KB  16.7MB/s   00:00    
sample_algorithm_selector                                                                                             0%    0     0.0KB/s   --:-- ETAscp: /home/orin-1/tensorrt/bin/sample_algorithm_selector: No space left on device
sample_algorithm_selector                                                                                           100% 2282KB  12.8MB/s   00:00    
sample_ssd_debug                                                                                                      0%    0     0.0KB/s   --:-- ETAscp: /home/orin-1/tensorrt/bin/sample_ssd_debug: No space left on device
sample_ssd_debug                                                                                                    100% 7491KB  21.2MB/s   00:00    
sample_dynamic_reshape_debug                                                                                          0%    0     0.0KB/s   --:-- ETAscp: /home/orin-1/tensorrt/bin/sample_dynamic_reshape_debug: No space left on device
sample_dynamic_reshape_debug                                                                                        100% 7313KB  13.6MB/s   00:00    
sample_char_rnn                                                                                                       0%    0     0.0KB/s   --:-- ETAscp: /home/orin-1/tensorrt/bin/sample_char_rnn: No space left on device
sample_char_rnn                                                                                                     100% 2410KB  11.4MB/s   00:00    
sample_int8_api_debug                                                                                                 0%    0     0.0KB/s   --:-- ETAscp: /home/orin-1/tensorrt/bin/sample_int8_api_debug: No space left on device
sample_int8_api_debug                                                                                               100% 7731KB  10.3MB/s   00:00    
scp: /home/orin-1/tensorrt/bin/dchobj: No space left on device
sample_uff_plugin_v2_ext                                                                                              0%    0     0.0KB/s   --:-- ETAscp: /home/orin-1/tensorrt/bin/sample_uff_plugin_v2_ext: No space left on device
sample_uff_plugin_v2_ext                                                                                            100% 2232KB   8.0MB/s   00:00    
trtexec_debug                                                                                                         0%    0     0.0KB/s   --:-- ETAscp: /home/orin-1/tensorrt/bin/trtexec_debug: No space left on device
trtexec_debug                                                                                                       100% 7374KB  15.3MB/s   00:00    
sample_onnx_mnist_debug                                                                                               0%    0     0.0KB/s   --:-- ETAscp: /home/orin-1/tensorrt/bin/sample_onnx_mnist_debug: No space left on device
sample_onnx_mnist_debug                                                                                             100% 7327KB  18.4MB/s   00:00    
sample_ssd                                                                                                            0%    0     0.0KB/s   --:-- ETAscp: /home/orin-1/tensorrt/bin/sample_ssd: No space left on device
sample_ssd                                                                                                          100% 2250KB  25.7MB/s   00:00    
sample_int8_api                                                                                                       0%    0     0.0KB/s   --:-- ETAscp: /home/orin-1/tensorrt/bin/sample_int8_api: No space left on device
sample_int8_api                                                                                                     100% 2334KB  21.2MB/s   00:00    
sample_googlenet                                                                                                      0%    0     0.0KB/s   --:-- ETAscp: /home/orin-1/tensorrt/bin/sample_googlenet: No space left on device
sample_googlenet                                                                                                    100% 2184KB  27.7MB/s   00:00    
scp: /home/orin-1/tensorrt/samples: No space left on device
scp: /home/orin-1/tensorrt/data: No space left on device

Which is a system still without anything else installed more than what is installed by default with the docker flashing.

Also the system seems slowed down and despite the CPUs seem free of load, I see a process taking a lot of memory - or just declaring so:

/usr/lib/xorg/Xrg vt2 -displayfd 3 -auth /run/user/1000/gdm/Xauthority -background none -noreset -keeptty -verbose 3

There are 3 of these processes with VIRT=24.1G.

Kindly let me know how could we solve these issues.

Thank you

2

Dear @Leonardo-MAI,
Could you restart the machine and see if those processes still run? Also, please share df -h output on target?

3

Dear @SivaRamaKrishnaNV,

Yes, after restart the processes are still running.
The output of df -h:

Filesystem       Size  Used Avail Use% Mounted on
/dev/vblkdev0p1   27G  6.2G   19G  25% /
none              14G     0   14G   0% /dev
tmpfs             14G  8.4M   14G   1% /dev/shm
tmpfs            2.8G  2.1M  2.8G   1% /run
tmpfs            5.0M     0  5.0M   0% /run/lock
tmpfs             14G     0   14G   0% /sys/fs/cgroup
tmpfs             14G     0   14G   0% /mnt
/dev/vblkdev1     32M  200K   28M   1% /mnt/persistent/metadata
/dev/vblkdev3    128M  5.3M  105M   5% /mnt/persistent/data
overlay           32M  200K   28M   1% /etc
overlay          128M  5.3M  105M   5% /home
tmpfs            2.8G   12K  2.8G   1% /run/user/110
tmpfs            2.8G   20K  2.8G   1% /run/user/1000

But still persists the above error of No space left on device.

Thanks

4

Dear @Leonardo-MAI,
I see 19GB available in / and 14GB in /mnt . May I know where you are trying to copy the data ?

6

You are right, I was saving them to /home which I see myself having now 128MB, but it wasn’t the case before the last flashing. Why do you think it changed -I didn’t change it- the folder the memory is mounted to?

7

Dear @Leonardo-MAI,
I am not sure about what caused it. You may use /mnt partition for your testing test

1 Like
9

Dear @Leonardo-MAI,
I am trying to copy tensorrt to the device according to the guide.

Could you point me to the referred link?

We recommend to copy just the required samples and its corresponding data for testing as the target has limited space. Does that work for you?

11

Okay, thank you

13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.