Only 525Mb left of disk space after flashing a 256Gb SD card with the SDK manager. What's going on?

Hi,

I’ve attempted this twice now with the same results both times.

I have a 256Gb SD card, on which I wrote Jetpack 4.4 via balenaEtcher (windows 10)

I didn’t setup the Xavier NX at that point, I only booted it and connected it via USB to my Ubuntu 20 host computer so that the SDK manager could detect the jetson.

After the SDK manager flashed my jetson, I logged in on the jetson (for the 1st time) and got the message “This computer has only 525.8Mb disk space remaining”.

No way it used 250Gb+ of space during flashing.

Anybody knows what’s going on and how to properly fix the partitions on windows (my ubuntu laptop doesn’t have an SD card reader, too old)

This is what the SD card looks like: https://i.imgur.com/xrAeIHf.png

Thanks.

probably it needs to be resized;
df -h
or lsblk will show the situation
also sudo gparted will do

1 Like

Hi,

gparted (ubuntu 20 laptop) shows that the boot partition (the 1st one on the left) has 414Gb unused.

Jetson Xavier tells me I’m out of space with just 300Mb left.

I still don’t understand what’s going on.

Gparted Screenshot: Imgur: The magic of the Internet

The SD card is detected as being 15Gb by Ubuntu when mounted. None of those partition are 15Gb.

could you run in a similar manner gparted at Jetson?
does the 414gb pertain to Host PC disc drive or to the mounted sdcard with Jetson OS?

It looks very different on the jetson compared to the Ubuntu Laptop: Imgur: The magic of the Internet

There’s a bunch of “msft” flags with unknown filesystem.

I flashed the OS on the card via windows 10 (didn’t have the ubuntu laptop at the time), it must be the source of the issue.
I’ll re-download & reflash the jetpack from Ubuntu, hopefully it’ll help.

let us know if it worked for you.
Otherwise there will be solutions either to manually move to the end of the disk every of the little partition so that it will be possible to extend the system partition in a way it will fill the rest of the space.
Otherwise. a script could be used for the purpose. There is an example script that could be used for moving all partitions to the end:

#!/bin/sh
set -ex
GPT_SIZE=40
UDA_NEW_SIZE=32
move_part() {
  name=$(sgdisk -i $1 /dev/mmcblk0 | grep "Partition name" | cut -d"'" -f2)
  typecode=$(sgdisk -i $1 /dev/mmcblk0 | grep "Partition GUID code:" | cut -d' ' -f4)
  guid=$(sgdisk -i $1 /dev/mmcblk0 | grep "Partition unique GUID:" | cut -d' ' -f4)
  sgdisk -d $1 -n $1:$2:$3 -c $1:"$name" -t $1:"$typecode" -u $1:"$guid" /dev/mmcblk0
  partprobe /dev/mmcblk0
}
read DISK_SIZE </sys/block/mmcblk0/size
START=$((DISK_SIZE-GPT_SIZE-UDA_NEW_SIZE))
move_part 11 $START $((START+UDA_NEW_SIZE-1))
for i in $(seq 10 -1 2); do
  dd if=/dev/mmcblk0p$i of=part$i.img
  read size </sys/block/mmcblk0/mmcblk0p$i/size
  START=$((START-size))
  move_part $i $START $((START+size-1))
  dd of=/dev/mmcblk0p$i if=part$i.img
  rm -f part$i.img
done
move_part 1 $GPT_SIZE 0
sgdisk --move-second-header /dev/mmcblk0
resize2fs /dev/mmcblk0p1

in the example above the size could be adjusted from 32gb to address the required size of the resulting partition. By default script resizes 14gb to 32gbm but it also moves all intermediatory partitions to the end. So that it will be possible to use gparted to extend/ resize partition with it or with resizefs.
reference threads:
source of the sript above:

Not sure, but if you have a

and gparted shows first partition:

You may have a partition table issue.
You may restart from scratch (this would loose any data on disk) making partition table with gdisk on Linux rather than with fdisk or else.
I’m not familiar with Etcher, but if there’s an option for GPT partition format, you may try to check it.

FYI, the binary partitions are just that…binary data. There is no filesystem for this. Think of this as equivalent to the CMOS BIOS/UEFI of a PC (and you certainly cannot mount nor examine the BIOS content…even if you had access, this is binary data).

The APP partition has a filesystem, but Windows does not understand ext4. If this were Fuse or VFAT, then Windows would understand it.

Updated: Fixed!

I redid the whole op from Ubuntu 18, and it works great.

My mistake was using 3 different OS in the process.

I started by formatting & writing the Jetpack on the SD card from Windows 10 (didn’t have a ubuntu laptop yet). That is most likely the step that messed up everything.

I then flashed the SD with the SDK manager from Ubuntu 18, then tried to fix the SD card partition from Ubuntu 20 (upgraded the laptop).

Seems that at one point I also mixed up the SD card with my laptop’s hard drive in Gparted. They’re both named “/SDD*”, got me confused.

So I reinstalled Ubuntu 18 on the laptop, did everythign from there: SD card formatting, writing the jetpack to the card, fixing the partition size with Gparted (super easy at that point), & finally flashing with the SDK manager.

Without the mess that Windows did on my SD card partitions, resizing it was really easy and obvious.

Thanks for the help!

Thank you for the update!

Actually I was wrong. Same issue.

Pretty sure it was fixed before flashing the SD again with the SDK, it detected the card as 200+Gb. After flashing, it’s back to being detected as a 15Gb card, and no way to move or resize anything.

The partition is all the way to the left, followed by a bunch of small unrecognized partitions (Gparted allows no actions on them), followed by a massive empty partition.

I tried on both Ubuntu laptop & Jetson.

I’m giving up on the 256Gb card, trying with a 32Gb. This is way too hard…

If you copied a 16Gb disk image, it may be expected. There are many partitions for various features of boot flow.
You may try to resize the APP (partition 1) from your Ubuntu host with gparted. You would run a filesystem check of this partition after resizing and it should be ok. Only then plug the SD card into your jetson.

Here is my 64GB SDCard partitions layout (seen from jetson, so mmcblk0, name may be different in your host depending on how it is connected):

I tried again on a 32Gb card. Same issue.
I took a screenshot of the SD card before SDK manager and after SDK manager:

Card with Jetpack written to it, before SDK manager:

I have plenty of space, looks great.

Same card after the SDK manager:

It flashed the OS again, on the left, where I can’t move it or resize it. I barely have any space left.

@Honey_Patouceul, how did you manage to have the SDK not write to the left of the SD card, where it can’t be moved?

I must be doing something wrong but I can’t figure out what.

When you flash with SDK Manager, it will create the APP partition as 16GB. That’s why @Honey_Patouceul mentioned manually re-sizing it (but I guess if your PC doesn’t have an SD card slot, you can’t do that). Note that with the pre-made SD card image, it will automatically resize (but again, you need an SD card slot on your PC for that).

What I would recommend is flashing L4T manually, and specifying the -S option to flash.sh (i.e. -S 29GiB), and then it would use more space on your SD card. Then you can go back through SDK Manager, and have it install the CUDA components (remember to de-select flashing L4T in SDK Manager though).

Here are the components for manually flashing L4T:

https://docs.nvidia.com/jetson/l4t/#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Frootfs_custom.html%23

https://docs.nvidia.com/jetson/l4t/#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fflashing.html%23wwpID0E0MJ0HA

Well, I’m a bit confused about this topic.
Since you’re trying to flash a SD card, I assume you have the Developer kit.
So you would just download the SD Card image (need to be logged in).

Then follow Instructions for Linux on host from this guide. I used Command line method.

When written, insert into Jetson, it should auto resize on first boot. Nothing more to do than updating with apt.

I tried with just the Jetpack first without using the Nvidia SDK, but that doesn’t come with CUDA & all of that pre-installed. Inference is super slow, even the ONNX samples, deepstream, …
I tried to install CUDA and all dependencies manually, spent a couple weeks on it and went nowhere, just made a mess of my system.

So I setup ubuntu on an old laptop, bought a SD card reader (via USB) so that I could use the Nvidia SDK.

I follow their instructions:

  • Write the jetpack first (I tried with a formatted SD card, it didn’t detect the Jetson so seems writing the jetpack first is required)
  • It downloads the files
  • I set my board in recovery mode with a jumper as instructed
  • It start installing, ask me to finish the OS setup on the Jetson, so I do
  • I enter my user/pass, it finish the installation

I tried resizing the partition before and after the SDK manager. Before it works, and I have a jetson with a lot of space, using the whole SDK card, but slow inference. After the SDK, a new partition is created all the way to the left, where it can’t be moved or resized. I have a jetson with fast inference but no space to actually do anything.

In that guide (Getting Started With Jetson Xavier NX Developer Kit | NVIDIA Developer) it shows the card being 32gb right out of etcher, but I get 15Gb every time. I need to use Gparted to resize and use the full card.
I used Etcher to write the Jetpack, but didn’t try the last steps " Command Line Instructions" (haven’t seen them before), I’ll try them.
That guide doesn’t talk about the SDK manager, which seems to be the issue, so not sure it’ll change anything.

If you have an SD card interface for your PC, then plug your SD card into your PC and use Etcher to flash the pre-built NX image to it:

https://developer.nvidia.com/jetson-nx-developer-kit-sd-card-image

Then put the SD card back into your Jetson. After the first boot of your Jetson, it will be resized to use the full space of your SD card. It already includes the CUDA components, ect. You don’t need to use SDK Manager at all to flash this pre-built image.

That’s what I did initially but inferences was super slow on everything.

I was getting 0.3fps on yolo-tiny via ONNX running on an mp4 videa. None of the python samples I tried ran any faster than what I can run on a Raspberry pi. I figured that to properly use Cuda I had to use the SDK manager.

I also tried to build OpenCV 4 via 2 distinct build scripts made for building on the Xavier + one attempt manually, CMAKE step can never find CUDA even when specifying the CUDA path. Some forum entry said to use the SDK manager to resolve.

I’m a bit confused why the Xavier is so slow on everything if the SDK manager isn’t required.

If you flashed our pre-built SD card image, then it should have already had CUDA installed. You could also confirm this by checking that you had /usr/local/cuda

After you flash the image again, what I would do to check that CUDA is installed correctly this time, is to compile/run the CUDA deviceQuery sample:

$ /usr/local/cuda/samples/1_Utilities/deviceQuery
$ sudo make
$ ./deviceQuery
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "Xavier"
  CUDA Driver Version / Runtime Version          10.2 / 10.2
  CUDA Capability Major/Minor version number:    7.2
  Total amount of global memory:                 7772 MBytes (8149061632 bytes)
  ( 6) Multiprocessors, ( 64) CUDA Cores/MP:     384 CUDA Cores
  GPU Max Clock rate:                            1109 MHz (1.11 GHz)
  Memory Clock rate:                             1109 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 524288 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            Yes
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 0 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.2, CUDA Runtime Version = 10.2, NumDevs = 1
Result = PASS

Also, you may want to change the power mode to MODE_15W_6CORE (mode 2). I think the default is MODE_10W_2CORE (mode 3).

$ sudo nvpmodel -m 2
$ sudo nvpmodel -q

NV Fan Mode:quiet
NV Power Mode: MODE_15W_6CORE
2

Hi,

This works, I see pretty much the same as your output.

So why is Tensorrt so slow for example?

I used the sample located at /usr/src/tensorrt/samples/python/yolov3_onnx and I’m getting about 1fps, even with power mode at MODE_15W_6CORE.

Inference time is between 1789ms & 2300+ms per frame. Around the same inference time in the default example that looks up a single image.

I ran the original sample then converted the code to run on a video: TensorRT YoloV3 running on a video via OpenCV · GitHub

It’s so slow, there’s no way it’s using the GPU to run. What am I missing?