Nvidia-fs install

[ijannick]

Hi,

i want to use nvidia-fs. in the “Install an Troubleshoot” Guide it says to disable iommu - what is the correct option for agxOrin arm64 ?

kernel params mentiones ARM64

    iommu.forcedac= [ARM64, X86] Control IOVA allocation for PCI devices.

                    Format: { "0" | "1" }
                    0 - Try to allocate a 32-bit DMA address first, before
                      falling back to the full range if needed.
                    1 - Allocate directly from the full usable range,
                      forcing Dual Address Cycle for PCI cards supporting
                      greater than 32-bit addressing.

    iommu.strict=   [ARM64, X86] Configure TLB invalidation behaviour
                    Format: { "0" | "1" }
                    0 - Lazy mode.
                      Request that DMA unmap operations use deferred
                      invalidation of hardware TLBs, for increased
                      throughput at the cost of reduced device isolation.
                      Will fall back to strict mode if not supported by
                      the relevant IOMMU driver.
                    1 - Strict mode.
                      DMA unmap operations invalidate IOMMU hardware TLBs
                      synchronously.
                    unset - Use value of CONFIG_IOMMU_DEFAULT_DMA_{LAZY,STRICT}.
                    Note: on x86, strict mode specified via one of the
                    legacy driver-specific options takes precedence.

    iommu.passthrough=
                    [ARM64, X86] Configure DMA to bypass the IOMMU by default.
                    Format: { "0" | "1" }
                    0 - Use IOMMU translation for DMA.
                    1 - Bypass the IOMMU for DMA.
                    unset - Use value of CONFIG_IOMMU_DEFAULT_PASSTHROUGH.

thanks & cheers
nj

[kayccc]

@ijannick

nvidia-fs

Not sure if it’s for x86 only, may I know where you get it or from which link?

[ijannick]

Hi,

I installed r35.3.1 as per ‘normal’ install without tweaking the image, just burned it onto agxOrin.
Then I followed simpleCudaUpgrade to catch the 12.2 cuda. And there it says in
cudaInstallOverview Note, i have to install it / not included in cuda install…

Went to GPUDirectStroage and got stuck at GRUBCmdlineFixIOMMU…

So my approch would then be:

iommu ?? ignore or set GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 iommu=offas it is possible according to kernel params

install mlnx_ofed for nvme access

install nvidia-fs

okay that way ?

Cheers & Thanks
nj

[AastaLLL]

HI,

GPUDirectStorage doesn’t support Jetson.

Do you get CUDA 12 work?
If the failure only happen when you try to use GPUDirectStorage?
Thanks.