Pytorch Error on the Jetson Platform

[sumin.lee]

We are testing JetPack 5.0 and Pytorch 1.11.0 and 1.12.0 both.
Both of Pytorch are downloaded Jetson Download Center.

And There is a issue with Pytorch_convolution_2d to use CPU
Issue Case: Ouput value is Nan, when “bias=None” desipite of nan is not included in input and weight.
General Case : Using 0 instead of false for bias and using Cuda has no issue.

[AastaLLL]

Hi,

Could you share the model and the reproducible script with us?
Thanks.

[song.bohyun]

Hi, AastaLLL
The following code executes a single 2d convolution operation.
I have tried with various inputs and weights (using real image data, not random numbers) but still, it returns NaN values.
Please let me know if you need any further clarification.
Thank you

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

with torch.no_grad():
input=torch.randn([1,3,256,256]).float().cpu()
weight=torch.randn([3,3,3,3]).float().cpu()
out=F.conv2d(input,weight,bias=None,stride=1,padding=1)
print(‘number_of_nans:’, torch.sum(torch.isnan(out)))

[AastaLLL]

Hi,

Confirmed that we can reproduce this issue with the l4t-pytorch:r34.1.1-pth1.12-py3 container.
Would you mind sharing the working version so we can compare?

Thanks.

[song.bohyun]

Hi
Currently, Pytorch v1.11.0, on top of Jetpack 5.0 is installed.
FYI, the Jetpack 5.0 was installed via nvidia sdk manager, and the pytorch is installed from the https://nvidia.box.com/shared/static/ssf2v7pf5i245fk4i0q926hy4imzs2ph.whl.

Thank you

[AastaLLL]

Hi,

At the top of the topic, you have mentioned that:

General Case : Using 0 instead of false for bias and using Cuda has no issue.

We can reproduce the NAN output with the container mentioned on Jul 28.
Would you mind sharing the script that can output the correct result so we can compare?

More, which device do you use?
JetPack 5 doesn’t support TX2.

Thanks.