PyTorch for Jetson

Does the same thing happen if you just import numpy?

If so, try setting ENV OPENBLAS_CORETYPE=ARMV8 in your Dockerfile
(Illegal instruction (core dumped) on import for numpy 1.19.5 on ARM64 · Issue #18131 · numpy/numpy · GitHub)