I’m checking with internal team for the plan, will update to you once clarified. Please stay tuned.
Thanks!
Is there an update on this? Why is this particular arbitrary PyTorch SHA the one that was released – is there a specific change required for the new base system?
It’s difficult to assess this image’s functionality when the packages released for it are unexplained and not even released software.
Hi @WasabiFan, the team within NVIDIA responsible for building deep learning frameworks has started building PyTorch wheels for Jetson (as opposed to me personally doing them), and they build from top-of-tree to get the latest patches/fixes/ect in the build. We are currently planning with them to resume the versioned builds.
In the meantime, you should still be able to build PyTorch 1.11 in a similar fashion to what I have previously done in this post.
Thanks for the clarification!
It’d be helpful if the documentation explained why NVIDIA was releasing these particular commits (even if the answer is “we pulled master in February and that’s what we got”) since otherwise it seems like there was some particular feature we might need.
I have a fleet of Jetsons which currently run a custom build of PyTorch 1.10 on JetPack 4.6. The key thing I’d like to know here is whether there’s something specific that was needed from newer PyTorch for JetPack 5, or whether I can continue staying pinned to 1.10 without issue. As far as I can tell, the right CUDA versions are supported, and from what you say it sounds like there wasn’t anything specific about 1.12. So if that’s the case I’ll go ahead with building a PyTorch 1.10 wheel.
My situation is the opposite - I’m building Docker images on L4T-base and I want the latest stable binaries that are built by someone else whenever possible. My current Docker build only uses cusignal / cupy and torchaudio built from source, and that takes about 2.3 hours on a Xavier NX. The last time I built a PyTorch wheel I think it was 3.5 hours on the AGX Xavier Dev kit. I’d take the image size hit and switch to the L4T-ML base before I’ll add hours to my build process.
AFAIK that was the commit at the time they did the build for JetPack 5.0 DP, and it wasn’t because of a specific reason.
1 Like
Hi @znmeb, you could also use l4t-pytorch container which I have built using this PyTorch 1.12 wheel for JetPack 5.0. It includes torchaudio. The dockerfiles are found here:
Yes - I’m switching to the PyTorch image for a base, I was holding off because of its size but my image is also huge. Not mounting host files into the container on JetPack 5.0DP made a huge difference.
Yes, the JetPack 5.0 containers are bigger (because they now have CUDA/cuDNN/ect installed inside them as opposed to mounted from the device), but on the plus side these containers shouldn’t need rebuilt to run then on newer versions of JetPack (>5.0) when those come out.
Thanks! torchaudio 0.11.0 thanks you too! :-)
Hello dusty_nv,
I installed pytroch successfully, but pytorch with GPU doesn’t work. When I traced back found that CUDA is not installed. I verified it in two ways:
- By navigating to jtop and checked the INFO and I see CUDA is not installed.
- Also I used nvcc -V to verify but it says CUDA version 11.4 I am guessing it is due to the fact that it is referred to the directory of CUDA installation on bashrc script. Not sure what is going wrong.
I did compare them against 4.6 version and the only difference I see is, it is not being detected on jtop info section.
It would be great if you or anyone can assist me on it.
Thank you so much for your time.
Hmm…are you able to run the deviceQuery sample okay under /usr/local/cuda/samples/1_Utilities/deviceQuery ?
1 Like
Hello dusty_nv,
Thank you for getting back to me.
Yes I compiled but not successful.
The error message is:
deviceQuery.cpp:16:10: fatal error: cuda_runtime.h: No such file or directory
made sure the cuda file location is correct as well. I also attach jtop output for reference
Thank you
Hmm, what does the following show for you?
$ ls /usr/local/cuda/include
builtin_types.h cupti_result.h nppi_linear_transforms.h
channel_descriptor.h cupti_runtime_cbid.h nppi_morphological_operations.h
common_functions.h cupti_target.h nppi_statistics_functions.h
cooperative_groups cupti_version.h nppi_support_functions.h
cooperative_groups.h curand_discrete2.h nppi_threshold_and_compare_operations.h
crt curand_discrete.h npps_arithmetic_and_logical_operations.h
cub curand_globals.h npps_conversion_functions.h
cublas_api.h curand.h npps_filtering_functions.h
cublas.h curand_kernel.h npps.h
cublasLt.h curand_lognormal.h npps_initialization.h
cublas_v2.h curand_mrg32k3a.h npps_statistics_functions.h
cublasXt.h curand_mtgp32dc_p_11213.h npps_support_functions.h
cuComplex.h curand_mtgp32.h nv
cuda curand_mtgp32_host.h nvblas.h
cuda_awbarrier.h curand_mtgp32_kernel.h nv_decode.h
cuda_awbarrier_helpers.h curand_normal.h nvfunctional
cuda_awbarrier_primitives.h curand_normal_static.h nvml.h
cuda_bf16.h curand_philox4x32_x.h nvperf_common.h
cuda_bf16.hpp curand_poisson.h nvperf_cuda_host.h
cuda_device_runtime_api.h curand_precalc.h nvperf_host.h
cudaEGL.h curand_uniform.h nvperf_target.h
cuda_egl_interop.h cusolver_common.h nvPTXCompiler.h
cudaEGLTypedefs.h cusolverDn.h nvrtc.h
cuda_fp16.h cusolverRf.h nvToolsExtCuda.h
cuda_fp16.hpp cusolverSp.h nvToolsExtCudaRt.h
cudaGL.h cusolverSp_LOWLEVEL_PREVIEW.h nvToolsExt.h
cuda_gl_interop.h cusparse.h nvToolsExtOpenCL.h
cudaGLTypedefs.h cusparse_v2.h nvToolsExtSync.h
cuda.h device_atomic_functions.h nvtx3
cudalibxt.h device_atomic_functions.hpp sm_20_atomic_functions.h
cuda_occupancy.h device_double_functions.h sm_20_atomic_functions.hpp
cuda_pipeline.h device_functions.h sm_20_intrinsics.h
cuda_pipeline_helpers.h device_launch_parameters.h sm_20_intrinsics.hpp
cuda_pipeline_primitives.h device_types.h sm_30_intrinsics.h
cuda_profiler_api.h driver_functions.h sm_30_intrinsics.hpp
cudaProfiler.h driver_types.h sm_32_atomic_functions.h
cudaProfilerTypedefs.h fatbinary_section.h sm_32_atomic_functions.hpp
cudart_platform.h generated_cuda_gl_interop_meta.h sm_32_intrinsics.h
cuda_runtime_api.h generated_cudaGL_meta.h sm_32_intrinsics.hpp
cuda_runtime.h generated_cuda_meta.h sm_35_atomic_functions.h
cuda_stdint.h generated_cuda_runtime_api_meta.h sm_35_intrinsics.h
cuda_surface_types.h generated_cuda_vdpau_interop_meta.h sm_60_atomic_functions.h
cuda_texture_types.h generated_cudaVDPAU_meta.h sm_60_atomic_functions.hpp
cudaTypedefs.h generated_nvtx_meta.h sm_61_intrinsics.h
cudaVDPAU.h host_config.h sm_61_intrinsics.hpp
cuda_vdpau_interop.h host_defines.h surface_functions.h
cudaVDPAUTypedefs.h library_types.h surface_functions.hpp
cudla.h math_constants.h surface_indirect_functions.h
cufft.h math_functions.h surface_indirect_functions.hpp
cufftw.h mma.h surface_types.h
cufftXt.h nppcore.h texture_fetch_functions.h
cupti_activity.h nppdefs.h texture_fetch_functions.hpp
cupti_callbacks.h npp.h texture_indirect_functions.h
cupti_driver_cbid.h nppi_arithmetic_and_logical_operations.h texture_indirect_functions.hpp
cupti_events.h nppi_color_conversion.h texture_types.h
cupti.h nppi_data_exchange_and_initialization.h thrust
cupti_metrics.h nppi_filtering_functions.h vector_functions.h
cupti_nvtx_cbid.h nppi_geometry_transforms.h vector_functions.hpp
cupti_profiler_target.h nppi.h vector_types.h
1 Like
builtin_types.h device_double_functions.h
channel_descriptor.h device_functions.h
common_functions.h device_launch_parameters.h
cooperative_groups device_types.h
cooperative_groups.h driver_functions.h
crt driver_types.h
cub fatbinary_section.h
cublas_api.h generated_cuda_gl_interop_meta.h
cublas.h generated_cudaGL_meta.h
cublasLt.h generated_cuda_meta.h
cublas_v2.h generated_cuda_runtime_api_meta.h
cublasXt.h generated_cuda_vdpau_interop_meta.h
cuComplex.h generated_cudaVDPAU_meta.h
cuda generated_nvtx_meta.h
cuda_awbarrier.h host_config.h
cuda_awbarrier_helpers.h host_defines.h
cuda_awbarrier_primitives.h library_types.h
cuda_bf16.h math_constants.h
cuda_bf16.hpp math_functions.h
cuda_device_runtime_api.h mma.h
cudaEGL.h nppcore.h
cuda_egl_interop.h nppdefs.h
cudaEGLTypedefs.h npp.h
cuda_fp16.h nppi_arithmetic_and_logical_operations.h
cuda_fp16.hpp nppi_color_conversion.h
cudaGL.h nppi_data_exchange_and_initialization.h
cuda_gl_interop.h nppi_filtering_functions.h
cudaGLTypedefs.h nppi_geometry_transforms.h
cuda.h nppi.h
cudalibxt.h nppi_linear_transforms.h
cuda_occupancy.h nppi_morphological_operations.h
cuda_pipeline.h nppi_statistics_functions.h
cuda_pipeline_helpers.h nppi_support_functions.h
cuda_pipeline_primitives.h nppi_threshold_and_compare_operations.h
cuda_profiler_api.h npps_arithmetic_and_logical_operations.h
cudaProfiler.h npps_conversion_functions.h
cudaProfilerTypedefs.h npps_filtering_functions.h
cudart_platform.h npps.h
cuda_runtime_api.h npps_initialization.h
cuda_runtime.h npps_statistics_functions.h
cuda_stdint.h npps_support_functions.h
cuda_surface_types.h nv
cuda_texture_types.h nvblas.h
cudaTypedefs.h nv_decode.h
cudaVDPAU.h nvfunctional
cuda_vdpau_interop.h nvml.h
cudaVDPAUTypedefs.h nvperf_common.h
cudla.h nvperf_cuda_host.h
cufft.h nvperf_host.h
cufftw.h nvperf_target.h
cufftXt.h nvPTXCompiler.h
cupti_activity.h nvrtc.h
cupti_callbacks.h nvToolsExtCuda.h
cupti_driver_cbid.h nvToolsExtCudaRt.h
cupti_events.h nvToolsExt.h
cupti.h nvToolsExtOpenCL.h
cupti_metrics.h nvToolsExtSync.h
cupti_nvtx_cbid.h nvtx3
cupti_profiler_target.h sm_20_atomic_functions.h
cupti_result.h sm_20_atomic_functions.hpp
cupti_runtime_cbid.h sm_20_intrinsics.h
cupti_target.h sm_20_intrinsics.hpp
cupti_version.h sm_30_intrinsics.h
curand_discrete2.h sm_30_intrinsics.hpp
curand_discrete.h sm_32_atomic_functions.h
curand_globals.h sm_32_atomic_functions.hpp
curand.h sm_32_intrinsics.h
curand_kernel.h sm_32_intrinsics.hpp
curand_lognormal.h sm_35_atomic_functions.h
curand_mrg32k3a.h sm_35_intrinsics.h
curand_mtgp32dc_p_11213.h sm_60_atomic_functions.h
curand_mtgp32.h sm_60_atomic_functions.hpp
curand_mtgp32_host.h sm_61_intrinsics.h
curand_mtgp32_kernel.h sm_61_intrinsics.hpp
curand_normal.h surface_functions.h
curand_normal_static.h surface_functions.hpp
curand_philox4x32_x.h surface_indirect_functions.h
curand_poisson.h surface_indirect_functions.hpp
curand_precalc.h surface_types.h
curand_uniform.h texture_fetch_functions.h
cusolver_common.h texture_fetch_functions.hpp
cusolverDn.h texture_indirect_functions.h
cusolverRf.h texture_indirect_functions.hpp
cusolverSp.h texture_types.h
cusolverSp_LOWLEVEL_PREVIEW.h thrust
cusparse.h vector_functions.h
cusparse_v2.h vector_functions.hpp
device_atomic_functions.h vector_types.h
device_atomic_functions.hpp
I guess its the same. Thank you once again!
Hmm, to be honest I’m not sure in that case why it wouldn’t build.
I don’t believe jtop has been updated for JetPack 5.0 yet, so that may or may not be accurately indicating some issue with CUDA.
If the issues persist, you may want to re-flash your device or try reinstalling CUDA toolkit with sudo apt-get install cuda-toolkit-11-4 (you may want/need to remove it with apt first)
1 Like
Thank you dusty_nv.
The problem was torch.cuda.is_available() gave False as output. Now after reinstalling CUDA it is now working ( built as .deb file).
Yes, JTOP is not updated for JP 5.0 yet.
Once again, thank you for all your assistance
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.