Something (NUMP, PTX) Error running Tensorflow on JETSON XAVIER NX

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
,
1

Hi，
I have JETSON XAVIER NX,Jetpack 5.1.4, Tensorflow 2.12.1, CUDA 11.8
The problem I encountered is

2024-11-19 17:21:02.513625: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:00:00.0/numa_node
Your kernel may have been built without NUMA support.

Should I be worried ? Or is it something negligible ?

Another issue is

fleInTensor3Simple<T, 2, 1, 0>, config.block_count, config.thread_per_block, 0, d.stream(), config.virtual_thread_count, in.data(), combined_dims, out.data()) status: INTERNAL: the provided PTX was compiled with an unsupported toolchain.

Is it because the CUDA version is too high and does not match JETSON XAVIER NX？

4

Hi,

The NUMA message is a harmless warning.
The second message indicates your package might not be built with Xavier GPU architecture.

Could you try our prebuilt instead?

Thanks.

7

Thank you very much for your prompt reply.
I have already tried the method you provided. Unfortunately, the project I’m trying to replicate requires calling the TensorFlow C++ API. Could you tell me any other suggestions?
The issue is

CMake Error at /usr/local/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:233 (message):
  Could NOT find TensorFlow (missing: TENSORFLOW_INCLUDE_DIR
  TENSORFLOW_LIBRARY TENSORFLOW_FRAMEWORK_LIBRARY)
Call Stack (most recent call first):
  /usr/local/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:603 (_FPHSA_FAILURE_MESSAGE)
  cmake/FindTensorFlow.cmake:18 (find_package_handle_standard_args)
  CMakeLists.txt:92 (find_package)
8

Hi,

If you need the C++ API, please build it from the source.

Do you build the original package yourself?
If yes, maybe you can try it again to solve the toolchain compatibility issue.

Thanks.

9

Hi，

I installed Nvidia-Jetpack 5.1.4 (CUDA 11.4, CUDNN 8.6) and compiled the source code for TensorFlow 2.12.0 C API, but it was not successful.

The following are the related errors:

ERROR: /home/xavier/Documents/tensorflow-2.12.0/tensorflow/BUILD:1171:20: Linking tensorflow/libtensorflow.so.2.12.0 failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command 
  (cd /home/xavier/.cache/bazel/_bazel_xavier/8a7d5b24b66eb3a4ebbfe30b01ff67e5/execroot/org_tensorflow && \
  exec env - \
    CUDA_TOOLKIT_PATH=/usr/local/cuda-11.4 \
    GCC_HOST_COMPILER_PATH=/usr/bin/aarch64-linux-gnu-gcc-10 \
    LD_LIBRARY_PATH=/usr/local/cuda/lib64: \
    PATH=/home/xavier/.cache/bazelisk/downloads/sha256/ee21cb14fff5b493fef45919b7810fca25c5090eea5e7e177081f16cfb990f80/bin:/usr/local/cuda/bin:/home/xavier/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/usr/bin/python3 \
    PYTHON_LIB_PATH=/usr/lib/python3.8/dist-packages \
    TF2_BEHAVIOR=1 \
    TF_CUDA_COMPUTE_CAPABILITIES=3.5,7.0 \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc @bazel-out/aarch64-opt/bin/tensorflow/libtensorflow.so.2.12.0-2.params)
# Configuration: edb4252fa3721ed3b10274d001897d2782931231a0078d8ef8eff63c4b97caf1
# Execution platform: @local_execution_config_platform//:platform
/usr/bin/ld: bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/stream_executor/cuda/libcuda_graph.pic.a(cuda_graph.pic.o): in function `stream_executor::gpu::CaptureCudaGraph(stream_executor::Stream*, absl::lts_20220623::AnyInvocable<tsl::Status ()>, cudaStreamCaptureMode)':
cuda_graph.cc:(.text._ZN15stream_executor3gpu16CaptureCudaGraphEPNS_6StreamEN4absl12lts_2022062312AnyInvocableIFN3tsl6StatusEvEEE21cudaStreamCaptureMode+0x424): undefined reference to `cudaGraphDebugDotPrint'
collect2: error: ld returned 1 exit status

This error message indicates that the linker (ld) cannot find the definition of the CUDAGraphDebug DotPrint function when linking TensorFlow library files.

Could you give me some suggestions about version of TensorFlow?

10

Hi,

Please check:

Thanks.

11

Thanks.