Building TensorFlow v1.14 in TensorRT Docker v19.08 fails

I’ve got the following error when building TensorFlow v1.14 in TensorRT Docker v19.08 (nvcr.io/nvidia/tensorrt:19.08-py3):

ERROR: /home/dvdt/.cache/bazel/_bazel_dvdt/066ffad4cbebc232272c68dae0fb6ec0/external/local_config_cuda/cuda/BUILD:1312:1: Executing genrule @local_config_cuda//cuda:cuda-bin failed (Exit 1)
cp: cannot stat '/usr/local/cuda-10.1/bin/./nv-nsight-cu-cli': No such file or directory
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 5229.870s, Critical Path: 470.19s
INFO: 7879 processes: 7879 local.
FAILED: Build did NOT complete successfully

As I understand, /usr/local/cuda-10.1/bin/nv-nsight-cu-cli should be a symbolic link to /usr/local/cuda-10.1/NsightCompute-/nv-nsight-cu-cli. However, this NsightCompute directory is corrupted in TensorRT Docker v19.08:

  • in TensorRT Docker v19.07:
$ nvidia-docker run -it --rm nvcr.io/nvidia/tensorrt:19.07-py3 bash
...
NVIDIA Release 19.07 (build 7106125)
...
root@a589f0bde0ea:/workspace# ls -la /usr/local/cuda/NsightCompute*
total 72
drwxr-xr-x  6 root root  4096 Jun 28 23:02 .
drwxr-xr-x  1 root root  4096 Jun 28 23:03 ..
-rw-r--r--  1 root root 33034 May  6 23:24 EULA.txt
drwxr-xr-x 10 root root  4096 Jun 28 23:02 docs
drwxr-xr-x  4 root root  4096 Jun 28 23:02 extras
-rwxr-xr-x  1 root root   127 May  6 23:24 nv-nsight-cu-cli
-rwxr-xr-x  1 root root   127 Jun 28 23:02 nv-nsight-cu-cli.orig
-rw-r--r--  1 root root   137 Jun 28 23:02 nv-nsight-cu-cli.rej
drwxr-xr-x  2 root root  4096 Jun 28 23:02 sections
drwxr-xr-x  4 root root  4096 Jun 28 23:02 target
  • in TensorRT Docker v19.08:
$ nvidia-docker run -it --rm nvcr.io/nvidia/tensorrt:19.08-py3 bash
...
NVIDIA Release 19.08 (build 7589526)
...
root@293891517f3a:/workspace# ls -la /usr/local/cuda/NsightCompute*
total 8
drwxr-xr-x 2 root root 4096 Aug 10 00:50 .
drwxr-xr-x 1 root root 4096 Aug 10 00:51 ..
-rw-r--r-- 1 root root    0 Aug 10 00:50 nv-nsight-cu-cli.orig

Any updates?