StyleGan2 Error

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

I am getting an error when trying to run StyleGan2. I have Installed JetPack 4.3 and installed Tensorflow as per, Installing TensorFlow For Jetson Platform.

With minor change in install
sudo pip3 install https://developer.download.nvidia.com/compute/redist/jp/v43/tensorflow-gpu/tensorflow_gpu-1.15.0+nv20.1-cp36-cp36m-linux_aarch64.whl

After installing it was unable to find nvcc, so I used these instructions nvcc-is-not-installed.

Now when I run run_generator.py or run_training.py, it is able to open dynamic libraries, but it says “ARM64 does not support NUMA - returning NUMA node zero” and is giving this error.

Traceback (most recent call last):
File “run_generator.py”, line 168, in
main()
File “run_generator.py”, line 163, in main
dnnlib.submit_run(sc, func_name_map[subcmd], **kwargs)
File “/home/ryan/Desktop/stylegan2/dnnlib/submission/submit.py”, line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File “/home/ryan/Desktop/stylegan2/dnnlib/submission/internal/local.py”, line 22, in submit
return run_wrapper(submit_config)
File “/home/ryan/Desktop/stylegan2/dnnlib/submission/submit.py”, line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File “/home/ryan/Desktop/stylegan2/run_generator.py”, line 21, in generate_images
_G, _D, Gs = pretrained_networks.load_networks(network_pkl)
File “/home/ryan/Desktop/stylegan2/pretrained_networks.py”, line 76, in load_networks
G, D, Gs = pickle.load(stream, encoding=‘latin1’)
File “/home/ryan/Desktop/stylegan2/dnnlib/tflib/network.py”, line 297, in setstate
self._init_graph()
File “/home/ryan/Desktop/stylegan2/dnnlib/tflib/network.py”, line 154, in _init_graph
out_expr = self._build_func(*self.input_templates, **build_kwargs)
File “”, line 491, in G_synthesis_stylegan2
File “”, line 455, in layer
File “”, line 99, in modulated_conv2d_layer
File “”, line 68, in apply_bias_act
File “/home/ryan/Desktop/stylegan2/dnnlib/tflib/ops/fused_bias_act.py”, line 68, in fused_bias_act
return impl_dict[impl](x=x, b=b, axis=axis, act=act, alpha=alpha, gain=gain)
File “/home/ryan/Desktop/stylegan2/dnnlib/tflib/ops/fused_bias_act.py”, line 122, in _fused_bias_act_cuda
cuda_kernel = _get_plugin().fused_bias_act
File “/home/ryan/Desktop/stylegan2/dnnlib/tflib/ops/fused_bias_act.py”, line 16, in _get_plugin
return custom_ops.get_plugin(os.path.splitext(file)[0] + ‘.cu’)
File “/home/ryan/Desktop/stylegan2/dnnlib/tflib/custom_ops.py”, line 156, in get_plugin
plugin = tf.load_op_library(bin_file)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/load_library.py”, line 61, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/ryan/Desktop/stylegan2/dnnlib/tflib/_cudacache/fused_bias_act_106775088e12a945ba39b285aebe4e5e.so: undefined symbol: _ZN10tensorflow12OpDefBuilder5InputESs

3

Hi,

The NUMA log is a harmless warning.

Based on your log, this error might come from the different building configure between your app and our package.

tensorflow.python.framework.errors_impl.NotFoundError: /home/ryan/Desktop/stylegan2/dnnlib/tflib/_cudacache/fused_bias_act_106775088e12a945ba39b285aebe4e5e.so: undefined symbol: _ZN10tensorflow12OpDefBuilder5InputESs

Could you check which TensorFlow version is required for StyleGan2 first?
More, could you check if this issue helps?

Thanks.

4

Hi,
Thanks for the reply. The GitHub page for Stylegan2 says

TensorFlow 1.14 or 1.15 with GPU support

I had tensorflow-gpu 1.15 installed, I just tried it with tensorflow-gpu 1.14 from Index of /compute/redist/jp/v42/tensorflow-gpu - 2019-10-29. But it is still giving the same error.

5

Hi,

Have you tried this StyleGan2 on a desktop environment before?
If not, would you mind to give it a try.

It will be good if we can figure out this issue is specified for Jetson or not.
Thanks.

7

I have run stylegan2 on Google cloud compute VM.

8

Hi,

Based on the original log, it looks like a package version compatibility issue.

Would you mind to check the CUDA, cuDNN and TensorRT version of the cloud VM first.
Then, flash Nano with a JetPack that includes the similar software and test it again.

Thanks.

9

Thanks,

VM:
CUDA: release 10.0, V10.0.130
cuDNN: CUDNN_MAJOR 7 , CUDNN MINOR 6 , CUDNN_PATCHLEVEL 5

Nano:
CUDA: release 10.0, V10.0.326
cuDNN: CUDNN_MAJOR 7 , CUDNN MINOR 6 , CUDNN_PATCHLEVEL 3

10

Hi,

Seems that this compatibility issue is from different ABI strategy used in TensorFlow package and stylegan2 source code.
Would you mind to check this issue and give the suggestion a try?

https://github.com/CharlesShang/TFFRCNN/issues/108

Thanks.