StyleGan2 Error

I am getting an error when trying to run StyleGan2. I have Installed JetPack 4.3 and installed Tensorflow as per, Installing TensorFlow For Jetson Platform.

With minor change in install
sudo pip3 install https://developer.download.nvidia.com/compute/redist/jp/v43/tensorflow-gpu/tensorflow_gpu-1.15.0+nv20.1-cp36-cp36m-linux_aarch64.whl

After installing it was unable to find nvcc, so I used these instructions nvcc-is-not-installed.

Now when I run run_generator.py or run_training.py, it is able to open dynamic libraries, but it says “ARM64 does not support NUMA - returning NUMA node zero” and is giving this error.

Traceback (most recent call last):
File “run_generator.py”, line 168, in
main()
File “run_generator.py”, line 163, in main
dnnlib.submit_run(sc, func_name_map[subcmd], **kwargs)
File “/home/ryan/Desktop/stylegan2/dnnlib/submission/submit.py”, line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File “/home/ryan/Desktop/stylegan2/dnnlib/submission/internal/local.py”, line 22, in submit
return run_wrapper(submit_config)
File “/home/ryan/Desktop/stylegan2/dnnlib/submission/submit.py”, line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File “/home/ryan/Desktop/stylegan2/run_generator.py”, line 21, in generate_images
_G, _D, Gs = pretrained_networks.load_networks(network_pkl)
File “/home/ryan/Desktop/stylegan2/pretrained_networks.py”, line 76, in load_networks
G, D, Gs = pickle.load(stream, encoding=‘latin1’)
File “/home/ryan/Desktop/stylegan2/dnnlib/tflib/network.py”, line 297, in setstate
self._init_graph()
File “/home/ryan/Desktop/stylegan2/dnnlib/tflib/network.py”, line 154, in _init_graph
out_expr = self._build_func(*self.input_templates, **build_kwargs)
File “”, line 491, in G_synthesis_stylegan2
File “”, line 455, in layer
File “”, line 99, in modulated_conv2d_layer
File “”, line 68, in apply_bias_act
File “/home/ryan/Desktop/stylegan2/dnnlib/tflib/ops/fused_bias_act.py”, line 68, in fused_bias_act
return impl_dict[impl](x=x, b=b, axis=axis, act=act, alpha=alpha, gain=gain)
File “/home/ryan/Desktop/stylegan2/dnnlib/tflib/ops/fused_bias_act.py”, line 122, in _fused_bias_act_cuda
cuda_kernel = _get_plugin().fused_bias_act
File “/home/ryan/Desktop/stylegan2/dnnlib/tflib/ops/fused_bias_act.py”, line 16, in _get_plugin
return custom_ops.get_plugin(os.path.splitext(file)[0] + ‘.cu’)
File “/home/ryan/Desktop/stylegan2/dnnlib/tflib/custom_ops.py”, line 156, in get_plugin
plugin = tf.load_op_library(bin_file)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/load_library.py”, line 61, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/ryan/Desktop/stylegan2/dnnlib/tflib/_cudacache/fused_bias_act_106775088e12a945ba39b285aebe4e5e.so: undefined symbol: _ZN10tensorflow12OpDefBuilder5InputESs

Hi,

The NUMA log is a harmless warning.

Based on your log, this error might come from the different building configure between your app and our package.

tensorflow.python.framework.errors_impl.NotFoundError: /home/ryan/Desktop/stylegan2/dnnlib/tflib/_cudacache/fused_bias_act_106775088e12a945ba39b285aebe4e5e.so: undefined symbol: _ZN10tensorflow12OpDefBuilder5InputESs

Could you check which TensorFlow version is required for StyleGan2 first?
More, could you check if this issue helps?


Thanks.

Hi,
Thanks for the reply. The GitHub page for Stylegan2 says

TensorFlow 1.14 or 1.15 with GPU support

I had tensorflow-gpu 1.15 installed, I just tried it with tensorflow-gpu 1.14 from https://developer.download.nvidia.com/compute/redist/jp/v42/tensorflow-gpu/ - 2019-10-29. But it is still giving the same error.

Hi,

Have you tried this StyleGan2 on a desktop environment before?
If not, would you mind to give it a try.

It will be good if we can figure out this issue is specified for Jetson or not.
Thanks.

I have run stylegan2 on Google cloud compute VM.

Hi,

Based on the original log, it looks like a package version compatibility issue.

Would you mind to check the CUDA, cuDNN and TensorRT version of the cloud VM first.
Then, flash Nano with a JetPack that includes the similar software and test it again.

Thanks.

Thanks,

VM:
CUDA: release 10.0, V10.0.130
cuDNN: CUDNN_MAJOR 7 , CUDNN MINOR 6 , CUDNN_PATCHLEVEL 5

Nano:
CUDA: release 10.0, V10.0.326
cuDNN: CUDNN_MAJOR 7 , CUDNN MINOR 6 , CUDNN_PATCHLEVEL 3

Hi,

Seems that this compatibility issue is from different ABI strategy used in TensorFlow package and stylegan2 source code.
Would you mind to check this issue and give the suggestion a try?

Thanks.