StyleGAN2 Not Working on CentOS?

I installed the latest nvidia cudatoolkit on the system level (I also tried cudatoolkits on the conda level but it didn’t change anything). I clone the githube repo GitHub - NVlabs/stylegan2: StyleGAN2 - Official TensorFlow Implementation . Then I created a conda environment with python=3.6 tensorflow-gpu=1.15. Then I installed the latest pillow and requests packages for python via conda since those modules were needed.

I run the basic nvcc test provided in the repo.

(stylegan2) [root@craig001 styletest]# nvcc test_nvcc.cu -o test_nvcc -run
CPU says hello.
GPU says hello.

Alright we’re off to the races! Let’s try their image generation example!

(stylegan2) [root@craig001 styletest]# python run_generator.py generate-images --network=gdrive:networks/stylegan2-ffhq-config-f.pkl --seeds=6600-6625 --truncation-psi=0.5
Local submit - run_dir: results/00001-generate-images
dnnlib: Running run_generator.generate_images() on localhost…
Loading networks from “gdrive:networks/stylegan2-ffhq-config-f.pkl”…
Setting up TensorFlow plugin “fused_bias_act.cu”: Preprocessing… Failed!
Traceback (most recent call last):
File “run_generator.py”, line 168, in
main()
File “run_generator.py”, line 163, in main
dnnlib.submit_run(sc, func_name_map[subcmd], **kwargs)
File “/root/styletest/dnnlib/submission/submit.py”, line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File “/root/styletest/dnnlib/submission/internal/local.py”, line 22, in submit
return run_wrapper(submit_config)
File “/root/styletest/dnnlib/submission/submit.py”, line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File “/root/styletest/run_generator.py”, line 21, in generate_images
_G, _D, Gs = pretrained_networks.load_networks(network_pkl)
File “/root/styletest/pretrained_networks.py”, line 76, in load_networks
G, D, Gs = pickle.load(stream, encoding=‘latin1’)
File “/root/styletest/dnnlib/tflib/network.py”, line 297, in setstate
self._init_graph()
File “/root/styletest/dnnlib/tflib/network.py”, line 154, in _init_graph
out_expr = self._build_func(*self.input_templates, **build_kwargs)
File “”, line 491, in G_synthesis_stylegan2
File “”, line 455, in layer
File “”, line 99, in modulated_conv2d_layer
File “”, line 68, in apply_bias_act
File “/root/styletest/dnnlib/tflib/ops/fused_bias_act.py”, line 68, in fused_bias_act
return impl_dict[impl](x=x, b=b, axis=axis, act=act, alpha=alpha, gain=gain)
File “/root/styletest/dnnlib/tflib/ops/fused_bias_act.py”, line 122, in _fused_bias_act_cuda
cuda_kernel = _get_plugin().fused_bias_act
File “/root/styletest/dnnlib/tflib/ops/fused_bias_act.py”, line 16, in _get_plugin
return custom_ops.get_plugin(os.path.splitext(file)[0] + ‘.cu’)
File “/root/styletest/dnnlib/tflib/custom_ops.py”, line 111, in get_plugin
_run_cmd(_prepare_nvcc_cli(‘“%s” --preprocess -o “%s” --keep --keep-dir “%s”’ % (cuda_file, tmp_file, tmp_dir)))
File “/root/styletest/dnnlib/tflib/custom_ops.py”, line 61, in _run_cmd
raise RuntimeError(‘NVCC returned an error. See below for full command line and output log:\n\n%s\n\n%s’ % (cmd, output))
RuntimeError: NVCC returned an error. See below for full command line and output log:

nvcc “/root/styletest/dnnlib/tflib/ops/fused_bias_act.cu” --preprocess -o “/tmp/tmpjezrq8iu/fused_bias_act_tmp.cu” --keep --keep-dir “/tmp/tmpjezrq8iu” --disable-warnings --include-path “/root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/include” --include-path “/root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/include/external/protobuf_archive/src” --include-path “/root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/include/external/com_google_absl” --include-path “/root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/include/external/eigen_archive” 2>&1

In file included from /usr/include/c++/4.8.2/unordered_map:35:0,
from /root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/include/tensorflow/core/framework/op.h:20,
from /root/styletest/dnnlib/tflib/ops/fused_bias_act.cu:9:
/usr/include/c++/4.8.2/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
#error This file requires compiler and library support for the
^
In file included from /root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/include/absl/base/config.h:66:0,
from /root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/include/absl/strings/string_view.h:31,
from /root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/include/tensorflow/core/platform/stringpiece.h:29,
from /root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/include/tensorflow/core/lib/core/stringpiece.h:29,
from /root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/include/tensorflow/core/lib/core/status.h:24,
from /root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/include/tensorflow/core/framework/op_def_builder.h:25,
from /root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/include/tensorflow/core/framework/op.h:23,
from /root/styletest/dnnlib/tflib/ops/fused_bias_act.cu:9:
/root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/include/absl/base/policy_checks.h:77:2: error: #error “C++ versions less than C++11 are not supported.”
#error “C++ versions less than C++11 are not supported.”

Okay so this issue is documented. I just need to add -std=c++11 to line 64 of dnnlib/tflib/custom_ops.py done! Let’s try it again:

(stylegan2) [root@craig001 styletest]# python run_generator.py generate-images --network=gdrive:networks/stylegan2-ffhq-config-f.pkl --seeds=6600-6625 --truncation-psi=0.5
Local submit - run_dir: results/00002-generate-images
dnnlib: Running run_generator.generate_images() on localhost…
Loading networks from “gdrive:networks/stylegan2-ffhq-config-f.pkl”…
Setting up TensorFlow plugin “fused_bias_act.cu”: Preprocessing… Compiling… Loading… Failed!
Traceback (most recent call last):
File “run_generator.py”, line 168, in
main()
File “run_generator.py”, line 163, in main
dnnlib.submit_run(sc, func_name_map[subcmd], **kwargs)
File “/root/styletest/dnnlib/submission/submit.py”, line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File “/root/styletest/dnnlib/submission/internal/local.py”, line 22, in submit
return run_wrapper(submit_config)
File “/root/styletest/dnnlib/submission/submit.py”, line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File “/root/styletest/run_generator.py”, line 21, in generate_images
_G, _D, Gs = pretrained_networks.load_networks(network_pkl)
File “/root/styletest/pretrained_networks.py”, line 76, in load_networks
G, D, Gs = pickle.load(stream, encoding=‘latin1’)
File “/root/styletest/dnnlib/tflib/network.py”, line 297, in setstate
self._init_graph()
File “/root/styletest/dnnlib/tflib/network.py”, line 154, in _init_graph
out_expr = self._build_func(*self.input_templates, **build_kwargs)
File “”, line 491, in G_synthesis_stylegan2
File “”, line 455, in layer
File “”, line 99, in modulated_conv2d_layer
File “”, line 68, in apply_bias_act
File “/root/styletest/dnnlib/tflib/ops/fused_bias_act.py”, line 68, in fused_bias_act
return impl_dict[impl](x=x, b=b, axis=axis, act=act, alpha=alpha, gain=gain)
File “/root/styletest/dnnlib/tflib/ops/fused_bias_act.py”, line 122, in _fused_bias_act_cuda
cuda_kernel = _get_plugin().fused_bias_act
File “/root/styletest/dnnlib/tflib/ops/fused_bias_act.py”, line 16, in _get_plugin
return custom_ops.get_plugin(os.path.splitext(file)[0] + ‘.cu’)
File “/root/styletest/dnnlib/tflib/custom_ops.py”, line 156, in get_plugin
plugin = tf.load_op_library(bin_file)
File “/root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/python/framework/load_library.py”, line 61, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /root/styletest/dnnlib/tflib/_cudacache/fused_bias_act_f0526cfb20b0ead5b27c533d56d631ce.so: undefined symbol: _ZN10tensorflow12OpDefBuilder6OutputESs

Alright this issue appears to be documented as well at stylegan2报错“undefined symbol: _ZN10tensorflow12OpDefBuilder6OutputESs”的解决方案_zaf赵的博客-CSDN博客 by replacing a 0 with a 1 on line 127 of dnnlib/tflib/custom_ops.py and let’s try it again:

(stylegan2) [root@craig001 styletest]# python run_generator.py generate-images --network=gdrive:networks/stylegan2-ffhq-config-f.pkl --seeds=6600-6625 --truncation-psi=0.5
Local submit - run_dir: results/00002-generate-images
dnnlib: Running run_generator.generate_images() on localhost…
Loading networks from “gdrive:networks/stylegan2-ffhq-config-f.pkl”…
Setting up TensorFlow plugin “fused_bias_act.cu”: Preprocessing… Compiling… Loading… Failed!
Traceback (most recent call last):
File “run_generator.py”, line 168, in
main()
File “run_generator.py”, line 163, in main
dnnlib.submit_run(sc, func_name_map[subcmd], **kwargs)
File “/root/styletest/dnnlib/submission/submit.py”, line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File “/root/styletest/dnnlib/submission/internal/local.py”, line 22, in submit
return run_wrapper(submit_config)
File “/root/styletest/dnnlib/submission/submit.py”, line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File “/root/styletest/run_generator.py”, line 21, in generate_images
_G, _D, Gs = pretrained_networks.load_networks(network_pkl)
File “/root/styletest/pretrained_networks.py”, line 76, in load_networks
G, D, Gs = pickle.load(stream, encoding=‘latin1’)
File “/root/styletest/dnnlib/tflib/network.py”, line 297, in setstate
self._init_graph()
File “/root/styletest/dnnlib/tflib/network.py”, line 154, in _init_graph
out_expr = self._build_func(*self.input_templates, **build_kwargs)
File “”, line 491, in G_synthesis_stylegan2
File “”, line 455, in layer
File “”, line 99, in modulated_conv2d_layer
File “”, line 68, in apply_bias_act
File “/root/styletest/dnnlib/tflib/ops/fused_bias_act.py”, line 68, in fused_bias_act
return impl_dict[impl](x=x, b=b, axis=axis, act=act, alpha=alpha, gain=gain)
File “/root/styletest/dnnlib/tflib/ops/fused_bias_act.py”, line 122, in _fused_bias_act_cuda
cuda_kernel = _get_plugin().fused_bias_act
File “/root/styletest/dnnlib/tflib/ops/fused_bias_act.py”, line 16, in _get_plugin
return custom_ops.get_plugin(os.path.splitext(file)[0] + ‘.cu’)
File “/root/styletest/dnnlib/tflib/custom_ops.py”, line 156, in get_plugin
plugin = tf.load_op_library(bin_file)
File “/root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/python/framework/load_library.py”, line 61, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /root/styletest/dnnlib/tflib/_cudacache/fused_bias_act_f0526cfb20b0ead5b27c533d56d631ce.so: undefined symbol: _ZN10tensorflow12OpDefBuilder6OutputESs
(stylegan2) [root@craig001 styletest]# vim dnnlib/tflib/custom_ops.py
(stylegan2) [root@craig001 styletest]# python run_generator.py generate-images --network=gdrive:networks/stylegan2-ffhq-config-f.pkl --seeds=6600-6625 --truncation-psi=0.5
Local submit - run_dir: results/00003-generate-images
dnnlib: Running run_generator.generate_images() on localhost…
Loading networks from “gdrive:networks/stylegan2-ffhq-config-f.pkl”…
Setting up TensorFlow plugin “fused_bias_act.cu”: Preprocessing… Compiling… Loading… Failed!
Traceback (most recent call last):
File “run_generator.py”, line 168, in
main()
File “run_generator.py”, line 163, in main
dnnlib.submit_run(sc, func_name_map[subcmd], **kwargs)
File “/root/styletest/dnnlib/submission/submit.py”, line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File “/root/styletest/dnnlib/submission/internal/local.py”, line 22, in submit
return run_wrapper(submit_config)
File “/root/styletest/dnnlib/submission/submit.py”, line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File “/root/styletest/run_generator.py”, line 21, in generate_images
_G, _D, Gs = pretrained_networks.load_networks(network_pkl)
File “/root/styletest/pretrained_networks.py”, line 76, in load_networks
G, D, Gs = pickle.load(stream, encoding=‘latin1’)
File “/root/styletest/dnnlib/tflib/network.py”, line 297, in setstate
self._init_graph()
File “/root/styletest/dnnlib/tflib/network.py”, line 154, in _init_graph
out_expr = self._build_func(*self.input_templates, **build_kwargs)
File “”, line 491, in G_synthesis_stylegan2
File “”, line 455, in layer
File “”, line 99, in modulated_conv2d_layer
File “”, line 68, in apply_bias_act
File “/root/styletest/dnnlib/tflib/ops/fused_bias_act.py”, line 68, in fused_bias_act
return impl_dict[impl](x=x, b=b, axis=axis, act=act, alpha=alpha, gain=gain)
File “/root/styletest/dnnlib/tflib/ops/fused_bias_act.py”, line 122, in _fused_bias_act_cuda
cuda_kernel = _get_plugin().fused_bias_act
File “/root/styletest/dnnlib/tflib/ops/fused_bias_act.py”, line 16, in _get_plugin
return custom_ops.get_plugin(os.path.splitext(file)[0] + ‘.cu’)
File “/root/styletest/dnnlib/tflib/custom_ops.py”, line 156, in get_plugin
plugin = tf.load_op_library(bin_file)
File “/root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow_core/python/framework/load_library.py”, line 61, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /root/styletest/dnnlib/tflib/_cudacache/fused_bias_act_2b9743ab274bc06608200c3b6cabec14.so: undefined symbol: _ZN10tensorflow12OpDefBuilder6OutputESs

Same error message. I’ve tried this with Tensorflow 1.14 and the same result. I also tried this same journey on my Ubuntu 16.04 machine and the fixes worked. So does anyone know what the difference is that allows it to work on Ubuntu but not on CentOS?

2 Likes

Hi, Paul-

Did you ever figure this out? Staring at the exact same error message…

Dude, I LOVE YOU! For some reason, changing the 0 to 1 on line 127 of custom_ops DID IT!!! I wasn’t expecting it to work, and then it did!

I’m glad it worked for you on 16.04. I had 20.04 but tried all of this on 18.04, and it looks like this worked.

2 Likes

Thanks you both! changing the 0 to 1 on line 127 of custom_ops was exactly what I needed!
For the next one who reads this, here is what you need to change:
In file stylegan2/dnnlib/tflib/custom_ops.py, line 127:
change from
compile_opts += ’ --compiler-options \‘-fPIC -D_GLIBCXX_USE_CXX11_ABI=0\’’
to
compile_opts += ’ --compiler-options \‘-fPIC -D_GLIBCXX_USE_CXX11_ABI=1\’’

1 Like