I was keen to test the new automatic mixed precision you are offering in your tensorflow container nvcr.io/nvidia/tensorflow:19.03-py3. My code works fine normally but when I enable auto mixed precision it fails for me.
root@7ea1fde48ca8:/workspace# export TF_ENABLE_AUTO_MIXED_PRECISION=1
root@7ea1fde48ca8:/workspace# python neural_style.py --content examples/1-content.jpg --styles examples/1-style.jpg --network imagenet-vgg-verydeep-19.mat --output blah16.jpg
2019-03-22 05:36:33.023660: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300040000 Hz
2019-03-22 05:36:33.024432: I tensorflow/compiler/xla/service/service.cc:161] XLA service 0x54547f0 executing computations on platform Host. Devices:
2019-03-22 05:36:33.024471: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (0): ,
2019-03-22 05:36:33.154671: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-03-22 05:36:33.155276: I tensorflow/compiler/xla/service/service.cc:161] XLA service 0x5524160 executing computations on platform CUDA. Devices:
2019-03-22 05:36:33.155316: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
2019-03-22 05:36:33.155783: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:1e.0
totalMemory: 15.75GiB freeMemory: 15.44GiB
2019-03-22 05:36:33.155816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-03-22 05:36:33.649698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-22 05:36:33.649768: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-03-22 05:36:33.649790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-03-22 05:36:33.650171: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14938 MB memory) → physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2019-03-22 05:36:34.072402: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1704] Running auto_mixed_precision graph optimizer
2019-03-22 05:36:34.072972: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1230] No whitelist ops found, nothing to do
2019-03-22 05:36:35.132201: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1704] Running auto_mixed_precision graph optimizer
2019-03-22 05:36:35.132875: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1230] No whitelist ops found, nothing to do
2019-03-22 05:36:36.585227: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-03-22 05:36:36.585311: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-22 05:36:36.585330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-03-22 05:36:36.585338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-03-22 05:36:36.585711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14938 MB memory) → physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2019-03-22 05:36:37.034372: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1704] Running auto_mixed_precision graph optimizer
2019-03-22 05:36:37.034714: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1230] No whitelist ops found, nothing to do
2019-03-22 05:36:37.082609: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1704] Running auto_mixed_precision graph optimizer
2019-03-22 05:36:37.083845: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1230] No whitelist ops found, nothing to do
2019-03-22 05:36:37.304684: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1704] Running auto_mixed_precision graph optimizer
2019-03-22 05:36:37.305175: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1230] No whitelist ops found, nothing to do
2019-03-22 05:36:37.616460: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1704] Running auto_mixed_precision graph optimizer
2019-03-22 05:36:37.617087: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1230] No whitelist ops found, nothing to do
2019-03-22 05:36:38.177550: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1704] Running auto_mixed_precision graph optimizer
2019-03-22 05:36:38.178386: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1230] No whitelist ops found, nothing to do WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-03-22 05:36:40.165986: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-03-22 05:36:40.166069: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-22 05:36:40.166089: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-03-22 05:36:40.166098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-03-22 05:36:40.166442: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14938 MB memory) → physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2019-03-22 05:36:40.190878: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1704] Running auto_mixed_precision graph optimizer
2019-03-22 05:36:40.191325: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1230] No whitelist ops found, nothing to do Optimization started…
2019-03-22 05:36:40.511646: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1704] Running auto_mixed_precision graph optimizer
2019-03-22 05:36:40.513016: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1660] Converted 44/97 nodes to float16 precision using 0 cast(s) to float16 (excluding Const and Variable casts) content loss: 2.14634e+06
2019-03-22 05:36:42.648689: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1704] Running auto_mixed_precision graph optimizer
2019-03-22 05:36:42.650502: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1660] Converted 58/149 nodes to float16 precision using 0 cast(s) to float16 (excluding Const and Variable casts)
2019-03-22 05:36:42.893699: I tensorflow/stream_executor/dso_loader.cc:153] successfully opened CUDA library libcublas.so.10 locally style loss: inf
2019-03-22 05:36:43.451683: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1704] Running auto_mixed_precision graph optimizer
2019-03-22 05:36:43.452169: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1230] No whitelist ops found, nothing to do tv loss: 0
2019-03-22 05:36:43.470820: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1704] Running auto_mixed_precision graph optimizer
2019-03-22 05:36:43.472961: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1660] Converted 59/204 nodes to float16 precision using 0 cast(s) to float16 (excluding Const and Variable casts) total loss: inf Traceback (most recent call last):
File “neural_style.py”, line 224, in
main()
File “neural_style.py”, line 184, in main checkpoint_iterations=options.checkpoint_iterations
File “/workspace/stylize.py”, line 145, in stylize
train_step.run()
AttributeError: ‘Tensor’ object has no attribute ‘run’
You can reproduce by cloning: GitHub - evanthomas/neural-style: Neural style in TensorFlow!
You will need to get the model here: FloydHub Blog
(The one linked in the github page has changed for some reason).
Evan.