float16 Scatter update does not work on GPU

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hi, the following code fails on my Jetson TX2 with tensorflow 1.9.0&CUDA 9.0:

import tensorflow as tf

tf_float = tf.float16
device = '/device:GPU:0'
with tf.device(device):
    queue = tf.get_variable(name="SomeTensor",
                            initializer=tf.ones([1],
                                                dtype=tf_float),
                            dtype=tf_float,
                            trainable=False,
                            collections=[tf.GraphKeys.LOCAL_VARIABLES])
    write = tf.scatter_update(queue, 0, 2).op

with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as session:
    session.run(queue.initializer)
    session.run(write)

The error message:

InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'SomeTensor': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.
Colocation Debug Info:
Colocation group had the following types and devices:
ScatterUpdate: CPU
Assign: CPU
Identity: GPU CPU
VariableV2: GPU CPU

Colocation members and user-requested devices:
  SomeTensor (VariableV2) /device:GPU:0
  SomeTensor/Assign (Assign) /device:GPU:0
  SomeTensor/read (Identity) /device:GPU:0
  ScatterUpdate (ScatterUpdate) /device:GPU:0

Registered kernels:
  device='CPU'
  device='GPU'; dtype in [DT_INT64]
  device='GPU'; dtype in [DT_DOUBLE]
  device='GPU'; dtype in [DT_FLOAT]
  device='GPU'; dtype in [DT_HALF]

         [[Node: SomeTensor = VariableV2[container="", dtype=DT_HALF, shape=[1], shared_name="", _device="/device:GPU:0"]()]]

Suggests that scatter_update cannot be placed to GPU.
Everything works fine with tf.float32 data type. I can also place scatter_update on CPU.
Am I doing something wrong?
Is there any way to fix this?

Thanks a lot in advance for any help.

2

Hi,

This issue is not specified for Jetson platform.
I can reproduce same issue on an x86 environment.

To fix this, you can enable the soft placement flag to allow mixed placement.

with tf.Session(config=tf.ConfigProto(allow_soft_placement=True,log_device_placement=True)) as session:

Please remember that this will lower the performance since the conversion of different tensor type is required.

Thanks.

3

Thanks for checking!