Hello everyone,
I am using tensorflow models to do some image analysis and I observe some strange prediction artifacts once the input to my model exceeds a certain size.
Here is a minimal working example to illustrate the artifact:
import tensorflow as tf
from tensorflow.keras.models import model_from_json
import numpy as np
from tifffile import imsave
# load u-net architecture, kernel-size = 3, depth = 2
json_file = open('model.json', 'r')
loaded_model_json = json_file.read()
json_file.close()
model = model_from_json(loaded_model_json)
# set all weights to 0.5
w = model.get_weights()
init = []
for l in w:
init.append(l*0 + 0.5)
model.set_weights(init)
tmp = np.zeros((180, 196, 240), np.float32)
for i in range(tmp.shape[0]):
tmp[i] = i
tmp = tmp[:, :, :, np.newaxis]
prediction_fail = np.moveaxis(model.predict(np.moveaxis(tmp[np.newaxis], 4, -1))[0], 3, 0)
tmp = tmp[:176]
prediction_succ = np.moveaxis(model.predict(np.moveaxis(tmp[np.newaxis], 4, -1))[0], 3, 0)
imsave('input.tif', tmp)
imsave('prediction_fail.tif', prediction_fail[0])
imsave('prediction_success.tif', prediction_succ[0])
The model used in the example is described with this model.json file:
{"class_name": "Model", "config": {"name": "model_3", "layers": [{"name": "input", "class_name": "InputLayer", "config": {"batch_input_shape": [null, null, null, null, 1], "dtype": "float32", "sparse": false, "name": "input"}, "inbound_nodes": []}, {"name": "down_level_0_no_0", "class_name": "Conv3D", "config": {"name": "down_level_0_no_0", "trainable": true, "filters": 32, "kernel_size": [3, 3, 3], "strides": [1, 1, 1], "padding": "same", "data_format": "channels_last", "dilation_rate": [1, 1, 1], "activation": "relu", "use_bias": true, "kernel_initializer": {"class_name": "VarianceScaling", "config": {"scale": 1.0, "mode": "fan_avg", "distribution": "uniform", "seed": null}}, "bias_initializer": {"class_name": "Zeros", "config": {}}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "inbound_nodes": [[["input", 0, 0, {}]]]}, {"name": "down_level_0_no_1", "class_name": "Conv3D", "config": {"name": "down_level_0_no_1", "trainable": true, "filters": 32, "kernel_size": [3, 3, 3], "strides": [1, 1, 1], "padding": "same", "data_format": "channels_last", "dilation_rate": [1, 1, 1], "activation": "relu", "use_bias": true, "kernel_initializer": {"class_name": "VarianceScaling", "config": {"scale": 1.0, "mode": "fan_avg", "distribution": "uniform", "seed": null}}, "bias_initializer": {"class_name": "Zeros", "config": {}}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "inbound_nodes": [[["down_level_0_no_0", 0, 0, {}]]]}, {"name": "max_0", "class_name": "MaxPooling3D", "config": {"name": "max_0", "trainable": true, "pool_size": [2, 2, 2], "padding": "valid", "strides": [2, 2, 2], "data_format": "channels_last"}, "inbound_nodes": [[["down_level_0_no_1", 0, 0, {}]]]}, {"name": "down_level_1_no_0", "class_name": "Conv3D", "config": {"name": "down_level_1_no_0", "trainable": true, "filters": 64, "kernel_size": [3, 3, 3], "strides": [1, 1, 1], "padding": "same", "data_format": "channels_last", "dilation_rate": [1, 1, 1], "activation": "relu", "use_bias": true, "kernel_initializer": {"class_name": "VarianceScaling", "config": {"scale": 1.0, "mode": "fan_avg", "distribution": "uniform", "seed": null}}, "bias_initializer": {"class_name": "Zeros", "config": {}}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "inbound_nodes": [[["max_0", 0, 0, {}]]]}, {"name": "down_level_1_no_1", "class_name": "Conv3D", "config": {"name": "down_level_1_no_1", "trainable": true, "filters": 64, "kernel_size": [3, 3, 3], "strides": [1, 1, 1], "padding": "same", "data_format": "channels_last", "dilation_rate": [1, 1, 1], "activation": "relu", "use_bias": true, "kernel_initializer": {"class_name": "VarianceScaling", "config": {"scale": 1.0, "mode": "fan_avg", "distribution": "uniform", "seed": null}}, "bias_initializer": {"class_name": "Zeros", "config": {}}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "inbound_nodes": [[["down_level_1_no_0", 0, 0, {}]]]}, {"name": "max_1", "class_name": "MaxPooling3D", "config": {"name": "max_1", "trainable": true, "pool_size": [2, 2, 2], "padding": "valid", "strides": [2, 2, 2], "data_format": "channels_last"}, "inbound_nodes": [[["down_level_1_no_1", 0, 0, {}]]]}, {"name": "middle_0", "class_name": "Conv3D", "config": {"name": "middle_0", "trainable": true, "filters": 128, "kernel_size": [3, 3, 3], "strides": [1, 1, 1], "padding": "same", "data_format": "channels_last", "dilation_rate": [1, 1, 1], "activation": "relu", "use_bias": true, "kernel_initializer": {"class_name": "VarianceScaling", "config": {"scale": 1.0, "mode": "fan_avg", "distribution": "uniform", "seed": null}}, "bias_initializer": {"class_name": "Zeros", "config": {}}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "inbound_nodes": [[["max_1", 0, 0, {}]]]}, {"name": "middle_2", "class_name": "Conv3D", "config": {"name": "middle_2", "trainable": true, "filters": 64, "kernel_size": [3, 3, 3], "strides": [1, 1, 1], "padding": "same", "data_format": "channels_last", "dilation_rate": [1, 1, 1], "activation": "relu", "use_bias": true, "kernel_initializer": {"class_name": "VarianceScaling", "config": {"scale": 1.0, "mode": "fan_avg", "distribution": "uniform", "seed": null}}, "bias_initializer": {"class_name": "Zeros", "config": {}}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "inbound_nodes": [[["middle_0", 0, 0, {}]]]}, {"name": "up_sampling3d_5", "class_name": "UpSampling3D", "config": {"name": "up_sampling3d_5", "trainable": true, "size": [2, 2, 2], "data_format": "channels_last"}, "inbound_nodes": [[["middle_2", 0, 0, {}]]]}, {"name": "concatenate_5", "class_name": "Concatenate", "config": {"name": "concatenate_5", "trainable": true, "axis": -1}, "inbound_nodes": [[["up_sampling3d_5", 0, 0, {}], ["down_level_1_no_1", 0, 0, {}]]]}, {"name": "up_level_1_no_0", "class_name": "Conv3D", "config": {"name": "up_level_1_no_0", "trainable": true, "filters": 64, "kernel_size": [3, 3, 3], "strides": [1, 1, 1], "padding": "same", "data_format": "channels_last", "dilation_rate": [1, 1, 1], "activation": "relu", "use_bias": true, "kernel_initializer": {"class_name": "VarianceScaling", "config": {"scale": 1.0, "mode": "fan_avg", "distribution": "uniform", "seed": null}}, "bias_initializer": {"class_name": "Zeros", "config": {}}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "inbound_nodes": [[["concatenate_5", 0, 0, {}]]]}, {"name": "up_level_1_no_2", "class_name": "Conv3D", "config": {"name": "up_level_1_no_2", "trainable": true, "filters": 32, "kernel_size": [3, 3, 3], "strides": [1, 1, 1], "padding": "same", "data_format": "channels_last", "dilation_rate": [1, 1, 1], "activation": "relu", "use_bias": true, "kernel_initializer": {"class_name": "VarianceScaling", "config": {"scale": 1.0, "mode": "fan_avg", "distribution": "uniform", "seed": null}}, "bias_initializer": {"class_name": "Zeros", "config": {}}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "inbound_nodes": [[["up_level_1_no_0", 0, 0, {}]]]}, {"name": "up_sampling3d_6", "class_name": "UpSampling3D", "config": {"name": "up_sampling3d_6", "trainable": true, "size": [2, 2, 2], "data_format": "channels_last"}, "inbound_nodes": [[["up_level_1_no_2", 0, 0, {}]]]}, {"name": "concatenate_6", "class_name": "Concatenate", "config": {"name": "concatenate_6", "trainable": true, "axis": -1}, "inbound_nodes": [[["up_sampling3d_6", 0, 0, {}], ["down_level_0_no_1", 0, 0, {}]]]}, {"name": "up_level_0_no_0", "class_name": "Conv3D", "config": {"name": "up_level_0_no_0", "trainable": true, "filters": 32, "kernel_size": [3, 3, 3], "strides": [1, 1, 1], "padding": "same", "data_format": "channels_last", "dilation_rate": [1, 1, 1], "activation": "relu", "use_bias": true, "kernel_initializer": {"class_name": "VarianceScaling", "config": {"scale": 1.0, "mode": "fan_avg", "distribution": "uniform", "seed": null}}, "bias_initializer": {"class_name": "Zeros", "config": {}}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "inbound_nodes": [[["concatenate_6", 0, 0, {}]]]}, {"name": "up_level_0_no_2", "class_name": "Conv3D", "config": {"name": "up_level_0_no_2", "trainable": true, "filters": 32, "kernel_size": [3, 3, 3], "strides": [1, 1, 1], "padding": "same", "data_format": "channels_last", "dilation_rate": [1, 1, 1], "activation": "relu", "use_bias": true, "kernel_initializer": {"class_name": "VarianceScaling", "config": {"scale": 1.0, "mode": "fan_avg", "distribution": "uniform", "seed": null}}, "bias_initializer": {"class_name": "Zeros", "config": {}}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "inbound_nodes": [[["up_level_0_no_0", 0, 0, {}]]]}, {"name": "conv3d_3", "class_name": "Conv3D", "config": {"name": "conv3d_3", "trainable": true, "filters": 1, "kernel_size": [1, 1, 1], "strides": [1, 1, 1], "padding": "valid", "data_format": "channels_last", "dilation_rate": [1, 1, 1], "activation": "linear", "use_bias": true, "kernel_initializer": {"class_name": "VarianceScaling", "config": {"scale": 1.0, "mode": "fan_avg", "distribution": "uniform", "seed": null}}, "bias_initializer": {"class_name": "Zeros", "config": {}}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "inbound_nodes": [[["up_level_0_no_2", 0, 0, {}]]]}, {"name": "add_19", "class_name": "Add", "config": {"name": "add_19", "trainable": true}, "inbound_nodes": [[["conv3d_3", 0, 0, {}], ["input", 0, 0, {}]]]}, {"name": "activation_52", "class_name": "Activation", "config": {"name": "activation_52", "trainable": true, "activation": "linear"}, "inbound_nodes": [[["add_19", 0, 0, {}]]]}], "input_layers": [["input", 0, 0]], "output_layers": [["activation_52", 0, 0]]}, "keras_version": "2.2.0", "backend": "tensorflow"}
In the example I initialize the model weights with 0.5 and then predict the 3D image volume (Z: 180, Y: 196, X: 240) with a gradient in Z direction. As output I get the gradient up to Z around 140 and then it changes to a constant smaller values.
If I input a slightly smaller (Z: 176, Y: 196, X: 240) image, the gradient is preserved in the output.
I am observing this behavior on a system with the following specs:
- Linux Mint 18
- cuda: 9.0.176
- cudnn: 7.1.2
- tensorflow: 1.10
- GPU: Tesla P40
I already tried different driver versions and tensorflow 1.8. But I always get the same behavior.
I am grateful for any explanation or any hints on what it could be or how to fix this issue (without cropping/tiling the input).
Best,
tibuch