Q:

No GPU kernel for an int32 Variable op

It seems that there is no GPU kernel for the Variable op when I initialize it with an integer tensor. For example, running:
with tf.device('/gpu:0'):
  var = tf.Variable(initial_value=([[1,2],[3,4]]))
sess = tf.Session()
sess.run(var)

Will result in an exception:
tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'Variable': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available

Does this mean that all variables in TensorFlow are stored on the memory associated with the CPU device (the RAM I guess) and not on GPUmemory? Isn't it slower when we want to train a model using only one GPU, because the variables values will have to be copied many times from the RAM to the GPU memory?
Thanks!

A:

TL;DR: There is no kernel for int32-valued Variable ops on GPU, but other element types (float16/float32/float64) are supported on GPU.

TensorFlow currently only supports (half-, single-, and double-precision) floating-point variables on GPU. Indeed, in general, TensorFlow has limited support for int32 operations on GPU, and you will see intriguing comments like this throughout the C++ code:
// A special GPU kernel for int32.
// TODO(b/25387198): Also enable int32 in device memory. This kernel
// registration requires all int32 inputs and outputs to be in host memory.

It's not particularly difficult to run operations on int32 data on a GPU, so why doesn't TensorFlow support it?

The default placement algorithm in TensorFlow greedily puts operations on the "best possible" device for that operation. That means on a GPU if one is available, otherwise on a CPU.
Empirically, we've found that most int32 operations in real models produce and consume "metadata" (like shapes of tensors), and this information is typically in host memory. These operations crop up in gradient code, to handle variable-sized tensors.
Registering int32 kernels on GPU means that these small metadata tensors must be copied to and from device memory, in order to run typically very cheap operations. Furthermore, copying results back to host memory can stall the GPU stream, which leads to much poorer performance.

As a result, we made a compromise that removes the registration of many int32 kernels, while we develop a smarter placement algorithm that can recognize when operations don't benefit from running on a GPU. Since very few operations are registered for int32 on GPU, there's little or no benefit to having variables sit on the GPU, consuming the relatively scarce memory.

