Q:

How to use Nomad with Nvidia Docker?

Is there any way to use nvidia-docker with Nomad?
The program for computing on Nvidia works locally but it doesn't work with nvidia-docker (it uses CPU instead of GPU).
What is the preferred way to do that?

Use nvidia-docker driver for Nomad
Use raw docker exec to run nvidia-docker
Somehow connect Nomad to nvidia-docker engine

Has anyone experience with that?

A:

The idea was to create a proper Docker image for that:
FROM debian:wheezy

# Run Ubuntu in non-interactive mode
ENV DEBIAN_FRONTEND noninteractive

# Provide CUDA environmental variables that match the installed version on host machine
ENV CUDA_DRIVER  375.39
ENV CUDA_INSTALL http://us.download.nvidia.com/XFree86/Linux-x86_64/${CUDA_DRIVER}/NVIDIA-Linux-x86_64-${CUDA_DRIVER}.run

# Configure dependencies
RUN \
# Update available packages
  apt-get update \
            --quiet \
# Install all requirements
  && apt-get install \
            --yes \
            --no-install-recommends \
            --no-install-suggests \
       build-essential \
       module-init-tools \
       wget \
# Clean up leftovers
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

# Install CUDA drivers
RUN wget \
      $CUDA_INSTALL \
        -P /tmp \
        --no-verbose \
      && chmod +x /tmp/NVIDIA-Linux-x86_64-${CUDA_DRIVER}.run \
      && /tmp/NVIDIA-Linux-x86_64-${CUDA_DRIVER}.run \
        -s \
        -N \
        --no-kernel-module \
      && rm -rf /tmp/*

ENTRYPOINT ["/bin/bash"]

and then:

build the base Docker image:
docker build . -t cuda

start a container with cuda base image execute:
docker run \
  --device=/dev/nvidia0:/dev/nvidia0 \
  --device=/dev/nvidiactl:/dev/nvidiactl \
  --device=/dev/nvidia-uvm:/dev/nvidia-uvm \
  -it \
  --rm cuda

A message like:
Failed to initialize NVML: Unknown Error

Could be due to a mismatch between the host and container driver versions of missing host /dev entries.

