<jupyter_start><jupyter_text>How to execute scripts in docker * **Difficulty level**: easy* **Time need to lean**: 10 minutes or less* **Key points**:  * The `container` option runs the script inside the specified docker container    Running a script inside docker SoS executes scripts inside docker by calling command `docker run` with appropriate parameters. Suppose you do not have ruby installed locally and would like to run a ruby script, you can execute it inside a `ruby` container by specifying the docker image to use:<jupyter_code>%run
ruby: container='ruby'
    line1 = "Cats are smarter than dogs";
    line2 = "Dogs also like meat";

    if ( line1 =~ /Cats(.*)/ )
      puts "Line1 contains Cats"
    end
    if ( line2 =~ /Cats(.*)/ )
      puts "Line2 contains  Dogs"
    end<jupyter_output>Line1 contains Cats<jupyter_text>The actual `docker run` command executed by SoS can be shown when you execute the script in dryrun mode (with `-n` option). In this mode, SoS would print the interpolated script (if option `expand=True` is set) and the docker command to execute it:<jupyter_code>%rerun -n<jupyter_output><empty_output><jupyter_text>As you can see, the docker command looks similar to```docker run --rm      -v $(pwd):$(pwd)    -v /tmp/path/to/docker_run_30258.rb:/var/lib/sos/docker_run_30258.rb    -t -P     -w=/Users/bpeng1/sos/sos-docs/src/tutorials    -u 12345:54321    ruby    ruby /var/lib/sos/docker_run_30258.rb```Basically, SoS downloads a docker image called `ruby` and runs command `docker run` to execte the specified script, with the following options* `--rm` Automatically remove the container when it exits* `-v $(pwd):$(pwd)` maps current working directory to the docker image so that it can be accessed from within the docker image* `-v /tmp/path/to/docker_run_30258.rb:/var/lib/sos/docker_run_30258.rb` maps a temporary script (`/Users/bpeng1/sos/sos-docs/src/tutorials/tmp2zviq3qh/docker_run_30258.rb` to the docker image.* `-t` Allocate a pseudo-tty* `-P` Publish all exposed ports to the host interfaces* `-w=/Users/bpeng1/sos/sos-docs/src/tutorials` Set working directory to current working directory* `-u 12345:54321` Use the host user-id and group-id inside docker so that files created by docker (on shared volumes) could be accessible from outside of docker.* `ruby` name of the docker image* `ruby /var/lib/sos/docker_run_30258.rb` command that execute the script.The details of these options could be found at the [docker run manual](https://docs.docker.com/engine/reference/run/). They are chosen by the default to work with a majority of the scenarios but can fail for some docker images, in which case you will have to use additional options to customized how the images are executed. This tutorial demonstrates the use of options for some common scenarios but please refer to the SoS documentation on [general action options](https://vatlab.github.io/sos-docs/doc/documentation/Targets_and_Actions.htmlAction-options-12) and [options that are specific to `docker_image`](https://vatlab.github.io/sos-docs/doc/documentation/Targets_and_Actions.htmldocker_image). Note that```ruby: container='ruby'```is a shortcut for```ruby: container='docker://ruby'```  Building docker images  Action `docker_build`  Building a docker image is usually done outside of SoS if you are maintaining a collection of docker containers to be shared by your workflows, your groups, or everyone. However, if you need to create a docker image on-the-fly or would like to embed the Dockerfile inside a SoS script, you can use the `docker_build` action to build a docker container.For example, you can build simple image<jupyter_code>docker_build: tag='test_docker'
  FROM ubuntu:14.04<jupyter_output>Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM ubuntu:14.04
 ---> 8cef1fa16c77
Successfully built 8cef1fa16c77
Successfully tagged test_docker:latest<jupyter_text>and use the image<jupyter_code>sh: docker_image='test_docker'
  ls /usr<jupyter_output>bin  games  include  lib  local  sbin  share  src<jupyter_text>This tutorial will use the `docker_build` action to build a few simple docker images to demonstrate the use of various options.  Docker build options Action `docker_build` accepts usual SoS action options such as `workdir`, `stdout`, and `stderr`. For example, you can suppress the output of the action using options `stdout=False` and/or `stderr=False`:<jupyter_code>docker_build: tag='test_docker', stdout=False
  FROM ubuntu:14.04<jupyter_output><empty_output><jupyter_text>This action also accepts all options to command `docker build`, as listed in the [docker build documentation](https://docs.docker.com/engine/reference/commandline/build/). Only the long format of the option names are accepted (e.g. `--memory` is acceptable but not `-m`). Option names containing hyphens (e.g. `disable-content-trust`) should have hyphens relaced with underscores. Boolean options should be specified as `name=True`.For example,<jupyter_code>docker_build: tag='test_docker', label='test image',
  compress=True, no_cache=True, memory='2G'

  FROM ubuntu:14.04<jupyter_output>Sending build context to Docker daemon     126B
Step 1/2 : FROM ubuntu:14.04
 ---> 8cef1fa16c77
Step 2/2 : LABEL "test image"=''
 ---> Running in 56f529bd5495
Removing intermediate container 56f529bd5495
 ---> 05724ccffb6f
Successfully built 05724ccffb6f
Successfully tagged test_docker:latest<jupyter_text>Running docker images  Customized working directory (`workdir` and `docker_workdir`) SoS by default sets the current working directory of the docker image to the working directory of the host system, essentially adding `-w $(pwd)` to the command line. For example, with the following docker image, the `pwd` of the script is the current working directory on the host machine.<jupyter_code>sh: container='ubuntu:14.04'
  echo `pwd`<jupyter_output>/Users/bpeng1/sos/sos-docs/src/tutorials<jupyter_text>Since the action option `workdir` can change the working directory of the script, you can use this option to change the script of the working directory of the docker image as well. For example, SoS in the following example will change the current working directory to the parent directory before executing `docker run` there.<jupyter_code>sh: container='ubuntu:14.04', workdir='..'
  echo `pwd`<jupyter_output>/Users/bpeng1/sos/sos-docs/src<jupyter_text>This default behavior is convenient when you use commands in docker images to process input files on your host machine but it has two caveats:1. The docker image might have its own `WORKDIR` for the command to work. For example, a docker image can provide an application that is not in standard `$PATH` so it can only be executed in a specified `WORKDIR`.2. You might need to specify a working directory inside of docker that does not exist in the host machine. Option `docker_workdir`, if specified, overrides `workdir` and allows the use of default or customized working directory inside of docker images. When `docker_workdir` is set to `None`, no `-w` option will be passed to the docker image and the default `WORKDIR` will be used. Otherwise an absolute path inside the docker image can be specified.For example, the following customized docker image has a `WORKDIR` set to `/usr`. It is working directory is set to host working directory by default, to `/usr` with `docker_workdir=None`, and `/home/random_user` with `docker_workdir='/home/random_user'`.<jupyter_code>docker_build: tag='test_docker_workdir'
  FROM ubuntu:14.04
  WORKDIR /usr

sh: container='test_docker_workdir'
  echo `pwd`
  
sh: container='test_docker_workdir', docker_workdir=None
  echo `pwd`
  
sh: container='test_docker_workdir', docker_workdir='/home/random_user'
  echo `pwd`<jupyter_output>Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM ubuntu:14.04
 ---> 8cef1fa16c77
Step 2/2 : WORKDIR /usr
 ---> Using cache
 ---> 336ce9da9b8f
Successfully built 336ce9da9b8f
Successfully tagged test_docker_workdir:latest
/Users/bpeng1/sos/sos-docs/src/tutorials
/usr
/home/random_user<jupyter_text>Note the directory is relative to the docker file system so it does not have to exist on the host system. Docker also creates `docker_workdir` if it does not exist so you do not have to create the directory in advance.   Sharing of input and output files (`volumes`) Because the working directory of the docker image is set by default to the current working directory, you can apply a command inside a docker image to work on files in the current working directory, and create files in it as well.<jupyter_code>sh: container='ubuntu:14.04'
  wc -l SoS_Docker_Guide.ipynb > docker_wc.txt
  
sh:
  cat docker_wc.txt<jupyter_output>935 SoS_Docker_Guide.ipynb<jupyter_text>This works because SoS automatically shares the current working directory of the host system to the docker image. Because the docker image can only "see" file systems shared by command `docker run`, your script will fail if your input files or output files are not under current working directory. For example, the following script write something to a thumbdrive (`/Volumes/SANDISK` under mac)<jupyter_code>sh:
  wc -l SoS_Docker_Guide.ipynb > /Volumes/SANDISK/filelist<jupyter_output><empty_output><jupyter_text>The script would fail to execute in a docker image because the image cannot see the `/Volumes` file system<jupyter_code>%sandbox --expect-error
sh: container='ubuntu:14.04'
  wc -l SoS_Docker_Guide.ipynb > /Volumes/SANDISK/filelist<jupyter_output>/var/lib/sos/docker_run_60521.sh: 1: /var/lib/sos/docker_run_60521.sh: cannot create /Volumes/SANDISK/filelist: Directory nonexistent<jupyter_text>The problem could be solved by specifying additional shared file systems through parameter `volumes`. This parameter accepts one (a string) or a list of volumes (list of strings) in the format of* A single path (e.g. `/Users`) which would be shared to the docker image under the same name (e.g.  `/Users:/Users`).* A full volume specification `host-src:]container-dest[:]`, in which case host and container directories can have different names (e.g. `/Users:/home`).A special rule here is that the current working directory will not be mapped separately if it is under one of the specified volumes. That is to say, if the current working directly is `/Users/bpeng1/project` and option `volumes='/Users:/home'` is specified, current working directory will be implicitly mapped to `/home/bpeng1/project`. Consequently, if you would like to read input files from or write output files to another volume, you can add the paths to option `volumes`<jupyter_code>sh: container='ubuntu:14.04', volumes='/Volumes'
  wc -l SoS_Docker_Guide.ipynb > /Volumes/SANDISK/filelist

sh:
  cat /Volumes/SANDISK/filelist<jupyter_output>935 SoS_Docker_Guide.ipynb<jupyter_text>As another commonly used technique, some users prefer using "standard directories" as input and output directories of a script so that the scripts are more portable. For example, the following script maps source directory as `/input` and destination directory as `/output` and use `/input` and `/output` in the docker image:<jupyter_code>sh: container='ubuntu:14.04',
  volumes=['~/sos/sos-docs/src/tutorials:/input', '/Volumes:/output']
  wc -l /input/SoS_Docker_Guide.ipynb > /output/SANDISK/filelist

sh:
  cat /Volumes/SANDISK/filelist<jupyter_output>935 /input/SoS_Docker_Guide.ipynb<jupyter_text>Although it is tempting to share common directories (e.g. `/home`, `/Users`, `/Volumes` etc) to the docker image, sharing of extra directories can cause unexpected problems. For example, a docker image might contain a useful `/Users` directory and sharing host `/Users` will override the directory inside docker. Furthermore, sharing `$HOME` will expose a lot of user settings (e.g. settings under `~/.R`, `~/.sos`) to the docker image and might affect how the docker image runs. If you really need to expose a home directory to a docker image, you might want to expose it as a different directory inside of docker. For example, the following script shared home directory as `/data` inside docker so that it does not interfere with the home directory of the docker image.<jupyter_code>sh: container='ubuntu:14.04', volumes='~:/data'
  ls ~ | wc -l
  ls /data | wc -l<jupyter_output>21
49<jupyter_text>Customize user and group ID (`user`) (Pending) By default SoS passes option `--user $(uid):$(gid)` to command `docker run` to execute the command as the same user as the user on host machine. This makes sure that the docker image has read/write permission to shared volumes and the files written by the docker image are readable by the host machine.FIXME: example of using image user (`user=None`)FIXME: example of using another user (`user=blah`)<jupyter_code>docker_build: tag='test_docker_workdir'
  FROM ubuntu:14.04
  USER blah<jupyter_output>Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM ubuntu:14.04
 ---> 8cef1fa16c77
Step 2/2 : USER blah
 ---> Running in c0f9cb79f0b8
Removing intermediate container c0f9cb79f0b8
 ---> 659e07eb2f37
Successfully built 659e07eb2f37
Successfully tagged test_docker_workdir:latest<jupyter_text>Docker images with `ENTRYPOINT` Some docker images have an entry point which determines the command that will be executed when the image is executed. When such images are executed, parameters passed from command line will be appended to `ENTRYPOINT` so our usually way of specifying an interpreter (e.g. `ruby`) and a script will not work. If we run the script directly, our "command" (e.g. `ruby /var/lib/sos/docker_run_30258.rb` will be appended to the entry point and will not be executed properly. Examples of such images include [`dceoy/gatk`](https://hub.docker.com/r/dceoy/gatk/~/dockerfile/), which has an entry point```["java", "-jar", "/usr/local/src/gatk/build/libs/gatk.jar"]```and does not accept any additional interpreter. What we really need to do is to append "arguments" to this pre-specified command.For example, the `test_docker_ls` image has an `ENTRYPOINT` with command `ls`.<jupyter_code>docker_build: tag='test_docker_ls'
  FROM ubuntu:14.04
  ENTRYPOINT ["ls"]<jupyter_output>Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM ubuntu:14.04
 ---> 8cef1fa16c77
Step 2/2 : ENTRYPOINT ["ls"]
 ---> Running in d1a3a4d6d919
Removing intermediate container d1a3a4d6d919
 ---> a7768bfbb101
Successfully built a7768bfbb101
Successfully tagged test_docker_ls:latest<jupyter_text>The image is expected to be executed directly with optional parameter and without an interpreter (e.g. `docker run test_docker_ls`). Because action `script` does not have a default interpreter, and option `args` can be used to construct a command line, we can use docker images with `ENTRYPOINT` in the format of<jupyter_code>script: args = '-l SoS_Docker_Guide.ipynb', container = 'test_docker_ls'<jupyter_output>-rw-r--r-- 1 1985961928 895809667 28872 Aug 14 16:45 SoS_Docker_Guide.ipynb<jupyter_text>which essentially passes `-l SoS_Docker_Guide.ipynb` to the image and executes command ```ls -l SoS_Docker_Guide.ipynb``` If the command line is long, you can use another trick, that is to say, to use `{script}` in `args` for scripts of the action. For example, the aforementioned command can be specified as<jupyter_code>script: args='{script}', container='test_docker_ls'
  -l SoS_Docker_Guide.ipynb<jupyter_output>-rw-r--r-- 1 1985961928 895809667 28872 Aug 14 16:45 SoS_Docker_Guide.ipynb
{'fraction_non_alphanumeric': 0.08508437318161247, 'fraction_numerical': 0.026702010732527315, 'mean_word_length': 5.0421875, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 79, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '251894', 'license': 'MIT', 'n_tokens_mistral': 5099, 'n_tokens_neox': 4580, 'n_words': 1886, 'path': 'src/user_guide/docker.ipynb', 'repo_name': 'andrewrocks/sos-docs', 'size': '15467'}