<commit_before>因为我们不提供非Ubuntu的bulid支持，所以如果用户用其他操作系统，比如CoreOS、CentOS、MacOS X、Windows，开发都得在docker里。所以需要能build本地修改后的代码。

我们可能需要两个 Docker images：

1. development image：不包括源码，但是包括开发环境（预先安装好各种工具），也就是说Dockerfile.dev里既不需要  COPY 也不需要 RUN git clone。虽然这个image和源码无关，但是不同版本的源码需要依赖不同的第三方库，所以这个image的tag里还是要包含git branch/tag name，比如叫做 `paddlepaddle/paddle:dev-0.10.0rc1`，这里的0.10.0.rc1是一个branch name，其中rc是release candidate的意思。正是发布之后就成了master branch里的一个tag，叫做0.10.0。

1. production image： 不包括编译环境，也不包括源码，只包括build好的libpaddle.so和必要的Python packages，用于在Kubernetes机群上跑应用的image。比如叫做 `paddlepaddle/paddle:0.10.0rc1`。

从1.生成2.的过程如下：

1. 在本机（host）上开发。假设源码位于 `~/work/paddle`。

1. 用dev image build 我们的源码：
   ```bash
   docker run -it -p 2022:22 -v $PWD:/paddle paddlepaddle/paddle:dev-0.10.0rc1  /paddle/build.sh
   ```  
   注意，这里的 `-v ` 参数把host上的源码目录里的内容映射到了container里的`/paddle` 目录；而container里的 `/paddle/build.sh` 就是源码目录里的 `build.sh`。上述命令调用了本地源码中的 bulid.sh 来build了本地源码，结果在container里的 `/paddle/build` 目录里，也就是本地的源码目录里的 `build` 子目录。

1. 我们希望上述 `build.sh` 脚本在 `build` 子目录里生成一个Dockerfile，使得我们可以运行：
   ```bash
   docker build -t paddle  ./build
   ```
   来生成我们的production image。
   
1. 有了这个production image之后，我们可能会希望docker push 到dockerhub.com的我们自己的名下，然后可以用来启动本地或者远程（Kubernetes）jobs：

   ```bash
   docker tag paddle yiwang/paddle:did-some-change
   docker push
   paddlectl run yiwang/paddle:did-some-change /paddle/demo/mnist/train.py
   ```

   其中 paddlectl 应该是我们自己写的一个脚本，调用kubectl来在Kubernetes机群上启动一个job的。


曾经的讨论背景：   
["PR 1599"](https://github.com/PaddlePaddle/Paddle/pull/1599)  
["PR 1598"](https://github.com/PaddlePaddle/Paddle/pull/1598)<commit_msg>Improve the design doc of Docker build<commit_after>We need to complete the initial draft https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/scripts/docker/README.md.

I am recording some ideas here, and we should file a PR later.

## Current Status

Currently, we have four sets of Dockefiles:

1. Kubernetes examples:

   ```
   doc/howto/usage/k8s/src/Dockerfile -- based on released image but add start.sh
   doc/howto/usage/k8s/src/k8s_data/Dockerfile -- contains only get_data.sh
   doc/howto/usage/k8s/src/k8s_train/Dockerfile -- this duplicates with the first one.
   ```

1. Generate .deb packages:

   ```
   paddle/scripts/deb/build_scripts/Dockerfile -- significantly overlaps with the `docker` directory
   ```

1. In the `docker` directory:

   ```
   paddle/scripts/docker/Dockerfile
   paddle/scripts/docker/Dockerfile.gpu
   ```

1. Document building

   ```
   paddle/scripts/tools/build_docs/Dockerfile -- a subset of above two sets.
   ```

## Goal

We want two Docker images for each version of PaddlePaddle:

1. `paddle:<version>-dev`

   This a development image contains only the development tools.  This standardizes the building tools and procedure.  Users include:

   - developers -- no longer need to install development tools on the host, and can build their current work on the host (development computer).
   - release engineers -- use this to build the official release from certain branch/tag on Github.com.
   - document writers / Website developers -- Our documents are in the source repo in the form of .md/.rst files and comments in source code.  We need tools to extract the information, typeset, and generate Web pages.

   So the development image must contain not only source code building tools, but also documentation tools:

   - gcc/clang
   - nvcc
   - Python
   - sphinx
   - woboq
   - sshd

   where `sshd` makes it easy for developers to have multiple terminals connecting into the container.

1. `paddle:<version>`

   This is the production image, generated using the development image. This image might have multiple variants:

   - GPU/AVX   `paddle:<version>-gpu`
   - GPU/no-AVX  `paddle:<version>-gpu-noavx`
   - no-GPU/AVX  `paddle:<version>`
   - no-GPU/no-AVX  `paddle:<version>-noavx`

   We'd like to give users choices of GPU and no-GPU, because the GPU version image is much larger than then the no-GPU version.

   We'd like to give users choices of AVX and no-AVX, because some cloud providers don't provide AVX-enabled VMs.

## Dockerfile

To realize above goals, we need only one Dockerfile for the development image.  We can put it in the root source directory.

Let us go over our daily development procedure to show how developers can use this file.

1. Check out the source code

   ```bash
   git clone https://github.com/PaddlePaddle/Paddle paddle
   ```

1. Do something

   ```bash
   cd paddle
   git checkout -b my_work
   Edit some files
   ```

1. Build/update the development image (if not yet)

   ```bash
   docker build -t paddle:dev . # Suppose that the Dockerfile is in the root source directory.
   ```

1. Build the source code

   ```bash
   docker run -v $PWD:/paddle -e "GPU=OFF" -e "AVX=ON" -e "TEST=ON" paddle:dev
   ```

   This command maps the source directory on the host into `/paddle` in the container.

   Please be aware that the default entrypoint of `paddle:dev` is a shell script file `build.sh`, which builds the source code, and outputs to `/paddle/build` in the container, which is actually `$PWD/build` on the host.

   `build.sh` doesn't only build binaries, but also generates a `$PWD/build/Dockerfile` file, which can be used to build the production image.  We will talk about it later.

1. Run on the host (Not recommended)

   If the host computer happens to have all dependent libraries and Python runtimes installed, we can now run/test the built program.  But the recommended way is to running in a production image.

1. Run in the development container

   `build.sh` generates binary files and invokes `make install`.  So we can run the built program within the development container.  This is convenient for developers.

1. Build a production image

    On the host, we can use the `$PWD/build/Dockerfile` to generate a production image.

   ```bash
   docker build -t paddle --build-arg "BOOK=ON" -f build/Dockerfile .
   ```

1. Run the Paddle Book

   Once we have the production image, we can run [Paddle Book](http://book.paddlepaddle.org/) chapters in Jupyter Notebooks (if we chose to build them)

   ```bash
   docker run -it paddle
   ```

   Note that the default entrypoint of the production image starts Jupyter server, if we chose to build Paddle Book.

1. Run on Kubernetes

   We can push the production image to a DockerHub server, so developers can run distributed training jobs on the Kuberentes cluster:

   ```bash
   docker tag paddle me/paddle
   docker push
   kubectl ...
   ```

   For end users, we will provide more convinient tools to run distributed jobs.
{'fraction_non_alphanumeric': 0.08769022148561097, 'fraction_numerical': 0.011149615790266687, 'mean_word_length': 3.721194879089616, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5876912', 'n_tokens_mistral': 2440, 'n_tokens_neox': 2371, 'n_words': 754}