<story><title>Docker to rate limit image pulls</title><url>https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/</url></story><parent_chain><item><author>bsamuels</author><text>There is very little reason for a build node to need to pull 200 images in 6 hours, and here is why:&lt;p&gt;When a machine issues a ``docker build`` command, the program reads the relevant dockerfile to check for any base images that need to be pulled (a la &amp;quot;FROM:&amp;quot;)&lt;p&gt;These base images are identified based on the image repository, image name, and image tag. The first thing docker does is it checks its local registry and tries to find a match for the base image the docker build is requesting. If a matching image is located in the local registry, it uses that one in lieu of downloading the image.&lt;p&gt;This is significant - if your organization only uses a few dozen base images from DockerHub, those images will only be downloaded by each build node _once_, then never again.&lt;p&gt;Many docker users erroneously believe that if their Dockerfile requests a &amp;quot;latest&amp;quot; tagged image, docker build will always download the newest version of the image. 
However, the &amp;quot;latest&amp;quot; tag is literally just a tag, it doesn&amp;#x27;t have any special functionality built in. If the docker build command finds an image tagged &amp;quot;latest&amp;quot; in the local registry, it stops there.&lt;p&gt;The only way to get docker build to always use the &amp;quot;actual latest&amp;quot; version of the base image is to add the &amp;quot;--pull&amp;quot; parameter to the docker build command. This arg will tell docker build to check the repository remote to see if the SHA hash of the image tagged &amp;quot;latest&amp;quot; has changed, and if so, re-download and use it. In the absolute worst case, this means each build node will pull 1 copy of each base image when the base image is updated. So unless you use 200 different base images that all have updates deployed to Dockerhub each and every day, you are fine.</text></item></parent_chain><comment><author>nrmitchi</author><text>I don&amp;#x27;t disagree with what you are saying, _but_:&lt;p&gt;&amp;gt; Docker defines pull rate limits as the number of manifest requests to Docker Hub.&lt;p&gt;&amp;gt; For example, if you already have the image, the Docker Engine client will issue a manifest request, realize it has all of the referenced layers based on the returned manifest, and stop. ... &amp;lt;excluded&amp;gt; ... So an image pull is actually one or two manifest requests,&lt;p&gt;This still implies that even if you are appropriately re-using layers on your machine, with a free plan you can only do maximum 200 builds (since docker still needs to verify it has the image) per 6 hours?&lt;p&gt;This change also seems to imply that builds steps which previously did not handle&amp;#x2F;require authentication against Docker hub (it was only pulling public images, and pushing elsewhere) will now be required to auth against docker hub in order to double the number of pulls&amp;#x2F;checks&amp;#x2F;builds it is allowed?</text></comment>
<story><title>Docker to rate limit image pulls</title><url>https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/</url></story><parent_chain><item><author>bsamuels</author><text>There is very little reason for a build node to need to pull 200 images in 6 hours, and here is why:&lt;p&gt;When a machine issues a ``docker build`` command, the program reads the relevant dockerfile to check for any base images that need to be pulled (a la &amp;quot;FROM:&amp;quot;)&lt;p&gt;These base images are identified based on the image repository, image name, and image tag. The first thing docker does is it checks its local registry and tries to find a match for the base image the docker build is requesting. If a matching image is located in the local registry, it uses that one in lieu of downloading the image.&lt;p&gt;This is significant - if your organization only uses a few dozen base images from DockerHub, those images will only be downloaded by each build node _once_, then never again.&lt;p&gt;Many docker users erroneously believe that if their Dockerfile requests a &amp;quot;latest&amp;quot; tagged image, docker build will always download the newest version of the image. 
However, the &amp;quot;latest&amp;quot; tag is literally just a tag, it doesn&amp;#x27;t have any special functionality built in. If the docker build command finds an image tagged &amp;quot;latest&amp;quot; in the local registry, it stops there.&lt;p&gt;The only way to get docker build to always use the &amp;quot;actual latest&amp;quot; version of the base image is to add the &amp;quot;--pull&amp;quot; parameter to the docker build command. This arg will tell docker build to check the repository remote to see if the SHA hash of the image tagged &amp;quot;latest&amp;quot; has changed, and if so, re-download and use it. In the absolute worst case, this means each build node will pull 1 copy of each base image when the base image is updated. So unless you use 200 different base images that all have updates deployed to Dockerhub each and every day, you are fine.</text></item></parent_chain><comment><author>solatic</author><text>&amp;gt; This is significant - if your organization only uses a few dozen base images from DockerHub, those images will only be downloaded by each build node _once_, then never again.&lt;p&gt;You&amp;#x27;re assuming that the set of build nodes is relatively static.&lt;p&gt;Plenty of architectures set up autoscaling for the underlying nodes, that terminate servers that aren&amp;#x27;t being used and relatively soon enough (tens of minutes, hours) spin up new servers to replace them as needed.&lt;p&gt;Rarely do the machine images used to spin up new servers include the base images of the containers that will be spun up to replace them. Much more often, the base machine image is a base OS image, and container images are downloaded on-the-fly as needed. Essentially, the engineering cost of making image-launching more efficient was externalized onto an external provider willing to pay the price.</text></comment>