On Aug. 24, 2020, Docker announced changes to its subscription model and a move to consumption-based limits. These rate limits for pulls of Docker container images go into effect on Nov. 1, 2020. For pull requests by anonymous users, this limit is now 100 pull requests per six hours; authenticated users have a limit of 200 pull requests per six hours.
As members of the global DevOps community, we have all come to rely on Docker as an integral part of CI/CD processes. So it is with no surprise that at GitLab, we have heard from several community members and customers seeking guidance on how the Docker pull rate limit change may affect their environments and CI/CD workflows.
Use a registry mirror
You can use the Registry Mirror feature to the number of image pull requests generated against Docker Hub. When the mirror is configured and GitLab Runner instructs Docker to pull images, Docker will check the mirror first; if it's the first time the image is being pulled, a connection will be made to Docker Hub. Subsequent requests of that image will then use your mirror storage instead of connecting to Docker Hub. More details on how it works can be found here.
If you are a user or customer on GitLab SaaS
For Shared Runners on GitLab.com we utilize Google's Docker Hub images mirror. This means that GitLab.com Shared Runner users' CI jobs won't be affected by the new pull policy. We will continue to monitor the impact of the changes once they go into effect at Docker.
If you self-host GitLab Runners
First of all, check if your cloud or hosting provider doesn't already provide an image Registry Mirror. If they do, it will be probably the easiest and most performant option. If for any reason a hosted Registry Mirror can't be used the administrator can install their own Docker Hub mirror.
Start the registry mirror
Please follow the instructions in the GitLab documentation:
-
Log in to a dedicated machine where the container registry proxy will be running
-
Make sure that Docker Engine is installed on that machine
-
Create a new container registry:
docker run -d -p 6000:5000 \ -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \ --restart always \ --name registry registry:2
You can modify the port number (
6000
) to expose the registry on a different port. This will start the server withhttp
. If you want to turn on TLS (https
) follow the official documentation. -
Check the IP address of the server:
hostname --ip-address
You should preferably choose the private networking IP address. The private networking is usually the fastest solution for internal communication between machines of a single provider (DigitalOcean, AWS, Azure, etc.) Typically the use of a private network is not accounted for in your monthly bandwidth limit.
-
Docker registry will be accessible under
MY_REGISTRY_IP:6000
Configure Docker to use it
The final part is to have the dockerd
process configured so that it
uses your mirror when docker pull
runs.
Docker
Either pass the --registry-mirror
option when starting the Docker
daemon dockerd
manually, or edit /etc/docker/daemon.json
and add the
registry-mirrors
key and value, to make the change persistent.
{
"registry-mirrors": ["http://registry-mirror.example.com"]
}
docker+machine
executor
Update the GitLab Runner configuration file config.toml
to specify
engine-registry-mirror
inside of MachineOptions
settings.
Docker-in-Docker to build Docker images
There are different ways to achieve this, and it depends on your configuration. An extensive list can be found in our documentation.
Verify that it is working
Make sure that Docker is configured to use the mirror
If you run docker info
where dockerd
is configured to use the mirror
you should see the following in the output:
...
Registry Mirrors:
http://registry-mirror.example.com
...
Check registry catalog
The Docker registry API can show you which repository it has cached locally.
Given that we ran docker pull node
for the first time with dockerd
configured to use the mirror we can see it by listing the
repositories.
curl http://registry-mirror.example.com/v2/_catalog
{"repositories":["library/node"]}
Check registry logs
When you are pulling images you should see logs coming through about the
request information by running docker logs registry
, where registry
is the name of the container running the mirror.
...
time="2020-10-30T14:02:13.488906601Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="192.168.1.79:6000" http.request.id=8e2bfd60-db3f-49a3-a18f-94092aefddf9 http.request.method=GET http.request.remoteaddr="172.17.0.1:57152" http.request.uri="/v2/library/node/blobs/sha256:8c322550c0ed629d78d29d5c56e9f980f1a35b5f5892644848cd35cd5abed9f4" http.request.useragent="docker/19.03.13 go/go1.13.15 git-commit/4484c46d9d kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.13 \(darwin\))" http.response.contenttype="application/octet-stream" http.response.duration=6.344575711s http.response.status=200 http.response.written=34803188
172.17.0.1 - - [30/Oct/2020:14:02:07 +0000] "GET /v2/library/node/blobs/sha256:8c322550c0ed629d78d29d5c56e9f980f1a35b5f5892644848cd35cd5abed9f4 HTTP/1.1" 200 34803188 "" "docker/19.03.13 go/go1.13.15 git-commit/4484c46d9d kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.13 \\(darwin\\))"
time="2020-10-30T14:02:13.635694443Z" level=info msg="Adding new scheduler entry for library/node@sha256:8c322550c0ed629d78d29d5c56e9f980f1a35b5f5892644848cd35cd5abed9f4 with ttl=167h59m59.999996574s" go.version=go1.11.2 instance.id=f49c8505-e91b-4089-a746-100de0adaa08 service=registry version=v2.7.1
172.17.0.1 - - [30/Oct/2020:14:02:25 +0000] "GET /v2/_catalog HTTP/1.1" 200 34 "" "curl/7.64.1"
time="2020-10-30T14:02:25.954586396Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="127.0.0.1:6000" http.request.id=f9698414-e22c-4d26-8ef5-c24d0923b18b http.request.method=GET http.request.remoteaddr="172.17.0.1:57186" http.request.uri="/v2/_catalog" http.request.useragent="curl/7.64.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.117686ms http.response.status=200 http.response.written=34
Alternatives to Docker Hub mirrors
Setting up a Docker registry mirror can also increase your infrastructure costs. With the Docker Hub rate limits it might be useful to authenticate pulls instead of having your rate limit increased or no rate limit at all (depending on your subscription).
There are different ways to authenticate with Docker Hub on Gitlab CI and it's documented in detail. A few examples:
DOCKER_AUTH_CONFIG
variable provided.config.json
file placed in$HOME/.docker
directory of the user running GitLab Runner process.- Run
docker login
if you are using Docker-in-Docker workflow.
In summary
As you can see, there are several ways you can adapt to the new Docker Hub rate limits and we encourage our users to choose the right one for their organization's needs. Along with the options described in this post, there's also the option of staying within the GitLab ecosystem using the GitLab Container Proxy, which will soon be available to Core users.