Community
Participate
Working Groups
Docker recently posted an announcement [1][2] stating that they will set rate limits on images pull starting Nov 1st. On the build cluster, all agent images pull from docker.io are done anonymously (including default agent images, aka jnlp, basic, centos, migration...) from a limited number of IPs. As such, we will hit the limits very fast, very often. This bug is to track the effort to work around the new limits. We will come up shortly with a detailed plan and we will reach out to projects who are using custom agent images. To meet with the November 1st deadline, general idea is to leverage a different service with higher or no rate limits (e.g. quay.io). More on that later. [1] https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/ [2] https://www.docker.com/pricing/resource-consumption-updates
FWIW - CDT has been using quay.io since the beginning of JIRO and it has seemed to work great.
tools.orbit and linuxtools could probably migrate to using quay.io as well. They both use custom images based off of eclipsecbi/fedora-gtk3-mutter .
If the only issue is the pull limit, wouldn't it be more profitable for the Foundation to buy and use a Docker Pro account than to ask all projects to move to quay.io (until quay also sets similar restrictions...) ? Another question is about the inactive image limit: does JIRO setting run a docker pull of docker image on everyrun, or at least frequently enough to keep images active?
(In reply to Mickael Istria from comment #3) > If the only issue is the pull limit, wouldn't it be more profitable for the > Foundation to buy and use a Docker Pro account than to ask all projects to > move to quay.io (until quay also sets similar restrictions...) ? Yes, that is another possibility. We're still investigating all options, e.g., we are also investigating setting up a local mirror. Note that the move of default images (basic, migration, centos...) to quay.io will be transparent for 98% of the projects (those who are using freestyle jobs or pipeline jobs with "agent any" or "agent { label "something"}". > Another question is about the inactive image limit: does JIRO setting run a > docker pull of docker image on everyrun, or at least frequently enough to > keep images active? It's up to each project using custom images to be careful about that. The Kubernetes jenkins plugins has a default image pull policy set to IfNotPresent. AFAICT, we cannot change this default. I means that as soon as the image has been pulled on all cluster nodes, it won't be pulled again until the image reference is updated in the pipeline. To avoid this situation, projects have to either set the imagePullPolicy to true in the yaml definition of your pod, or to use the alwaysPullImage option in the pod template configuration (see https://github.com/jenkinsci/kubernetes-plugin#kubernetes-plugin-for-jenkins for details). For the provided agents image, we build and push them every 3 days to ensure that they always contain latest security fixes, so those won't go away any time soon.
Any updates on any potential workarounds? We haven't yet taken any action to move custom (Java JDK 8 + JDK 11) images used on https://ci.eclipse.org/jakartaee-tck to quay.io. We do have the Jakarta EE 9 release that we are getting ready for and do not want any slow down in our CI test performance.
@Scott, thanks for reaching out to us. We are working on deploying docker pro accounts on the cluster. This will relax the time constraint from docker.com's short notice and and will let us serenely define a plan about how and when to proceed migration to a new container registry service. I will comment here once the pro accounts will be active. Note that it will be transparent to projects, you won't have to do anything.
We've started to roll out image pull credentials. It will complete by tomorrow EOB. It does not require any intervention from projects and does not interrupt any running build.
Roll out is complete. We're ready for the new pull rates policy (starting Nov 1st). Thanks everybody!
👍
