The Kubernetes project is freezing the k8s.gcr.io container image registry on 3rd April 2023 and is asking users to migrate to the registry.k8s.io image registry. In addition, the way that the older k8s.gcr.io registry works will change from the 20th of March. In this blog post I’ll explain what you need to know, why this is happening, and which actions you need to take.

What you need to know

The new registry registry.k8s.io has been available and stable for several months and has officially replaced the old registry k8s.gcr.io. New Kubernetes images from v1.27 and patches for older Kubernetes images will only be pushed to the new registry.

The benefits you get from switching are:

• faster image pulls (for clients running inside AWS, image layers will usually be served from S3)
• you avoid depending on a legacy service
• you’re ready for any security update that comes out (once frozen, the old registry won’t have new images even for security updates)

When it’s happening

20th March 2023
From this date, accesses to the old k8s.gcr.io may be redirected to the new registry, registry.k8s.io. This may affect you if you’ve set up specific egress filtering by address range, or a reverse proxy that has a domain allow list and that inspects TLS traffic.

3rd April 2023
From this date container images hosted on the k8s.gcr.io registry will be frozen and no further images will be pushed to the old registry.

However, you can change over at any time - you don’t need to wait for the redirect to be in place, or for the freeze.

Why it’s happening

Historically, Google provided a custom Google Container Registry (GCR) to host the k8s.gcr.io registry. This worked well for many years but it became insufficient with the increased popularity of Kubernetes. In addition, other cloud providers started hosting registries to better serve their users. Both Google and Amazon have donated cloud credits to the CNCF. To accommodate multiple cloud providers, the Kubernetes project set up a new image registry registry.k8s.io that uses multiple layer storage services (including Amazon S3). The design also allows other cloud providers to come on board in the future.

Couldn’t this be implemented with a simple redirect from the old to the new registry? Well, this was trialled in 2022 (see discussions in the Kubernetes Slack channel) but all sort of snags came out with that first attempt and the redirection was considered not fit for purpose.

What you need to do

You can check if your Kubernetes cluster still uses the old image registry via the command line, via policies, or via a plugin as described in the following sections.

Command Line

This command lists all the images used by Pods in your cluster that make use of the old registry:

kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" |\
tr -s '[[:space:]]' '\n' |\
sort |\
uniq -c \
grep "k8s.gcr.io"

You may have other dependencies for other resources (e.g. Jobs, DaemonSets, etc.) so you may need to run a similar command for other resources other than Pods.

Gatekeeper and Kyverno Policies

If you’re using Gatekeeper or Kyverno policies in your cluster, you can use these sample policies to detect container images coming from the old registry. These policies check resources such as Pods, Deployments, DaemonSets, Jobs, StatefulSets, and ReplicaSets.

kubectl plugin

You can install the kubectl community-images plugin in your cluster to display running images from community owned repositories such as k8s.grc.io.

Once you have identified that you’re still using the old registry, you can fix it by taking one of the following actions:

1. Replace k8s.gcr.io with registry.k8s.io in resource manifest files. Also check Helm charts, manually configured add-ons, and so on - the most likely place you’re using Kubernetes container images is for fundamental integrations, such as the official AWS EBS CSI driver.
2. Use a private registry such as ECR, mirror the images from the old registry to your private registry, and pull the images from the private registry. This action is particularly suitable for Kubernetes clusters running in a restricted environment with strict domain and IP address access policies.
3. If you can’t migrate to the new registry or a private registry right now, nodes in your cluster can temporarily pull the images from cache. However, this solution would only give you some additional time to think about a migration strategy as the images will eventually require security updates and become stale. Watch out if you use imagePullPolicy: Always for containers; if you do, the kubelet always tries to check that the image manifest is up to date. Any routine Pod replacement could land you in trouble.

Further details

For more info read the following official Kubernetes announcements:

k8s.gcr.io Image Registry Will Be Frozen From the 3rd of April 2023 (6 February 2023)

registry.k8s.io: faster, cheaper and Generally Available (28 November 2022)

k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know (10 March 2023)

---

Do you need expert advice on Kubernetes? We are a Kubernetes Certified Service Provider and have a wealth of experience with Kubernetes, EKS, and containers. Book a Kubernetes review today.

---

This blog is written exclusively by The Scale Factory team. We do not accept external contributions.