Amazon EKS Distro (EKS-D): The Kubernetes Distribution Used by Amazon EKS
Amazon at the re:Invent 2020 virtual event made strategic announcements related to container services, the announcement include EKS-A (EKS Anywhere), EKS-C (EKS Console), ECS-A (ECS Anywhere) and EKS-D (EKS Distro). While EKS-A, EKS-C, ECS-A are expected to be available in 2021, AWS open sourced EKS-D — Kubernetes distribution that powers its Elastic Kubernetes Service (EKS).
A Kubernetes distribution is Kubernetes with a chosen configuration and a chosen set of addons. There are multiple Kubernetes distributions like Rancher Kubernetes, VMware Tanzu Kubernetes Grid, Charmed Kubernetes from Canonical, Red Hat OpenShift etc. available upstream. Kubernetes is not secure by default, nor by itself, a security-focused Kubernetes distribution will configure Kubernetes with restrictive security policies which are mission critical for production environments.
EKS Distro (EKS-D) combines open-source (upstream) Kubernetes components — Kubernetes control plane components (such as kube-controller-manager, etcd, and CoreDNS), Kubernetes worker node components (kubelet, Kubernetes CSI, and CNI), command-line clients (kubectl and etcdctl), security patches, and third-party tools including configuration database, network, and storage components necessary for cluster creation. It will also include all of the upstream patches used by Amazon EKS including fixes to which Amazon has contributed back to the community. In addition, it will include patches that AWS deem important for operational stability and security fixes.
With EKS-D, AWS has built an open source bridge to its managed Kubernetes platform and a medium to drive the adoption of its cloud platform. With EKS-D, users can rely on the same versions of Kubernetes and its dependencies deployed on Amazon EKS and other cloud platforms (public/private).
EKS-D reduces the friction between using a different Kubernetes distribution for on-prem and cloud-based environments by using the same validated versions and configurations of Kubernetes and its dependencies, as deployed globally by Amazon EKS. Since it’s free, customers are more likely to evaluate it before considering other Kubernetes distros and enables AWS to efficiently compete with independent software vendors like VMware, Red Hat, Rancher etc. which leverage Amazon EC2 to run their managed Kubernetes offering.
EKS vs EKS-D
EKS — In Amazon EKS on Amazon EC2 setting a managed control plane — control plane runs in an account managed by AWS, and the Kubernetes API is exposed via the Amazon EKS endpoint associated with your cluster. Each Amazon EKS cluster control plane is single-tenant and unique, and runs on its own set of Amazon EC2 instances. Users can opt for a self-managed data plane or AWS managed dataplane using managed node groups.
Amazon EKS nodes run in users AWS account and connect to cluster’s control plane via the API server endpoint and a certificate file that is created for the cluster. Cluster creation (bootstrap) is seamless and constitutes multiple entities created by a AWS CloudFormation stack. Users can pick any of the supported versions to create the cluster.
Users can only access the workload components, daemonsets deployed on the nodes and all the control plane components cannot be accessed.
EKS-D — All components are customer managed. Bootstrapping of cluster can be done using any of the supported installation methods. Already a number of partners are providing install methods as well a integrations with EKS Distro. Tooling required for lifecycle management and other aspects such as provisioning the nodes, HA control plane, fail over etc. should be taken care by the user.
EKS-D can be used by organizations that wish to standardize and use the exact same Kubernetes distribution as they use with Amazon EKS on their non-AWS environments as well.
Users can access all the components (control plane and other workloads), this enables users to have control over the core control plane components.
Release Channels and Releases
Each Amazon EKS Distro release follows the EKS process (supports at least four production-ready versions of Kubernetes at any given time), verifying new Kubernetes versions for compatibility. The Amazon EKS Distro source code, open-source tooling, binaries, and container images as well as configuration are provided for reproducible builds via public Git and S3 storage locations.
The EKS Distro releases Kubernetes versions at the same pace as EKS, and updates are issued as releases in release channels. A release channel tracks minor versions (v<major>.<minor>.<point>) of Kubernetes, and a channel will be retired when EKS ceases supporting a particular minor version of Kubernetes.
New releases and release channels will be announced via an SNS topic when they are launched. Releases and release channels are structured as Kubernetes Custom Resource Definitions (CRDs) and the schema can be found in the eks-distro-build-tooling GitHub repository.
Release Channel and Releases — Custom Resource Definitions (CRD’s)
Release Channel spec with snsTopicARN (Amazon Resource Names)
Releases of the EKS Distro are in-step with versions of individual components (CNI plugins, CoreDNS, etcd etc.) used by or recommended for use with Amazon EKS beginning with Kubernetes v1.18.9. A list of all the components and assets that make up a release including URIs to all the compressed archives, binaries, and container images are available in the release manifests.
Release spec with list of individual components and the corresponding image versions:
All container images for these components are based on Amazon Linux 2, and are available for users on the ECR public registry for amd64 and arm64 architectures or users can build the images from scratch using the tooling provided. New releases will be created when there is an updated component version.
The EKS Distro is built using Prow, the Kubernetes CI/CD system. EKS operates an installation of Prow which manages all the build operations.
Container Images
Users can use Amazon’s public ECR repository to pull the images of specific release. The repository includes all the core control plane components and other add-ons.
All images are tagged with the components individual version and Kubernetes release version as shown below:
Users can build all the images from scratch using the base image provided in EKS Distro Build Tooling repository (amazonlinux:2) and all container images using EKS Distro repository. The default target for make files uses buildkit to build containers. The alternative to this is building and pushing containers with Docker.
An up-to-date Amazon Linux 2 base image can be created using the tooling repo and pushed to ECR. This base will be updated whenever there are any security updates to RPMs contained in the base image.
All images built can be pushed to private ECR repository and can be used with docker using ECR Credential Helper.
Individual private repositories in ECR (sample):
The tooling tags the release version and component version:
ECR auto scans (configuration) all the images pushed for any software vulnerabilities in container images.
All the images are built using amazon-linux as base and include all the patches that AWS deem important for operational stability and security fixes, most of the vulnerabilities are fixed in EKS-D when compared with the official ubuntu based upstream Kubernetes images.
To test, a vulnerability scan is done below using Snyk on kube-proxy image from upstream (k8s.gcr.io/kube-proxy) and EKS-D (public.ecr.aws/eks-distro/kubernetes/kube-proxy).
The scan below shows that most of the vulnerabilities are fixed in EKS-D based kube-proxy image:
k8s.gcr.io/kube-proxy:v1.18.9 :
public.ecr.aws/eks-distro/kubernetes/kube-proxy:v1.18.9-eks-1–18–1 :
Tryout — Bootstrapping a cluster with EKS Distro
EKS Distro can be installed using any of the supported installation methods like Rancher, Kubeone, Canonical, Kubestack etc., partners also provide services such as monitoring (datadog, instana), security (alicide, aqua security), GitOps (weaveworks) etc.
The official development guide from the eks-distro repository uses Kops for bootstrapping a cluster. Technically EKS-D can be installed using tools like Kubeadm as this just involves modifications of images (use EKS-D based images from public ECR) used and addition of specific flags/parameters in the manifests derived from upstream. There are other easier ways like EKS Snap to quickly get a feel of EKS-D. Other partners like Rancher are already supporting production grade EKS-D with RKE-2 (RKE Government) and many others providing installation methods, monitoring and security.
EKS Snap — Ubuntu
With EKS Snap, a full EKS experience now comes in a single, immutable package with automatic updates, for low-maintenance and increased security.
The eks snap packages all the Kubernetes binaries of Amazon EKS Distro (EKS-D) and combines them with a MicroK8s like experience, for an opinionated, self-healing, highly available EKS-compatible Kubernetes anywhere.
This EKS Distro is based on MicroK8s. All the control plane components run as systemd services.
The installation method provides multiple commands to manage addition and deletion of nodes, manage certs, manage db etc. As shown below the snap install automatically installs and initializes multiple services.
Kops
Kops, short for Kubernetes Operations, is a set of tools for installing, operating, and deleting Kubernetes clusters in the cloud. As of today the official development guide for starting a EKS-D cluster is with Kops. Kops framework provided in the eks-distro repository uses EKS-D for bootstrapping the cluster.
Kops creates required EC2 instances and bootstraps a Kubernetes cluster using EKS-D.
As shown below all control-plane components can be accessed by the user.
All the images are derived from the eks-distro:
Kops needs a “state store” to store configuration information of the cluster, Kops uses S3 as a state store. In this scenario it stores all the information such as cluster-spec, manifests, pki etc.,The state is stored during the initial cluster creation. Any subsequent changes to the cluster are also persisted to this store as well.
EKS-D is an investment from Amazon to reduce the friction involved in adopting AWS when using a commercial Kubernetes distribution. Apart from being just a Kubernetes distribution EKS-D lay a strong foundation for future hybrid cloud and multi-cloud services expected from AWS.
With EKS-A coming in 2021 EKS-D will play a vital role in hybrid cloud setting where users can have the same distro of Kubernetes everywhere with optional connectivity to other AWS services. EKS-A will eventually become the de facto compute environment for multi-cloud, hybrid-cloud and edge (like snowball) environments enabling AWS to efficiently compete with homologous products such as Google Anthos and Azure Arc which are already prevalent in the market.
EKS-D’s upstream alignment demonstrates a continued commitment to the Kubernetes open source community from AWS and also brings additional engineering strength and ideas to the table. This announcement clearly reinforces that enterprises will run Kubernetes as an abstraction across on-premises and public cloud.
