KubeEdge — Extending Kubernetes to Edge
Open source edge computing is experiencing the most dynamic development phase in its industry. So many open source platforms, so much integration and so many standardization initiatives! In the Internet of Things (IoT) era, billions of sensors and actuators are deployed worldwide. To manage IoT devices and use cloud computing resources to process data from them all the major cloud providers have their own services like AWS GreenGrass, Azure IoT, Google IoT solutions etc. each cloud provider have their own implementation of deploying to edge (AWS Lambda, Azure Edge Hub).
The IoT platform uses a publish/subscribe broker such as MQTT or AMQP to handle communication channels between IoT devices and cloud services where Pub/Sub semantic-based central cloud edge node communication is achieved through similar channels designed for IoT devices.
KubeEdge was born out of Huawei Cloud IEF service and is the first open source project in the field of edge computing with production environment deployment capability. The previous version (0.1) was made to work with Huawei’s cloud IEF service, the v0.2 open-sourced all the cloud code enabling cloud-providers to integrate the service within their platforms eliminating user concerns about vendor binding.
Mosquitto (MQTT) and KubeEdge
KubeEdge uses MQTT (Message Queuing Telemetry Transport) for communication between deviceTwin, devices/apps, machine-to-machine communication and duplex communication between the edge and the control plane. MQTT is a lightweight messaging protocol that uses publish/subscribe operations to exchange data between clients and the server. Furthermore, its small size, low power usage, minimized data packets and ease of implementation make the protocol ideal of the “machine-to-machine” or “Internet of Things” arena.
There are multiple MQTT brokers available upstream like Mosquitto, HiveMQ, Mosca etc. while KubeEdge uses Mosquitto (https://mosquitto.org/) from Eclipse as a MQTT broker.
EventBus of Kubeedge (interface for sending/receiving messages on mqtt topics) can be started in multiple MQTT modes (internal, external and both) and acts as an interface for sending/receiving messages on relevant MQTT topics. Once EventBus is started the same subscribes to the topics below.
1. "$hw/events/node/+/membership/get": get membership details of a node i.e the devices that are associated with the node
2. "$hw/events/device/+/state/update": update the state of the device
3. "$hw/events/device/+/twin/+": deviceID on whose twin the operation is to be performedThe device twin (cloud-based digital representation of a physical device) feature enables users to take advantage of the collection, transformation and normalization of different formats of remote edge-host data into a single logical model. If the the message is received on the above topics, the message is sent to deviceTwin, else the message is sent to cloud via edgeHub.
Components and Modules
KubeEdge comprise two core components Cloud and Edge with multiple sub-modules. KubeEdge uses Beehive, a messaging framework based on go-channels for communication between the different modules. The control plane typically resides in cloud, though scalable and extendable. At the same time, the edge can work in offline mode. Also it is lightweight and containerized, and can support heterogeneous hardware at the edge.
KubeEdge enables users to host the Cloud and Edge components as a deployment on Kubernetes or Docker (Only Edge) and also as a binary which can be cross-compiled between platforms.
Tryout
A model cluster below depicting an edge scenario constitute a single node Kubernetes cluster (master+worker) called Kubeedge which hosts Kubernetes control plane and the KubeEdge-Cloud module. The two edge sites namely ‘edge01’ and ‘edge02’ comprise KubeEdge-EdgeCore modules with only Docker installed on them. All the three machines run Mosquitto in external mode.
Workflow
Initiating Cloud — EdgeController
On the ‘Kubeedge’ Kubernetes cluster KubeEdge-Cloud module is initiated as a binary with the configuration below. The modules include a Controller: bridge between Kubernetes Api-Server and edgecore and a CloudHub: mediator between EdgeController and the Edge side.
Registering a Edge Node
All Edge Nodes can be registered/added using Kubernetes native Node spec as shown below:
{
"kind": "Node",
"apiVersion": "v1",
"metadata": {
"name": "edge01",
"labels": {
"name": "edge-node"
}
}
}Initiating Edge — EdgeCore
On the two edge-sites above the edge module is initiated. Both the hosts have Docker installed on them (no Kubernetes) and Mosquitto for mqtt. The modules include EdgeD: manages pod lifecycle and deploy on edge nodes, EventBus: interface for sending/receiving messages on mqtt topics, MetaManager: message processor between edged and edgehub, EdgeHub: interacts with a cloud service like Huawei Cloud IEF service, and a DeviceTwin with sub modules like membership module, communication module, device module and device twin module responsible for storing device status, dealing with device attributes, handling device twin operations etc.
Once the Edge is initialized the MQTT broker interacts with the KubeEdge-Cloud on Kubernetes cluster to subscribe to events produced by the KubeEdge-Cloud-Controller. As seen below there will be an initiation of pub/sub with the CloudHub module on the EdgeController:
EdgeController with its ‘Downstream Controller’ watches K8S-Api-server and send updates to EdgeCore via CloudHub and the ‘Upstream Controller’ receives messages from EdgeCore and sync the updates to K8S-Api-server.
MetaManager on EdgeCore processes messages from Edged to EdgeHub. EdgeHub communicates the same to the Cloud services like Huawei Cloud IEF (which has an integration as of today). EdgeCore module on the edge-site starts communicating with the CloudHub on the controller and starts sending sync messages to the upstream controller module which delivers the same to Kubernetes API Server:
After successful connectivity and exchange of upstream & downstream topics the ‘edge01’ will be registered as a node and becomes ready.
Deploying to Edge Sites
Deploying a sample Nginx deployment on Kubernetes cluster results in creating a Docker container on the edge-site. If the user is using HuaweiCloud (IEF) then there will be a creation of Kube-Proxy instance on the edge-site which ties the edge-application to the central Kubernetes cluster running EdgeController where users can manage the whole life-cycle of the application using native Kubernetes primitives. As there is no middleware/agent in this scenario although a Kube-Proxy container is created along with the application on the edge-site that may not be fully functional.
When a sample Nginx deployment is created on the Kubernetes Cluster with node-selector choosing the edge-site2 in the example below. The EdgeController — CloudHub starts to send a event to EdgeCore on edge-site-2:
EdgeCore through MQTT receives the event generated above as the same is configured to subscribe for the events produced by EdgeController:
Docker CRI in the edge-site initiates pulling the image and creates the container (nginx):
As seen below the docker container created on the edge-site consists of the Kubernetes labels as the same was deployed in the same fashion as a pod on Kubernetes.
As of today, Kubeedge is still not adapted out of Huawei Cloud IEF (provides integrated edge-cloud synergy services). The networking becomes a bottleneck where connecting the Kubernetes cluster with the edge applications is an issue, for example Azure Virtual Kubelet implementation on AKS was able to provide networking for edge-sites using its advanced networking module, in this scenario the networking cannot be possible without another virtual-router or something like Bird.
With open edge and cloud modules, KubeEdge brings a complete cloud vendor-neutral, lightweight, heterogeneous edge computing platform. It is now ready to support building a complete Kubernetes ecosystem for edge computing and leveraging most existing cloud native projects or software modules. This allows KubeEdge to support demanding use cases such as data analysis, video analytics machine learning and more. As Kubeedge is totally not vendor neutral yet and still evolving as a unified control plane for edge computing , It’s success vastly depends on the adoption by mainstream cloud providers including Amazon, Google, and Microsoft.
