Virtual Application Networks (VAN) for Multi-Cloud, Multi-Cluster and Cloud-Edge Interconnect
Modern software development is moving away from client/server toward more flexible architectures with services located everywhere. There are various reasons for deploying or replicating an application in multiple locations: geographically distributed applications for enhanced performance and availability, maintain compliance, connected vehicles, local breakouts in 5G, remote edge sites, etc. This requirement makes application-oriented multi-cloud and multi-cluster connectivity an inevitable trend of cloud computing.
VPN technologies like OpenVPN, Strongswan, Wireguard etc. are widely used to create secure site-to-site connectivity between sites, although VPN is conformant in facilitating connectivity between remote-sites there are caveats and and complexities involved. Some key caveats: VPNs can obfuscate the network path, increased network latency, monitoring the network path through a VPN tunnel is a challenge, as new sites are added the number of site-to-site connections to manage grows exponentially — scaling sites becomes impractical to maintain, cloud-native applications change the topologies based on the advanced scheduling framework where the connectivity has to auto detect the same and lastly the configuration and management of VPN is complex.
Virtual Application Network (VAN) enables advanced communication capabilities between distributed software components in a cloud-native application setting without having to deal with complex layer 3 network controls like VPNs, firewall rules, and access policies.
VAN is an application-layer network that is overlaid on the TCP/IP-based internet. A VAN can be as small as a single laptop or as big as a global network spanning time zones and continents. A VAN can be set up quickly by a non-privileged developer and its topology can be added to and changed just as easily. Applications written to use existing protocols like HTTP, gRPC, AMQP, or anything that runs over TCP or UDP can run on a VAN without modification. VAN addressing uses arbitrary strings to refer to endpoints directly without depending on the underlying host and port information.
The Skupper Project is a free and open-source implementation of a VAN built for Kubernetes. Skupper can enable connectivity between applications running in different clusters with simple configuration without having to worry about elevated privileges, cluster access, or admin access (user just need access to specific namespaces that he is allowed to use). Skupper can cater wide range of use cases, some of them are as shown below.
VAN address is a DNS-style name and references a process
Traditional TCP/IP addressing uses a ‘host:port’ pair to refer to an endpoint, legacy TCP/IP internet is fast and reliable at moving data from one point to another, but it is not well suited to support the modern needs of cloud-native software systems. In the modern setting of containers and serverless computing, the notion of a host for a port adds an extra overhead and cannot scale. Coordinating port allocations across multiple developers or teams that provide containers is very difficult to do at scale, and exposes users to cluster-level issues outside of their control.
VAN addressing follows DNS style naming where the address is a name. The VAN routes application traffic based on the VAN address (simply ‘name’) without regard for the underlying IP addresses that are used by the systems hosting the application. A VAN address references a running process whereas an IP address references the host on which the process runs.
VAN addresses are multi-access
IP addressing is primarily unicast, with each address referring to a single host. IP does have a notion of multicast addressing where IPV4 and IPV6 support multicast and IPV6 supports anycast, but it is not used in practice beyond the scope of a local area network. VAN addressing is either anycast or multicast. It builds in the idea that multiple software components can attach to the network with the same address for multicast delivery or for load balancing. VAN multicast and anycast is not constrained by component location and therefore provides network-wide multicast delivery and anycast load balancing.
Secured site-to-site connections
An application running on a VAN benefits from the security mechanisms inherent to the network. All of the inter-site connections in a VAN are locked down using mutual TLS (Transport Layer Security) with a private, dedicated certificate authority. Access from the outside world only occurs where the developer wishes to provide public access to a portal or web front-end. There is no incidental exposure of services to the internet.
VAN Router — Apache Qpid Dispatch Router
A VAN is established by deploying a VAN router (dispatch router) in each site that will be part of the distributed application namespace and by establishing secure connections between pairs of sites. A site can be a LAN in a data center, a single host, or a namespace in a Kubernetes cluster.
Skupper uses the Apache Qpid Dispatch Router as its VAN router. The router is based on the AMQP protocol and is lightweight and stateless. In addition to the base router, Skupper provides tools to facilitate network setup and to provide mappings from commonly used protocols like HTTP, gRPC, and TCP onto the VAN network.
Dispatch Router is a lightweight AMQP message router for building scalable, available, and performant messaging networks. Dispatch Router flexibly routes messages between any AMQP-enabled endpoints, including clients, servers, and message brokers. AMQP apart from being an open standard protocol that allows messaging interoperability between systems, it is a general-purpose networking protocol that has real advantages over HTTP for service and API delivery.
The network of routers uses Link-State routing protocol, similar to OSPF or IS-IS, to detect topology and compute least-cost paths between each pair of sites. The routing protocol quickly and automatically reacts to changes in the topology. This serves to provide resilience against failure and also makes it easy to add locations, and remove locations from the network as expansion or changes are needed.
In the sample topology below Private Network-A and Private Network-B are two different sites with two different private networks. Client (C1) is in Private Network-B and backend (B2) is in Private Network-A, with dispatch routing client C1 can access queues, topics and other AMQP services that are in Private Network-A using the connectivity established between routers in Private Network-B and Public Network.
For example, The router Ra and Rp can be configured to expose queues in broker B2 and broker B1 to the network. Client C1 can then establish a connection to Rb, the local router, open a subscribing link to “b1.event-queue” and “b2.event-queue”, and receive messages stored on that queue in broker B1 & B2.
In this scenario, neither B1, B2 nor C1 have been modified in any way and neither need to be aware of the fact that there is a message-router network between them.
Skupper — Components
The control-plane components of Skupper include a service-controller and router. Both the components are deployed as deployments in namespace of interest. The router component is actual Apache Qpid Dispatch Router and the service-controller facilitates syncing of exposed services.
The router facilitates connections to other routers across sites and depends on the bridge configuration (configmap which will be programmed when connectors are configured) to retrieve any configured connectors. A connection is established using a token generated and the token is namespace scoped using ‘set-context — current — namespace’.
Service controller tracks information of exposed services. The skupper-services configmap in the namespace is programmed with list of exposed services when the user configures the same.
All router related config is provided as config map to the router deployment. skupper-site comprise site-name and other global configuration parameters. skupper-internal and skupper-services programmed by the controller when a user exposes a service to be available in remote sites, this can be configured using skupper CLI (skupper expose deployment) or providing annotations (skupper.io/proxy and skupper.io/port) to the deployments.
All inter-site connections to other routers in a VAN are locked down using mutual TLS (Transport Layer Security) with a private, dedicated certificate authority provided as secrets.
In public cloud setting the skupper-controller (router) service is exposed as a service-type load balancer with public connectivity as it is required for private sites to peer. In private/edge setting the only requirement is to have egress internet connectivity.
The skupper-controller provides two consoles one being the Apache Qpid Dispatch Console and other is Skupper console (both of these can be activated with skupper-site configmap or passing the parameters with ‘skupper init’). Apache Qpid Dispatch Console provides vast information related to the router and the associated connections along with concrete monitoring and schema information.
The connections section provides list of all router-router connections, local connections and AMQP based connections established with other routers in the network. The topology section provides a graphical view of routers in the network and the message flow information.
Skupper console provides much simple and informative information on the connected sites and detailed graphical view of deployments and services.
Try Out — Creating a VAN between public cloud and edge Sites with private connectivity
The test topology below includes a GKE cluster, EKS cluster and two private clusters bootstrapped using Kubeadm, Skupper just needs a Kubernetes cluster and doesn’t depend on the platform/distro or provisioner.
In the topology below there are four individual K8’s clusters spanning across cloud platforms and networks. GKE and EKS contain skupper-controller exposed as a LoadBalancer and the routers are peered with each other. The other two private clusters (no public connectivity from internet) peer with the router on the GKE cluster. Private clusters can be in different isolated networks, in the example below although both the private clusters are in the same network they are not peered between each other (simulating remote sites with no connectivity between each other).
1. Connecting GKE and EKS
GKE Cluster:
EKS Cluster:
Skupper router and service controller on GKE:
Skupper router and service controller on EKS:
Skupper controller exposed as ‘Type: External Load Balancer’ on GKE:
Connecting routers is authenticated using a namespace scoped connection-token. This can be generated using skupper CLI (skupper connection-token) or by creating a resource — ‘skupper.io/type: connection-token-request’ which will yield the connection-token. In this case, a connection token is created on GKE and used in EKS for establishing the connection.
Once the connection is established the infrastructure AMQP message flow will start between the dispatch routers. Users can use the QPID console to gather various information on the topology, logs and flow information.
EKS peering with GKE:
Message flow visualization between EKS and GKE:
Dispatch router on EKS connecting to skupper-controller service exposed as LoadBalancer on GKE:
Skupper console portal with connected site information:
2. Connecting edge-west and edge-east to GKE
Clusters edge-west and edge-east are private clusters running on Udoo X86 connected to home network.
As shown below the edge-west router peers with the router on the GKE using the public service endpoint. In this case the GKE connection token is used to create the connection between edge-west and gkeuseast.
A SSL profile is created for each connector and skupper-router in the edge sites establish a secured mutual TLS connection with router on GKE:
Skupper console view of connected sites:
QPID topology view:
Qpid console — flow metrics:
Similarly the fourth cluster/site edge-east is also peered with gkeuseast as shown below:
The topology now has a VAN connectivity between two clusters with public connectivity to Skupper service and two private clusters without ingress connectivity from internet. No specific ports were opened and the TLS connections are established between the routers.
Try Out — VAN connectivity between individual Hipster services distributed across above four connected clusters
Hipster Shop is composed of many microservices written in different languages that talk to each other over gRPC. This application deployment topology includes services which are part of the application (frontend, backends and databases) in different namespaces, on different clusters located in different platforms. Technically, this means that they have no way to communicate unless they are exposed to the public internet.
Introducing Skupper into each namespace facilitates creation of a virtual application network that can connect services in different clusters. Any service exposed on the application network is represented as a local service in all of the connected namespaces. When the frontend sends a request to all other the backend entities, Skupper forwards the request to the namespace where the backend is running and routes the response back to the frontend.
Skupper enables users to configure the protocol (tcp, http, or http2. default ‘tcp’) to be used for communication along with the port. In the above example the connectivity of services to the database is ‘tcp’, so users can configure specific protocol by annotating the service object with ‘skupper.io/proxy=tcp and skupper.io/port=6379’.
The frontend service is deployed on gkeuseast (GKE) cluster, there is no need to expose this service to Skupper as this is user facing and will be accessed using the LoadBalancer IP (public_ip) provided by GKE.
Services: cart, checkout, currency and redis-cart are deployed on EKS
Services: productcatalog and recommendations are deployed on edge-west
Services: emailservice, paymentservice and shippingservice are deployed on edge-east
As seen below the hipster application is non-functional as other services in different clusters are not exposed to Skupper yet.
Services productcatalog and recommendations are exposed to VAN on edge-west.
Once the services are exposed, services send requests to skupper-router which are then forwarded to other routers in the network as shown below:
The controller adds the skupper-internal configmap bridges.json and skupper-services configmap file with connector, corresponding address, protocol and port information of the services in the namespace where the specific router is located which will be used by the skupper-router and service-controller for configuring connections.
Skupper router exposes the services with port and siteID information (router-id):
Skupper service controller maintains information of all exposed services:
Services: ads, cart, checkout, currency are exposed on EKS cluster.
Services: email, payment and shipping are exposed on edge-east.
By default all the services are annotated with protocol: tcp, for example the topology above comprise multiple service that communicate over ‘http’ and cart service which is a redis database that needs ‘tcp’ connectivity. Annotations can be added to individual services based on the connectivity required.
Once all the services are exposed to VAN, the app is functional and can now communicate with all other integral services located in different clusters on different platforms. With VAN, services in private clusters are accessible for other services located in public and other remote private clusters.
Skupper console deployments tab showing all the deployments across the member sites:
Graphical view of deployments and sites with service connectivity information:
Monitoring traffic flow between services :
As shown above ‘10.48.2.6’ is the pod_ip of the frontend service located in GKE cluster. With VAN the pod can communicate to other pods (services) in different clusters and network spaces without any site-site VPN connectivity.
Qpid console provides detailed information on the service connections :
Skupper — graphical view of services with health, flow and protocol information :
Skupper console — graphical view of message flow between sites and services :
Qpid console provides flow level monitoring and live traffic flow :
Redundant paths in the topology provide resiliency against failure of a site or a connection. It is ok for multiple private networks to participate even if they have overlapping CIDR subnetworks and there is no central controller or single point of failure. For example in the topology used, a connection can be made from two edge clusters to EKS cluster for redundancy as shown below :
Topology of two edge clusters peered with routers on both EKS and GKE :
More and more applications and frameworks are no longer directly tied to any cloud by design and are not constrained to be deployed in a single cluster. With VAN the developer of a cloud-native application does not want to have to think about the hosts or the ports involved in running the application services. VAN can cater a wide set of complex use-cases with a simpler approach and configuration.
