Go Crypto and Kubernetes — FIPS 140–2 and FedRAMP Compliance
Encryption is key to keeping sensitive data protected. However, there are a number of algorithms available as well as varying capabilities for providing encryption. As a result, it is inevitably challenging to know which algorithm and security standard to use. To establish standards vendors can rely on, the U.S. government and its National Institute of Standards (NIST) have established FIPS 140–2. FIPS stands for Federal Information Processing Standard, and the FIPS-140 series is a collection of computer security standards set by the National Institute of Standards & Technology (NIST) for the United States government. FIPS 140–2 defines the critical security parameters vendors must use for encryption before selling into the U.S government.
If a vendor wants to supply cloud-based services to the US Federal Government, then they have to get FedRAMP approval. This certification process covers a whole host of security issues, but is very specific about its requirements on cryptography: usage of FIPS 140–2 validated modules wherever cryptography is needed, these encryption standards protect the cryptographic module from being cracked, altered, or otherwise tampered with. FIPS 140–2 validated encryption is a prerequisite for FedRAMP.
Most of the public cloud vendors already have GovCloud (AWS, Azure, Oracle, GCP, etc.) which are isolated from general cloud setting and designed to allow customers and the U.S government agencies to move their confidential data into the cloud to address their compliance and specific regulatory requirements (ITAR regulations, the FedRAMP requirements, Defense Federal Acquisition Regulation Supplement (DFARS), etc.).
Managed Kubernetes service like Amazon Elastic Kubernetes Service (EKS) recently launched in AWS GovCloud (US), Azure Kubernetes Service (AKS) now available in Azure Government enabling public sector customers with FedRAMP, DoD CC SRG, ITAR, etc to run fully-managed Kubernetes clusters. AWS and other providers facilitate FIPS endpoints for all supported services in AWS GovCloud (US) which operate using FIPS 140–2 validated cryptographic modules.
Several operating system vendors have obtained FIPS 140–2 Level 1 validation for the OpenSSL Cryptographic Module shipped with their respective operating systems: Red Hat, Inc.: Red Hat Enterprise Linux OpenSSL Cryptographic Module, Canonical Ltd.: Ubuntu OpenSSL Cryptographic Module, Amazon Linux: Amazon Linux Cryptographic Modules, Oracle Corporation: Oracle Linux OpenSSL Cryptographic Module. Even if the vendor supports using a Linux OS with FIPS enabled crypto, the components that make up the Kubernetes platform should be able to take advantage of the FIPS validated crypto without bypassing them or the whole distribution should be compiled with a custom compiler that uses FIPS validated cryptography.
Go Cryptography and Kubernetes — FIPS 140–2
Kubernetes is a Go project, as are most of the Kubernetes subcomponents and ecosystem. Golang has a crypto standard library, Golang Crypto which fulfills almost all the application crypto needs (TLS stack implementation for HTTPS servers and clients all the way to HMAC or any other primitive that are needed to make signatures to verify hashes, encrypt messages.). Go has made a different choice compared to most languages, which usually come with links or wrappers for OpenSSL or simply don’t provide any cryptography in the standard library (Rust doesn’t have standard library cryptography, JavaScript only has web crypto, Python doesn’t come with a crypto standard library).
In a Kubernetes cluster apart from user to kube-apiserver interactions, the core components and other components of the stack such as etcd, contained, etc. work collectively and need to communicate with each other. In order to ensure that communication is kept private, not interfered with, and ensure that each component of the cluster is talking to another trusted component, where TLS plays a significant role. The TLS protocol supports the use of numerous cryptographic protocols. Knowing how to use and or configure these protocols can be difficult if not near impossible to understand, but this knowledge is key to providing the level of security required to thwart attacks.
The native go crypto is not FIPS compliant and there are few open proposals to facilitate Go code to meet FIPS requirements. Users can use prominent go compilers/toolsets backed by FIPS validated SSL libraries provided by Google or Redhat which enables Go to bypass the standard library cryptographic routines and instead call into a FIPS 140–2 validated cryptographic library. These toolsets are available as container images, where users can use the same to compile any Go based applications.
BoringCrypto
BoringCrypto (BoringSSL based crypto) maintained by Google is an open-source, general-purpose cryptographic library that provides FIPS 140–2 approved cryptographic algorithms to serve BoringSSL and other user-space applications.
BoringSSL is Google’s forked version of OpenSSL cryptographic library and BoringSSL is used in all Google website product’s TLS stacks since June 2014 — including Google Android OS and Google Chrome. Google initially forked the OpenSSL code with BoringSSL, but have now formally released Google Tink.
Upstream golang/go repository maintains a separate dev branch dev.boringcrypto.go<go-version>. This branch holds all the patches to make Go using BoringCrypto and the community has specified that there is no intention to merge this branch to main.
Go+BoringCrypto release has the form <GoVersion>b<BoringCryptoVersion>, where <GoVersion> is the Go version the release is based on, and <BoringCryptoVersion> is an integer that increments each time there is a new release with different BoringCrypto bits. Google maintains docker images with patched go toolchain, similar to golang:x.y.z.
RKE2 a open source CNCF project, also known as RKE Government uses the goboring/golang which replaces the standard Go crypto libraries with the FIPS validated BoringCrypto module to statistically compile most of the Kubernetes, etcd and other runc components ensuring all levels of stack from Kubernetes daemons, database to container orchestration mechanics are FIPS 140–2 compliant. RKE2 also revalidated this module as Rancher Kubernetes Cryptographic Library for the additional platforms and systems supported by RKE2.
Redhat’s go-toolset
Go Toolset is a Red Hat offering for developers on the Red Hat Enterprise Linux platform. Go Toolset requires a valid subscription (paid) and is distributed as a collection of RPM packages (yum module install go-toolset).
When a RHEL system is booted in FIPS mode, Go will instead call into OpenSSL via a new package that bridges between Go and OpenSSL. This also can be manually enabled by setting GOLANG_FIPS=1. The Go Toolset is available as a container image that can be downloaded from Red Hat Container Registry. Red Hat mentions that this as a new feature built on top of existing upstream work (BoringSSL).
The latest versions of Red Hat Openshift use this go-toolset, when OpenShift runs on Red Hat Enterprise Linux booted in FIPS mode, OpenShift calls into the Red Hat Enterprise Linux FIPS validated cryptographic libraries.
Differences — BoringCrypto | Redhat’s go-toolset
As of now dev.boringcrypto is being kept up-to-date against master and actively maintained as a dev branch part of the official Go repository by the community and there is no guarantee of support in the future. Red Hat offers support contracts and FIPS mode for RHEL and recent releases of Openshift use the toolset to run in FIPS mode, so they are committed to supporting their FIPS complaint go- toolset, this comes with a subscription cost. With Red Hat, FIPS mode can be enabled at runtime.
If a user needs to pass a FIPS audit or validation they have to show FIPS compliance in both the OS and Kubernetes level, so even with using BoringCrypto for Kubernetes components, the underlying OS should be running in FIPS mode.
Traditional Go Crypto vs FIPS Validated Go Crypto
To be FIPS 140–2 compliant, the module must use FIPS 140–2 complaint algorithms, ciphers, key establishment methods, and other protection profiles. FIPS-approved algorithms do change at times; not extremely frequently, but more often than they come out with a new version of FIPS 140. Users can use the latest publications to gather information on what is supported: Documents — Annexes A (approved security functions), B (approved protection profiles), C (approved random bit generators), and D (approved key establishment methods) provide a list of the approved functions applicable to FIPS 140–2.
Some of the fundamental requirements (not limited to) are as follows:
- Support for TLS 1.0 and TLS 1.1 is now deprecated (only allowed in certain cases). TLS 1.3 is the preferred option, while TLS 1.2 is only tolerated.
- 3DES is not tolerated anymore.
- DSA/RSA/ECDSA are only approved for key generation/signature.
- Allowed Hash Standards: SHA-1, SHA-224, SHA-256, SHA-384, SHA-512, SHA-512/224, SHA-512/256
- Allowed Message Authentication: CCM, GCM, GMAC, CMAC, HMAC
- Cipher suites with CBC+HMAC must use the Encrypt-then-MAC extension.
- More TLS extensions should be avoided, like max_fragment_length, ec_point_formats and early_data.
- Session resumption should use key exchange.
- The 0-RTT option in TLS 1.3 should be avoided.
- FIPS 140–2 compliant ciphers: cipher-specifications
Using a sample Go based server application, to access the compatibility/compliance from the information above. Below, the same application is built/compiled using traditional upstream Go and the other is using goboring which replaces the standard/native Go crypto with FIPS 140–2 compliant boringcrypto, the patch here is a drop-in replacement, where no changes to the code are needed if required users can make changes/configure to filter specific ciphers, etc.
The app below is built using traditional/official golang:1.16.2:
Testing the above application using testssl.sh which tests a service on the given port for the support of TLS/SSL ciphers, protocols and recent cryptographic flaws.
As shown below upstream Go crypto:
- Allows unsupported/deprecated TLS 1 and TLS 1.1. TLS 1.0 is prohibited and TLS 1.1 is deprecated for government sites and not supported. Recently AWS removed support for 1.0 and 1.1 and enabled 1.2 for all AWS FIPS Endpoints.
- Allows Triple DES (a TLS 1.2 standard) which has a really “small” blocksize with only 64 bits, which led to attack such as Sweet32 against TLS sessions.
- Allows usage of obsoleted CBC ciphers which are proven to be vulnerable, these can expose the TLS connection to padding oracle attacks/lucky thirteen attack.
As shown below Go crypto allows unsupported TLS v1 and v1.1 and also TLS v1.2 ciphers such as ECDHE-RSA-DES-CBC3-SHA and DES-CBC3-SHA based on Triple DES which are not FIPS compliant. Go crypto also allows multiple non-compliant CBC ciphers with TLS v1.2
The Forward Secrecy below supports some non-compliant ciphers such as ECDHE-RSA-AES256-SHA, ECDHE-RSA-CHACHA20-POLY1305.
The same application above is compiled using goboring/golang, this replaces standard Go crypto with FIPS validated cryptography. An assert step is added to assure that the build is using boringcrypto instead of native golang crypto.
Users can check whether a given binary is using BoringCrypto by running go tool nm on the binary and check for *_Cfunc__goboringcrypto_*. For example in case of the sample application above:
As shown below the boringcrypto which replaces the standard Go crypto library:
- TLS 1 and 1.1 are not offered
- Triple DES is dis-allowed/not offered
- Obsoleted CBC ciphers are dis-allowed/not offered
As shown below the app compiled with boringcrypto only allows supported TLSv1.2 & v1.3 and only allow FIPS 140–2 compatible ciphers. The forward secrecy also only allows complaint ciphers.
The same scenario applies to all Kubernetes components and other entities of the ecosystem, wherewith boringcrypto all the components only allow TLS connections using the FIPS 140–2 compliant ciphers and algorithms.
CAVP and CMVP (Cryptographic Algorithms Validation Program & Cryptographic Module Validation Program)
ISVs (Independent Service Vendors) looking to obtain FedRAMP accreditation must comply with FIPS 140–2 encryption standards. FIPS Compliance means that different components of a product have received FIPS validation, but the product in its entirety has not passed testing or has not been tested at all. FIPS Validation means a product has undergone and passed detailed conformance testing at an accredited national laboratory.
In order to perform FIPS 140 conformance testing, a laboratory must become an accredited CST (Cryptographic and Security Testing) laboratory under the National Voluntary Laboratory Accreditation Program (NVLAP). The current list has 12 accredited laboratories in USA.
The Cryptographic Module Validation Program (CMVP), a joint effort of the U.S. National Institute of Standards and Technology (NIST) and the Canadian Centre for Cyber Security (CCCS), validates cryptographic modules to the Security Requirements for Cryptographic Modules standard (i.e., FIPS 140–2) and related FIPS cryptography standards. The FIPS 140–2 security requirements cover 11 areas related to the design and implementation of a cryptographic module. The NIST Information Technology Laboratory operates a related program that validates the FIPS approved cryptographic algorithms in the module (Cryptographic Algorithms Validation Program (CAVP)).
The CMVP process consists of four entities namely Vendor, CMT (Cryptographic Module Testing) Labs, CMVP Validation Authorities, and the Users. The cryptographic module of the vendor is submitted to CMT Lab where testing is performed. After the testing by CST Lab, the report is given to CMVP Validation Authorities for final validation and certification. Users, in turn, verifies the cryptographic module.
Obtaining a FIPS 140–2 Certification can take several months. The time frame is typically between 2–4 months with the lab (depending on the type of module to be tested and the overall security level) after the lab receives all the required documentation and module(s) to be tested from the vendor. These are high-end time frames and can be shorter depending on the completeness of the documentation, product meeting, and passing all the requirements.
Cryptographic Module is validated and tested with the help of two programs namely: Cryptographic Algorithms Validation Program (CAVP) and the Cryptographic Module Validation Program (CMVP).
Cryptographic algorithm validation is a prerequisite of cryptographic module validation.
In CAVP, Algorithms in Cryptographic Module are checked (whether the Cryptographic Module is using FIPS approved algorithm or not) whereas CMVP checks the whole Cryptographic Module as per the FIPS 140–2 Derived Test Requirements.
The CMVP validates commercial cryptographic modules to Federal Information Processing Standard (FIPS) 140, NIST-recommended standards, and other cryptography-based standards. The CMVP is a government validation program that is jointly managed by NIST and CCCS.
Fips 140–2 validation Process — Analysis using Rancher’s Kubernetes Cryptographic Library as an example
RKE2, also known as RKE Government, is Rancher’s open source next-generation Kubernetes distribution. It is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector.
RKE2 is built with FIPS validated cryptographic libraries using goboring Go compiler maintained by Google. In RKE2 almost all the components (cluster components, runtime) are built with the version of Go from the goboring compiler which replaces the standard Go crypto libraries with the FIPS validated BoringCrypto module. In order to support additional platforms, Rancher opted for revalidation of the module as Rancher Kubernetes Cryptographic Library through one of the NVLAP accredited laboratories (all information procured from publicly available sources).
The following are the steps that are involved for Rancher to procure a FIPS 140–2 validation for their cryptographic module:
- CMT Lab Engagement: As shown below Rancher submitted their module to Acumen Security (NVLAP Code: 201029–0). Document Source: rke2_fips_engagement
2. CAVP (Cryptographic Algorithm Validation Program): The lab above tests the algorithm implementations using test tools provided by NIST and CCCS and an algorithm implementation successfully tested by a lab and validated by NIST is added to an appropriate validation list, which identifies the vendor, implementation, operational environment, validation date and algorithm details (source).
3. CMVP (Cryptographic Module Validation Program): Once CAVP is complete, the lab sends the reports to CMVP where it will be going through module pending, module review stages and reaches the finalized stage (~4 months). On completion of CMVP a certificate number is associated to the module and validation dates, sunset dates are specified. (source)
The tested configuration section includes a list of environments where the module is tested and also includes different configurations in the same environment. The FIPS and Allowed Algorithms sections specify the list of algorithms used in the module or under the scope of validation.
The last section includes the vendor details, the NVLAP lab name & details, a link to module-specific security policy, and a consolidated certificate.
5. Once CMVP is complete the module will be added to the list of modules (some of the most recently validated) in a consolidated certificate.
FIPS 140–2 Levels and Kubernetes
The FIPS 140–2 standard provides four increasing, qualitative levels of security: Level 1, Level 2, Level 3, and Level 4. These levels are intended to cover a wide range of potential applications and environments into which cryptographic modules may be employed.
A particular level requires that the previous levels also be met, but not every product must reach FIPS Level 4. For example, Level 1 provides the most basic security with practically no physical requirements, such as a personal computer encryption board, which is a validated Security 1 cryptographic module. In order for a system to be Security Level 2 validated, it would need to comply with all the standards outlined in Level 1, and additionally meet role-based authentication requirements to account for tamper-evidence required in FIPS Level 2. Certain levels are only appropriate for certain products or solutions.
As Kubernetes is a software stack, modules specific to Kubernetes will be provided with a Level-1 certification. Here cryptography is the entity that is tested and other levels 2–4 are non-applicable in this scenario as they are more related to physical and hardware-based security. For example, the following are the links to some of the validated modules from different vendors: D2iQ, Rancher, Oracle.
Taking D2iQ BoringCrypto Cryptographic Security Module as a specific example:
The ‘Overall Level’ field specifies the certification level provided to the module (other modules: Rancher, Oracle). Every module will have a certification expiration date (Sunset Date).
The caveat section contains information required by the CMVP for the cryptographic module. Common caveats describe operational requirements for “FIPS mode” and entropy. For the module above it mentions ‘section 12.1 of the security policy’, the Security Policy document link will be provided with every validated module and section 12.1 above provides the information on how D2iQ compiles the FIPS enabled module and guidance on the installation.
Taking another example Rancher Kubernetes Cryptographic Library which is awarded Level-1 certification.
The caveat section here specifies ‘Section 9.1’ of Security Policy, this provides installation guidance and instruction on compiling the module.
The ‘Security Level Exceptions’ specifies the list of entities that are not applicable for the module and not in the scope of validation. As shown below the D2iQ BoringCrypto Cryptographic Security Module lists physical security and migration of other attacks as not applicable (the same with Rancher, Oracle), the Level-1 considers the software components (specifically tamper-proof cryptography) and doesn’t require any hardware or other physical security requirements (levels 2–4).
The ‘Module Type’ specifies the module category ‘Software/Hardware/Firmware/Hybrids’. ‘Tested Configuration’ lists the different distros, versions, compilers and hardware on which the module is tested. A module can contain multiple algorithms, ‘Fips Algorithms’ list all the individual algorithms used in the module and their certificate links. ‘Allowed Algorithms’ specify a list of encryption algorithms that the module allows.
The last section includes details of the vendor, the NVLAP (National Voluntary Laboratory Accreditation Program) accredited laboratory information, a link to the Security Policy of the module, and a link to Consolidated Validation Certificate.
Many government organizations and commercial organizations in government-regulated industries are adopting Kubernetes as their standard for orchestrating containers, vendors by maintaining FIPS 140–2 compliance enables government agencies to adopt a DevSecOps transformation to build on a solid base of government-ready defaults, removing the heavy lifting required to meet a lot of government compliance requirements.
FIPS 140–2 is one of the many requirements to operate in the government sector, there are many other Kubernetes specific security hardening requirements such as PodSecurityPolicies, NetworkPolicies, fine-grained authorization(RBAC), dynamic secret management, CIS Hardening, SELinux enablement, Usage of distroless containers, etc. These entities together enable public sector customers to run some of the most mission-critical workloads in the world.
