This project contains GoCardless' Kubernetes extensions, in the form of operators, admission controller webhooks and associated CLIs. The aim of this project is to provide a space to write Kubernetes extensions where:
- Doing the right thing is easy; it is difficult to make mistakes!
- Each category of Kubernetes extension has a well defined implementation pattern
- Writing meaningful tests is easy, with minimal boilerplate
Theatre provides various extensions to vanilla Kubernetes. These extensions are
grouped under separate API groups, all of which exist under the
Utilities to extend the default Kubernetes role-based access control (RBAC) resources. These CRDs are motivated by real-world use cases when using Kubernetes with an organisation that uses GSuite, and which frequently onboards new developers.
DirectoryRoleBindingis a resource that provisions standard
RoleBindings, which contain the subjects defined in a Google group.
Note: In a GKE Kubernetes cluster this may soon be superseded by the Google Groups for GKE functionality.
Extends core workload resources with new CRDs. Extensions within this group can be expected to create or mutate pods, deployments, etc.
Consoles: Provide engineers with a temporary
dedicated pod to perform operational tasks, avoiding the need to provide
pods/execpermissions on production workloads.
- Default priority classes: Mutate all pods within a namespace to set a default priority class.
Utilities for interacting with Vault. Primarily used to inject secret material into pods by use of annotations.
secrets-injector.vault.crd.gocardless.comwebhook for injecting the
theatre-secretstool to populate a container's environment with secrets from Vault before executing.
Command line interfaces
As well as Kubernetes controllers this project also contains supporting CLI utilities.
theatre-consoles is a suite of commands that provides the ability to create,
list, attach to and authorise consoles.
go run cmd/theatre-consoles/main.go
See the command README for further details.
go run cmd/theatre-secrets/main.go
Theatre assumes developers have several tools installed to provide development and testing capabilities. The following will configure a macOS environment with all the necessary dependencies:
brew cask install docker brew install go kubernetes-cli kustomize kind mkdir /usr/local/kubebuilder curl -fsL https://go.kubebuilder.io/dl/2.3.1/$(go env GOOS)/$(go env GOARCH) | tar -xvz --strip=1 -C /usr/local/kubebuilder export KUBEBUILDER_ASSETS=/usr/local/kubebuilder/bin
make should now compile binaries into
Local development environment
For developing changes, you can make use of the acceptance testing
infrastructure to install the code into a local Kubernetes-in-Docker
kind is installed (as per the [getting started
steps][#getting-started]) and then run the following:
go run cmd/acceptance/main.go prepare # prepare the cluster, install theatre
At this point a development cluster has been provisioned. Your current local Kubernetes context will have been changed to point to the test cluster. You should see the following if you inspect kubernetes:
$ kubectl get pods --all-namespaces | grep -v kube-system NAMESPACE NAME READY STATUS RESTARTS AGE theatre-system theatre-rbac-manager-0 1/1 Running 0 5m theatre-system theatre-vault-manager-0 1/1 Running 0 5m theatre-system theatre-workloads-manager-0 1/1 Running 0 5m vault vault-0 1/1 Running 0 5m
All of the controllers and webhooks, built from the local working copy of the code, have been installed into the cluster.
As this is a fully-fledged Kubernetes cluster, at this point you are able to
interact with it as you would with any other cluster, but also have the ability
to use the custom resources defined in theatre, e.g. creating a
If changes are made to the code, then you must re-run the
prepare step in
order to update the cluster with images built from the new binaries.
Theatre has test suites at several different levels, each of which play a specific role. All of these suites are written using the Ginkgo framework.
Unit: Standard unit tests, used to exhaustively specify the functionality of functions or objects.
Invoked with the
Integration: Integration tests run the custom controller code and integrates this with a temporary Kubernetes API server, therefore providing an environment where the Kubernetes API can be used to manipulate custom objects.
This environment has no Kubernetes nodes, and does not run any other controllers such as
kube-controller-manager, therefore it will not run pods.
These suites provide a good balance between runtime and realism, and are therefore useful for rapid iteration when developing changes.
Invoked with the
Acceptance: Acceptance is used for full end-to-end (E2E) tests, provisioning a fully functional Kubernetes cluster with all custom controllers and webhooks installed.
The acceptance tests are much slower to set up, and so are typically used sparingly compared to the other suites, but provide essential validation in CI and at the end of development cycles that the code correctly interacts with the other components of a Kubernetes cluster.
go run cmd/acceptance/main.go run