We see many platform teams wondering which set of tools is ideal for building an Internal Developer Platform (IDP) that suits their organization’s needs best. There is of course no one right answer nor any one-fits-all solution, but we are starting to gather best practices and reference implementations from top performing engineering organizations. In this article, we will dive deeper into one such example, focusing on the following tools:
- Github Actions as CI tool
- GCR as an image registry
- Humanitec as the Platform Orchestrator
- Crossplane for IaC
- Hashicorp Vault as secrets manager
- GCP as the cloud platform, including GKE, Cloud SQL
To explain how services would be handled by the platform we imagine the following trivial application: One containerized python service, connecting to a Cloud SQL, running on a GKE namespace and exposed to the public internet with DNS.
At the end, our platform setup should support the following requirements:
- Declarative representation of all components in Git
- Dynamic configuration management (env agnostic separated from env specific elements)
- Standardization by design with low maintenance effort through workload and infrastructure profiles
- Low cognitive load for developers without abstracting them
- Scaffold new services including dependent resources
- Continuous Integration and testing
- Secret Management
- Deployment and deployment automation
- Diffs between deployment, roll-backs
- Ephemeral Environments, partial releases from env to env, PR environment and general environment management
- Debugging, visualization (what’s running where), logging
- RBAC fencing
This is not supposed to be a step by step guide but only describes the architectural relationship between all the components.
Also note that we are looking at the core components of an IDP doing the heavy lifting of application configuration and infrastructure orchestration. We will cover only to a certain degree the UI and interfaces to the platform, or where to fit in for example a developer portal or service catalog for visualization. If you want to dive deeper into the UI and DevEx side of things, check out this PlatformCon talk “architecting and IDP with Backstage and Humanitec”, where Backstage is used as a click-ops UI in combination with Humanitec as the Platform Orchestrator.
Our target architectural diagram will look something like this:
In the next step, we will dissect what we’re using, what components and how all of this fits together bit by bit. We are going to assume that we’re starting from scratch. All we have is our little containerized demo service. We’ll configure this including all dependent resources, build it and spin up the first environment from scratch.
Dynamic configuration management via VCS/GitHub
Our setup starts in the Version Control System (VCS). Because we’re aiming to support dynamic configuration management, we’ll describe our application in a Declarative Application Model.
Contrary to a setup with static configuration management, where we have unstructured YAML, Crossplane files and Helm charts all over the place for each environment; dynamic configuration means we create the final representation of the application for every deployment in each environment.
This approach drives standardization by design by letting platform teams share reusable components across workloads. It reduces the cognitive load on the individual contributor by letting them choose the level of abstraction. And it unleashes a variety of previously impossible sets of functionality, such as spinning up a new environment, diff deployments to understand material changes, rolling back, etc.
To get the setup to be ready for dynamic config management, we’ll need to pull apart environment agnostic from environment specific elements of the configuration.
To do that we create the following repositories:
- Workload (simple python service + docker file)
- Abstract workload configuration - for instance, the Platform Agnostic Workload Specifications (PAWS) to describe in an abstract way how the workload relates to other workloads as well as to dependent resources
- Workload Profile (think of it like an empty Helm chart that weaves in default configurations)
- Infrastructure Profiles (this is where we put all our Crossplane files so the Platform Orchestrator can execute them at the correct trigger).
- Resource Matching (we’re matching certain trigger criteria “I need S3 for an environment of type “ephemeral” with the infrastructure profile)
Note that we can reuse the workload profile, infrastructure profiles and the resource matching across more workloads or teams, which has a positive effect on the cleanliness of our config setup. On a daily basis the individual would focus on the workload and the abstract workload config.
This Declarative Application Model now contains all information about how to dynamically create the representation of the app just in time with the deployment. We’ll learn below how the Platform Orchestrator will interpret this model and orchestrate/deploy everything.
From build to production with GitHub Actions and the Platform Orchestrator
As described above we use Github Actions for our example to build and test our code. The pipeline will have a curl command at its last step to inform the Platform Orchestrator (Humanitec in this case) that a new build is available in GCR.
For testing, the IDP is supposed to spin up an ephemeral environment with a full set of new infrastructure, as similar as possible to production.
To do this via the CLI (developers could also use the UI, the API or a git-based approach, more below), the command looks like this:
humctl create ephemeral --name "demoenv"
This command will kick off the following stream of activities:
- The orchestrator has already received the notification from the pipeline that the image is available in GCR.
- The orchestrator reads the latest changes to the Declarative Application Model and creates manifests.
- It uses the manifests and runs them against kubectl to configure Kubernetes.
- Next it looks up what infrastructure profile to use for the environment context, which in our case is “development”.
- In our case, the infrastructure profile is using Crossplane to create a database in an existing Postgres instance, a new namespace and a new DNS with Route53.
- It will receive the credentials from the resources and inject them into the containers as secrets at run-time.
How do developers interact with this IDP?
So how do individual contributors interact with this IDP setup? The answer is: usually, not that often. If developers only update code and the image, they will stay in their editor, do their git push and the automation will look at the tag or other criteria to figure out where to deploy their update. But the setup also provides capabilities for more advanced cases like rolling back, spinning up a new environment, changing the architecture or resources, updating your Crossplane files, or any of the above.
Avoid context switching - just stay in the editor
To avoid context switching between different tools and interfaces, the Declarative Application Model sits as code in the repo. If you want to change the architecture (the relationship between services and their dependent resources such as databases, storage or DNS), you can do this in the abstract workload configuration.
Assume we have the dependency of the workload on a database expressed like this (using PAWS):
But I want to add the dependency on a DNS. We’d just enrich the file as follows:
Developers can simply run a git push and the app gets configured and deployed. If they want to change which Crossplane file is used at a matching criteria, they just edit the resource matching file and alter the matching criteria. If they like to change the Crossplane file directly, they can do so and point the resource matching at the new version.
UI, portals and service catalogs
In case individual contributors feel more comfortable with a click-ops approach, Humanitec also provides a UI.
Let’s for instance, run a partial release from one environment to the next. Partial release means we are rolling a subset of services from one environment into another environment with one click. You clone the configuration state from a previous deployment, select which services you want to release into the new environment and off you go. The UI will show you a delta between the state of the environment now and after you’re applying the change including all the material changes between them.
You can also use Humanitec in combination with a developer portal like Backstage, which comes with a service catalog and API documentation on top of our platform setup. An often used example is scaffolding a new service using Backstage. Simply select the sample in the service catalog; it already contains the declarative application model. If you hit “create” Backstage will pass this on and Github, Actions, Humanitec and Crossplane will take care of the rest.
CLI, API
Humanitec’s Platform Orchestrator also comes with an API and CLI. This provides the opportunity to build more advanced automations, auto-deployment rules for instance (read more here):
You can also pull lots of information from the API to build cool things. Such as figuring out what workload runs where in which state.
Summary
In this brief architectural review we looked at how teams can combine GitHub, GitHub Actions, GCP, Humanitec and Crossplane. This is based on many real life implementations we have seen, there are of course many more combinations possible, with other tools. Gitlab or Jenkins for CI, using AWS or Azure, Terraform, Pulumi and others. On the infrastructure front, many teams also use the infra control planes offered by the makers of Terraform, Crossplane and others. In the case of Crossplane, that would be Upbound. In the Declarative Application Model you can also put static drivers to just call Upbound or Terraform Cloud to start the creation/update of a respective resource. We’ll cover this in one of our upcoming posts.
If you have certain combinations of tools you’d want to see covered, let us know!
