Your team likely has a well defined and practised strategy in place for managing changes to a codebase using version control. However, when you use Kubernetes (K8s) to create and manage your application infrastructure, K8s configuration files fast become as important to manage as your code. These configuration files are key to creating, maintaining, scaling, and rolling back the infrastructure that powers your application, and managing them effectively is an essential strategy to have in place.
Changes to configuration files might include everything from small changes such as the name of a variable or resource limits, up to major rearchitecting of resources and their relations to each other. These changes can be as significant to your application as changes to the application code and have as much impact on customer experience. Why do we often treat changes to configuration as less important? In this post, I look at tools and practices to help you manage changes to configuration.
Use a tool to help
While writing and versioning YAML is not the most complex process, it is tedious and prone to introducing errors. One solution is to use a tool that lets you create templates of your configuration which it populates with current values at a time you specify.
Helm
Considered the “package manager for Kubernetes,” Helm is the most popular option many turn to for templating and versioning a complex application. Helm creates “charts,” that allow you to define complex K8s applications into one easier to update and version package.
A chart consist of the following files:
- Chart.yaml: Metadata about the chart.
- values.yaml: Default configuration values for the chart that you set and update during a build process.
- charts/: A directory of other charts (“subcharts”) the chart uses.
- templates/: A directory of template YAML configuration files that helm combines with values during a build process.
This post won’t go into detail on creating a chart (and a lot is possible), the Helm documentation is the best source for that, but look at the template part of a chart.
Our post on orchestrating an application with Kubernetes had a StatefulSet. Here’s a part of it:
<p> CODE: https://gist.github.com/ChrisChinchilla/675e6db0a5906141264157abc73abb1e.js <p/>
Say for example that you want to experiment with different versions of the Postgres Docker image between environments.
The Helm equivalent of the YAML file is the following, with the image as a variable:
<p> CODE: https://gist.github.com/ChrisChinchilla/7fa8970b162b5a594f6356411f0812c1.js <p/>
And the Values.yaml file has the associated variable:
postgresVersion: latest
Whereas Values.yaml for a testing environment (contained in a git branch or tag) has:
postgresVersion: 9.6.19
Kustomize
With a limited, but more relevant feature set, Kustomize is comparable to a specialized diff view for K8s configuration. Kustomize traverses a K8s manifest to add, remove or update configuration options without forking. Now part of kubectl since version 1.14, you can use kustomize to apply patches to configuration files, and create variants of files to match development environments by specifying the differences between common file bases.
Configuration managed by kustomize needs one base directory that contains a kustomization.yaml file that has metadata, resources included, and other fields that match kustomize functionality. The base directory also contains the “default” K8s resources.
You also need sub directories to match environments in an overlays folder. All these sub directories need their own kustomize.yaml file (with similar content to the above) and patches that contain instructions on how to change the base resources.
Confusingly, while kustomize uses the term “patch,” it does not mean the same as a git patch. Kustomize has two patch formats, and the most commonly used is a “patchStrategicMerge” (or SMP). An SMP contains just enough information to change relevant values, which has the potential to be confusing. But as K8s integrates kustomize better, there are solutions to generate the patch files with kubectl.
Taking the Postgres example above, the overlay patch is:
<p> CODE: https://gist.github.com/ChrisChinchilla/9b06db8282b26e6dff4558be83ef450a.js <p/>
setElementOrder is a directive that specifies the order of a YAML list, which makes it easier for kustomize to apply the patch.
You then use kubectl apply -k {OVERLAY_FOLDER} to apply the patches.
Use a tool suited to the job
Whether you use a tool to assist you to generate configuration differences or not, you’re only solving half the problem. You can keep different versions of your environment configuration in different repositories, branches or tags, but this is not how git was designed. Git, and other version control systems (VCS), are designed for handling short-lived changes that will be merged into a longer-living branch at some point. This process does not describe how teams typically handle environments, some of which can exist, in some form, for months, or years.
Using a traditional VCS to manage environment configuration requires you to duplicate configuration content (and keep those changes in sync). For example, if you used Helm to create a base chart, and change the Values.yaml file to suit each branch or tag, you still need to keep any changes to the base configuration in sync across these branches or tags.
Humanitec takes a different approach to managing changes to environment configuration. Configuration is managed alongside the environment that it is intended for. This avoids trying to reconcile merging towards one single source of truth and the fact that there will be a (maybe growing) number of environments on hand.
The benefits of separated code and configuration
A different pace of development
Your configuration is a fundamental part of your application but moves at a different pace, and changes to each of them are not always related or should trigger pushes to a cluster.
Rollback changes
Maintaining configuration in a different way to your application means you can roll back to a previously good state of either code or config should you need to, and makes it easier for teams to recreate and restore clusters whenever they need to.
Clear connection
Using a different tool for managing changes to your environments makes it easier to match changes to a running environment, when the changes are visible in the environment itself.
