It’s funny that we’re all building delivery setups that work if we don’t change anything. Everything is freshly scripted into some CI/CD pipeline. There’s even Terraform spinning up stuff (so 2021). Developers do a simple git-push and the image ends up in a namespace. That’s amazing. Until it’s not. Things usually break if anything changes except for somebody pushing a new image. Lightweight example: configuration has to be updated, you need one more environment variable. Not a big deal. Let’s say we’re on Kubernetes using Helm. Open the file, add environment variable, merge, good to go. The CI pipeline will execute and we’re live - easy. Until you add 100 changes on 20 microservices in three different environments and then production fails. You know it has something to do with configs and you start looking into this. You have a long night in front of you.
Let’s add one more level of complexity to what developers may want beyond swapping an image: a new environment. The list of what you have to do this time is already significantly longer. You’ve done it and I’ll spare you here, but let’s just say it involves touching a lot of scripts and systems. There are tons of more cases I can give like spinning up new resources, dealing with hotfixes, adding services, rolling back, or debugging.
The “we’re not changing anything” case, the one we all optimize for by scripting static pipelines, is the dominant one, certainly. But if you change something only once every five times (which sounds reasonable), plus your team is of considerable size, these add up real fast. There are two strategies on how to mitigate this issue: developers do the change themselves -> leads to an unmaintainable setup where everybody scripts stuff all over the place. Or developers turn to operations and operations are doing ticket-ops. Developers wait, ops hate it.
The glorious new buzzword that has to execute all these scripts is called GitO(o)ps. As I argued before, the problem is the static input, not GitOps. There’s a way to solve this and mitigate the downside: dynamically auto-creating manifests and IaC with every deployment depending on the context and the history of the application. That’s just what an Internal Developer Platform does. And it’s what Chris and Kostis will show you in our upcoming webinar (which I am really looking forward to). This is the recording of part 1, this is where you can sign up for part 2.
It’s so insane what lineup of speakers we have on Humanitec.com/webinars for the rest of the year, definitely worth checking out.
Stay safe, talk soon
Kaspar
