Why does everything always take so long? Most of us ask ourselves this question every day. In this article Humanitec’s CEO Kaspar von Grünberg shares some thoughts about developer productivity and how to improve it. 7 points about your DevOps workflow you should definitely keep in mind!
There’s a lot written about how the way developers structure their daily work can cause unproductivity. An example is when unnecessary meetings are scheduled across the day so nobody can get into deep focus mode. Today I want to look into the biggest killers in developer productivity: the way you configure and setup your DevOps workflow. In almost all situations I’ve come across there were some quick-wins that help you avoid most of the problems.
When teams work in a monolithic setup everything sort of works. The toolchain is prepared to handle this one monolith well, but yes, changing one small thing requires the deployment of the whole monolith. End to end tests need to be run in order to verify that everything is still fine. The bigger the monolith is, the less efficient this will be. So the team goes ahead and adopts microservices. Their first experience is great, colleagues can work on individual services independently, deployment frequency goes up and everybody is happy.
The problems start when teams get carried away with microservices and take the “micro” a little too seriously. From a tooling perspective you will now have to deal with a lot more yml files, docker files, with dependencies between variables of these services, routing issues etc. They need to be maintained, updated, cared for. Your CI/CD setup as well as your organizational structure and probably your headcount needs a revamp.
If you go into microservices for whatever reason, make sure you plan sufficient time to restructure your tooling setup and workflow. Just count the number of scripts in various places you need to maintain. Think about how long this will take, who is responsible and what tools might help you keep this under control. If you choose tools, make sure they have a community of users also using them for microservice setups.
Containerization is an awesome technology for a lot of situations. However it comes with a price tag and can have an impact on your productivity. Containers add overhead from a security perspective and through necessary configuration and environment management etc. They can also hurt your productivity and developer experience if you don’t agree on certain conventions as a team.
The most common mistake I’m seeing is: building your config files or environment variables into your container. The core idea of containerization is portability. By hard coding configuration you will have to start writing files and pipelines for every single environment. You want to change a URL? Nice, go ahead and change this in 20 different places and then rebuild everything.
Before you start using containers at scale and in production, sit down as a team and agree what config conventions are important to you. Make sure to consistently cover this in code-reviews and retros. Refactoring this a-priori is a pain.
All the cool kids are hyped about this open source project called Kubernetes. However, Kubernetes is hard to keep running and hard to integrate into your developer flow while keeping productivity and experience high. A lot can go wrong:
Kubernetes worst case: Colleague XY really wanted to get his hands dirty and found a starter guide online. They set up a cluster on bare-metal and it worked great with the test-app. They then started migrating the first application and asked their colleagues to start interacting with the cluster using kubectl. Half of the team is now preoccupied learning this new technology. The poor person that is now maintaining the cluster will be full time on this the second the first production workload hits the fan. The CI/CD setup is completely unprepared for dealing with this and overall productivity is going down as the entire team is trying to balance Kubernetes.
What can be done to prevent this: Kubernetes is an awesome technology and can help achieve a PaaS like developer experience if done right. After all, it descends from Borg - the platform Google built to make it easy for their Software Engineers to build massively scalable applications. Thus, it's kind of an open source interpretation of Google's internal platform.
“Wait, I already have a CI Tool”. There is a common misconception that the job is done well if there is a Continuous Integration setup. You are still missing Continuous Delivery! The confusion is not helped by a lot of these vendors coining the term “CI/CD tool” giving you the impression you’ve nailed Continuous Delivery if you have Jenkins, CircleCI etc. - that’s not the case.
A well tuned Continuous Delivery setup, either self-scripted or “as-a-Service” is much more the “glue” in a teams toolchain:
Efficient testing is not possible without automation. With continuous delivery comes continuous responsibility to not break anything.
You need to continuously make sure to not fall into the trap of inverting your test pyramid. For this you need to be able to run the right kind of tests at the right point of your development lifecycle.
Sufficient CI tooling will help you to put your unit and integration tests into the right place while CD tooling with configuration management and environment management helps you to run your automated end to end tests in a reliable way.
Well done setups allow developers or testers to dynamically spin up environments that are preconfigured. Strictly externalize your configuration and make sure to have a configuration management that injects these variables at deployment time. This leads to a number of positive improvements:
The teammate who just left was responsible for setting up MongoDB for a client project and of course used the open source project to run it themselves. And of course the handover was ‘flawless’ and of course the database wasn’t protected properly and one evening this shows up where the data was supposed to be:
And of course:
You check the backups.
There was a syntax error.
You now have to reverse engineer all the data.
This is a real life example that happens frequently.
Self managed DBs are an operational and security risk. They are distracting, boring and unnecessary. Use Cloud SQL or other offerings and sleep well. We are commonly seeing managed offerings from companies such as Aiven.io. These companies offer most databases, they can run them on all the big cloud providers for you and they are more feature-rich, mature and sophisticated. Also, they are often cheaper and ensure zero lock-in at a higher developer convenience which if it goes hand in hand I’d always prefer.
There is a difference between just going multi-cloud and trying to design your systems to be cloud agnostic and portable. The latter has a lot of different advantages such as dynamic environments and makes more sense than going multi-cloud. Sure there’s a historical legacy: some teams had been using GCP and the other department started with AWS and here you are. Others include specialization. One might argue that GPUs run more efficiently on GCP than on AWS or cost-reasons. But these effects to really surface you need a sufficient size. Uncomplicated multi-cloud setups require a high degree of automation and shielding of provisioning and setup tasks from developers. Otherwise one ends up in scripting hell.
As a general rule: don’t do multi-cloud if not absolutely necessary.
I hope these points help you avoid the biggest mistakes in this field. Remember what Nicole Forsgren, Jez Humble and Gene Kim write in their book “Accelerate”: “the top 1% of teams ship 10x more often”.
This is because they are getting the most of what is possible today. I spend 1 hour a month looking at my personal workflows, my to-do lists, the way I organize my apps. Why? Because it really adds up over the weeks if you have inefficient flows. These tiny things such as searching for your photo app distract your brain. Stop and spend an afternoon a month to ensure your productivity is streamlined. It will help you focus on innovation rather than configuration and will make for a happier team.
Reach out directly to me if you have ideas, comments or suggestions. Or register for one of our free webinars to get in touch with us.