We believe that every developer needs three ways to view their work and their progress.
- Their editor or IDE where they write code.
- Version control where they collaborate and merge their work.
- Their Internal Developer Platform where they deliver their work to users.
Many tools exist to help developers deliver their work to users, so why go to the lengths required to create your own, or use an Internal Developer Platform such as Humanitec?
In these interviews, we speak with tech leaders at major global companies who have created Internal Developer Platforms to discover what problems they were trying to solve, how they did it, and what advice they have for others following the same process.
In this article, we summarize a conversation we had with Paolo Garri, the director of technology at Sport1, a German-based sports TV channel with online delivery and mobile apps. They have a small development team of about 25 application developers and 5 people in Ops and have built their own Internal Developer Platform, and we explore the reasoning behind and their process in this article.
What made Sport1 build an Internal Developer Platform
An Overly Complex Development Process
As with many companies building Internal Developer Platforms the key pain points Sport1 faced became more urgent as their setup became more complex. In their case it was growing complexity in terms of amount of services as well as developers. They used to manage everything with Jenkins and were drowning in multiple versions of scripts in multiple languages.
A Reliance on Custom Scripts
The old codebase and development process consisted of 100s (3-400) of scripts used for testing and deployment. They were written in various languages, and no one truly knew what they did or how they all worked. Any machines that needed to run them would have to support multiple programming languages to accommodate them. That posed several problems. One was time lost maintaining these scripts. If people would switch jobs it would take them some time to understand and handle these scripts. Another problem was serving developers. In most cases when teams try the “you build it you run it” approach, what actually happens is an “App Dev builds and throws over the fence”. If you run on a complexly scripted setup your ops teams need to help your application development teams get their code live, which increases the pressure on them.
Sinking Time trying to fix broken Setups
When Sport1 was running the monolithic codebase, an equally monolithic Jenkins server handled every aspect from testing through to deployment. Attempting a small change to the Jenkins pipelines could take up to three months, as there was generally so much trickle-down effect from any change to other services and jobs, it was important to be sure what the implications of every change could be. This took up a lot of the Ops team’s time and focus.
Key Incidents that drove the Development
There wasn’t one particular incident that motivated Paolo to push to create an Internal Developer Platform. But as the setup grew in complexity (they even tried multi-cloud at one point) the development team kept slowing down. After watching one external team spend three weeks trying to integrate a new website into the Jenkins setup, he threw together a quick business case to show how much time and resources were wasted in continuing to work with the current setup versus working on something new. As we’ve seen in other cases the analysis was a headcount driven one. He realized that only a different setup with less variance in processes, more standards, a higher degree of automation, and developer self-service could help with the desired speed in development. This in the end was only possible with a platform that would allow the team a structured approach of letting developers apply changes to baseline templates rather than working in unstructured environments. This platform could also let Ops and developers focus on their respective areas exclusively.
Tech - How Sport1 built their Internal Developer Platform
After Paolo decided it was time to build an Internal Developer Platform, the team didn’t start building a fully-fledged system. They started with an MVP and migrated service by service, or as services needed updating. In the long run, this made deploying new features much easier and faster than with the previous monolithic codebase.
Deciding on Technology and Implementation
Sport1’s Internal Developer Platform is a configurable platform that supports security and performance testing and works well with Python and GitLab, which were a key part of Sport1’s development workflow. Under the hood, the team initially experimented with orchestration using Docker Swarm but moved to Kubernetes after encountering lots of issues.
The platform is mostly CLI and configuration file-based and has no UI, which occasionally leads to some confusion for tracking down the source of errors or test results, but suits the typical workflow of the development team. The new system doesn’t rely on unstructured scripts, but standardized options such as Kubernetes operators. It abstracts the complexity of Kubernetes away and integrates features such as spinning up new environments or databases. It aims to make developers 100% autonomous.
Even though Sport1 is now multi-cloud, the experience and setup is the same irrespective to what cloud a service is deployed to.
The Timeline
Building the platform and migrating involved up to 3 people from the Ops team for the first 2 months until the platform reached MVP status. It then took another 5 months with 3 people working on the platform to deliver version 1. Currently, one person maintains the Internal Developer Platform and another is working part-time to extend the feature set. That leaves everyone else focussed on the actual core business.
What was the impact of the Internal Developer Platform?
Changes for the Development Team
To begin with, the developers were happier with the older setup, as the Ops team was responsible for applying changes and developers only had to contact them if they wanted something deployed. The development team had a lot of freedom with implementing new features and changes, but this led to problems with testing and security that the DevOps team had to deal with.
Paolo wanted to limit the freedom of the development team somewhat and pass that flexibility to the DevOps team so that they could iterate and experiment and not spend all their time getting developer changes into production. The migration enabled all teams to innovate but in step with each other by introducing boundaries, rules, and standards. And eventually, the development team grew to like and enjoy the new process and allowed them to push boundaries but save time.
What Changed for the DevOps Team?
A key change was that they no longer have people contacting them with deployment requests and had more free time for other meaningful tasks such as increasing monitoring, detecting usage patterns, and defining service level objectives for every service.
Measuring the Return on Investment (ROI)
Paolo measured the ROI of the change in a couple of ways. The reduced cost of external services and maintaining custom scripts, reducing downtime, and introducing internal SLAs for teams that also matched business goals.
Getting Feedback
Paolo seeks regular feedback from his developers during the migration, as well as on an ongoing basis. He generally asks them if they like new features, how they would improve them, and what they are looking for in new features.
Lessons Learned
What’s Next?
With the new platform finished, stable, and in use, next Paolo would like to investigate testing dependencies between microservices and correlating dependent merge requests.
Key Takeaways
If he could start the process again, Paolo would hope to move to an MVP more quickly and spend less time planning and researching before building one. He would also be fascinated to know how new tools and techniques could have changed their journey and process.
