Alright, you’ve been in DevOps hell for a while now. You and your team are constantly flooded with tickets and Slack messages, while developers are wasting time waiting for your help. There’s a massive lack of standardization across deployments, workflows, apps, and infra config. Too much time is being spent on maintenance, and you’re battling daily with complex infrastructure and a multitude of tools that are hard to integrate.
At this point you’re likely on the verge of DevOps burnout. But perhaps you’ve read and heard multiple times that platform engineering could be your way out of DevOps hell. So you and your team are now looking to start your platform engineering journey and build an Internal Developer Platform (IDP). Finally, a solution to solve all your DevOps problems and free you from being the Ops bottleneck for the engineering organisation. Happy days, paradise here we come.
Or, maybe your organization has already embarked on your platform engineering journey and is fast escaping DevOps hell. But it turns out building an enterprise-grade IDP is trickier than you thought. Plus you didn’t follow best practice, you didn’t treat your platform as a product, and you even decided your team was going to build EVERYTHING end to end. All of a sudden you’re in a new type of hell: The platform engineering hell.
So you find yourself wondering, will my life just be a constant series of hell after hell?
Maybe, but it doesn’t have to. The challenge here is that many engineering organizations trade the false promise of DevOps (e.g. everyone can build and run everything, on their own, on top of an increasingly complex cloud-native toolchain) for another not-so-great idea. The let-me-build-everything-on-my-own idea.
As engineers, we love to build. We want to solve problems and build sh*t. And that’s cool. But as an engineering organization it’s very important to ask ourselves where we draw the line. At what point are we crossing into the danger zone of reinventing the wheel just for the sake of it?
The question is, what are the key (i.e. truly value-adding) activities we should focus on as a platform engineering team when building our IDP? What are the components we can take from the market, whether open source or commercial, and just tweak vs perpetually seeking out the secret sauce needed to build entirely in-house?
In this blog post, we zoom in on the core engine of an enterprise-grade IDP. We discuss what it would take to build something like the Humanitec Platform Orchestrator on your own and where and when that might make sense for your organization.
Building an enterprise-grade Internal Developer Platform
First of all, you might wonder why you need a Platform Orchestrator in the first place. Or even before that, why build an IDP at all, when you can buy one. For the latter, I’d point you to our build vs buy an IDP article. The TLDR is that you can’t buy an IDP. That’s called a PaaS (Platform as a Service), and a PaaS just doesn’t scale to the needs of enterprises, ever.
So why do you need a Platform Orchestrator? The truth is, you can build an IDP without one. You could for example stitch together some Terraform and ArgoCD for infrastructure and app configs, as well as deployments. You then add a portal as the top layer, something like Backstage which acts as an interface for developers, and off you go. While that will provide some good functionality, it won’t scale well. In fact, it’ll likely create a lot of pain for your platform and Ops teams down the line. Why? Because it’s still a static, infrastructure-centric approach. One that requires a developer to maintain separate config files per environment and manage infrastructure autonomously, or outsource to other teams. Which in turn causes ticket ops and bottlenecks.
Such static setups can quickly (depending on number of engineers, apps, and how fast you grow) lead to config drift. Teams start creating new workloads by copying the structure of the config from an existing workload, and then customize it to fit their needs. Makes sense, but over time these customizations compound, resulting in subtle config differences between workloads. This makes your setup increasingly hard to maintain and your IDP becomes more of a bottleneck than an enabler. In short, you end up with a static platform unable to serve an enterprise team at scale.
If you want to build an enterprise-grade IDP, you need a Platform Orchestrator.
How does the Platform Orchestrator work?
Unlike the static setup described above, the Humanitec Platform Orchestrator promotes a dynamic approach to configuration management. To be precise:
“Dynamic Configuration Management (DCM) is a methodology used to structure the configuration of compute workloads. Developers create workload specifications, describing everything their workloads need to run successfully. The specification is then used to dynamically create the configuration, to deploy the workload in a specific environment. With DCM, developers do not need to define or maintain any environment-specific configuration for their workloads.”
DCM simply means that developers describe how their workload relates to resources using a workload specification like Score. The specification is generalized and works across environments. In the example below, the score.yaml file is read by the Platform Orchestrator and matched to the configuration baselines (for app and infra configurations) defined by the platform team. The Platform Orchestrator then creates the resources or connects to existing ones, and deploys the workloads or hands them over to existing deployment tools (e.g. ArgoCD). This is the Read, Match, Create, Deploy pattern followed by the Platform Orchestrator.
As both app and infra configs are generated with every deployment, the number of files a team needs to maintain and operate is reduced by up to 95% (For example, any app with ten services and dependencies across four environments requires 300+ config files in the static approach vs ten files when managed with the Platform Orchestrator). Because these files are generated with each deployment, they are highly standardized, which in turn leads to improved auditability and security. The Platform Orchestrator also enables a fine-grained RBAC, which lets the platform team control roles and permissions across the entire organization and application lifecycle.
That is how the Platform Orchestrator makes your IDP truly enterprise-grade.
Build a Platform Orchestrator
If you’re part of an enterprise team, at this point you probably agree that having a Platform Orchestrator to dynamically generate and manage configs is essential to scale your platform engineering efforts. But as we said, we are engineers. So what does it take to build this bad boy?
First, it requires research. Your team and product manager will have to get very comfortable with the nuances of some foundational concepts:
- Resource management and resource dependencies
- Environment specific vs. environment-agnostic variables
- Context-based configuration (depending on environment type)
- Separation of concerns (developers vs. Ops, product teams vs platform teams)
- RBAC design and implementation on multiple levels, e.g. org, app, and environment type level
Building a Platform Orchestrator also requires a full product team (including a product manager) in addition to the existing platform team. Who by the way, won’t be able to simply add the Platform Orchestrator design and implementation to the backlog of tasks they’re already working on. You’ll have to hire senior talent with in-depth expertise in different platform engineering domains, and also consider the different integrations that must be built and maintained.
Depending where you are on this double-hell journey, it could be very tricky finding the time to build something of this scale. Most platform engineering teams in the industry are already up against backlogs. If you do manage to reserve adequate resources, it’ll usually take at least 24 months to build a Platform Orchestrator (or enable an equivalent degree of Dynamic Configuration Management). Any wrong architectural decision on the five domains bulleted above can easily cost you months and hundreds of thousands of dollars to reverse.
Finally, and this is a part that most teams seem to forget about, maintenance and ongoing development will be very costly and time-consuming. Many engineers get very excited at the prospect of building something new. But most find having to provide a robust and secure product that keeps iterating with developer feedback a total buzz kill.
Building a Platform Orchestrator can certainly be done, but it’s an expensive project and takes time. So again the question should be, is this a critical piece of business logic that needs to be developed in-house for your specific needs? Or can you tweak existing solutions so you only need to work on last-mile optimization for your specific use cases, instead of starting from scratch?
The fastest way to build your IDP
Using the Humanitec Platform Orchestrator is the fastest and most reliable way to build an enterprise-grade Internal Developer Platform (IDP). It takes on average eight to twelve weeks, to roll out across multiple teams, instead of at least 24 months to build one from scratch. The Humanitec Platform Orchestrator is reliably successful with an NPS score of 60 across all our customers and CSAT score of 90. Developers love platforms built with Humanitec because they can choose their preferred level of abstraction (i.e. how much context they want) and their ideal interface (code-based with Score, or the UI, CLI or API). Platform teams can define conventions and golden paths once and focus on optimizing the IDP to your needs, instead of reinventing the wheel.
Our Platform Orchestrator is built on learnings from hundreds of engineering teams, all with differing complexities and edge cases. This means our tool is way more resilient than an in-house solution which will likely over-optimize on the current use case, but be incompatible with different technologies and stacks in the future (e.g. In the case of tech paradigm shifts, or simply, an acquisition or merger). Our Platform Orchestrator integrates with pretty much everything on the market today and it’s easily extensible through our library of OSS Drivers.
Roast test the Humanitec Platform Orchestrator
You know what team we’re on. But hopefully this article is a fair description of the challenges to expect when building a Platform Orchestrator in-house to make your platform enterprise-grade IDP.
At Humanitec we’ve been strong advocates of enterprises building their own IDPs. You simply can’t buy an IDP that works in a complex enterprise setup at scale. But that doesn’t mean you should build every single component from scratch. The data from the latest 2023 DevOps Benchmarking Report confirms this, with only 16.2% of top-performing platform engineering organizations building their IDP completely on their own. In contrast, the vast majority of top performers used a combination of open-source and commercial solutions.
Where are your time and resources best spent? On building foundations and standards that are consistent across different engineering organizations? Or on last mile optimization for those special requirements that are truly unique to your needs and setup? If you feel you need therapy after this, get in touch. Our platform therapists are here for you.
