Hey there,
Greetings from Garz, the Temnitztal salutes you and hopes you are in best shape! Today we’re talking about platform engineering architectural design. And the best way to go about it.
My son caught a mouse yesterday. It explicitly asks me to extend its regards and wishes you all the best on your journey! I have a little piece of paper next to me, often this year's tax filing. In Germany it has A LOT of pages which makes it perfect to take notes on the back of it. Anyhow, I use this to take notes on what approaches to platform engineering different teams take if it comes to architectural design. And every time I end a call with a customer I note down the distribution. Wherever possible I try to get the info a year later how successful they’ve been. Is this a scientifically watertight method? Absolutely not! Is there other research or do I have the time to do it properly? Also absolutely not. So you and I are left with my tax filing notes. And here we go:
There are essentially only two paths that teams take. Group one is starting frontend first, group two is starting backend first. Observing the dynamics group 1 is usually managed by a manager with little to no technical experience. They’re falling for the fallacy “if I see something it becomes more productive”. If you’re in the “frontend first” group and you really like software engineering you should change your job. If that’s not possible at the moment you can at least try to choose the right path…
If you’re in the frontend first group you again have two sub-options:
1. Use an OSS
- First results: months
- Degree of automation: low
- Degree of standardization: low
- Examples: Backstage
- Pro: easy to sell to management
- Con: low adoption, high overhead
- Return on investment: negative
2. Use a vendor
- First results: weeks
- Degree of automation: medium
- Degree of standardization: low
- Examples: Port/Cortex/Humanitec
- Pros: low cost of ownership
- Cons: limited adoption rate
- Return on investment: Very low
If you have to start frontend first at least remember Jane Austin: “It is a truth universally acknowledged that an organization in possession of a good fortune should not go the OSS path”. Let me quickly do some math for you: keeping Backstage up and running fully loaded for a team of only 300 developers will cost you a minimum of USD 400,000 if you’re honest with yourself. Can you do napkin math for me how you are ever going to amortize this? How much faster does it make “finding stuff” for the average person? How often would your developer need to use this every day? And why would they?
Let’s turn to group 2, the backenders. And now we’ll get controversial because there’s a new kid in town and the traditionalists revolt. So let’s start with traditional backend design for platforms: pipelines. You heard it right. 90% of the platform engineering teams today use pipelines to design their business logic. They use GH actions or Argo workflows or one of the 10,000 other CNCF projects. The problem is that pipelines are a terrible way to execute complex logic with changing desired outcomes. Pipelines are used for linear progression, building an image, running integration tests etc. But not for complex logic handling. The problem is not that it isn’t possible, the problem is that the amount of code required to handle the degree of complexity is growing exponentially with the number of workloads and resources handled. Pipelines scale well to 50 engineers, they require significant workforce between 50-200 developers and they just don’t really scale beyond. Â
3. Pipeline based
- First results: months
- Degree of automation: medium
- Degree of standardization: medium
- Pros: often already in place
- Cons: scale hard/maintenance
- Return on investment: low
Which brings us to the frontrunner group: graph based backends! That essentially means you are dynamically generating an acyclic resource graph of workloads and their dependent resources. You generate this from abstract requests from developers against general rules provided by the platform team. This is new, you’re not used to it and tired of learning new stuff. But mark my words, this is what’s going to prevail and be the standard in 5 years.
4. Graph based
- First results: weeks
- Degree of automation: high
- Degree of standardization: high
- Pros: scales easy/low maintenance
- Cons: requires new skills
- Return on investment: high
I wrote a little blog post on how to design backends, I hope you enjoy it and I would love to be criticized.
I would also invite you to Luca’s webinar on this topic. He will go into extensive detail, and show you how to get started.
‍
Cheerio
Kaspar
‍
Btw - this group is producing some pretty dope sounds, I thought you might enjoy them.
