About the healthcare provider
The provider is one of the largest healthcare providers in the world and is known for its comprehensive coverage, high quality care, and commitment to patient welfare. It plays a vital role in the health and wellbeing of the UK population, and its services are highly valued by patients and healthcare professionals alike.
Infrastructure and tooling setup
The provider was in the middle of a enterprise wide transformation from on-prem with Openshift into the public cloud. While the provider had a wide range of cloud resource types in use, the main applications were hosted on AWS with EKS. They primarily use GithubActions for CI and JFrog Artifactory as a container registry. They operate a wide range of databases from MongoDB to Postgres. The primary APM provider was Datadog, Hashicorp Vault was used as a secrets manager. Spotify’s Backstage was deployed as a developer portal. Cloudflare was the primary provider of DNS. Before adopting Humanitec, the provider managed their applications as configuration as code, primarily with Helm. All resources were represented as “IaC” with Terraform for their Infrastructure configurations.
Key challenges
A year into the migration, the progress was mediocre. 90% of workloads were still running on the Openshift installation, while the guarantees given to the public cloud provider with AWS kept costing the firm large amounts of money. The migration stalled because developers were overwhelmed by the sheer number of tools, interfaces and configuration formats they now had to operate. At the same time security policies had to be rolled out but took longer and longer to be enforced. This significantly slowed down delivery times as developers were distracted from their work or had to wait for operation teams. As a consequence, the provider experienced a heavy increase in lead time.
The overwhelmed developers created a lot of work for the operations team as they constantly needed help to understand, debug, and operate their application and infrastructure configurations. The operations team would have had to add more members to keep up, which was not possible because of budget pressure. As the situation came to a tipping point, a developer and operator mistake led to a day-long outage and a security incident that cost the organization a 7-digit number and trust in the market.
- Ops bottleneck: key person dependencies slowed down overall delivery. Waiting times for databases, environments and other resources blocked development.
- Low degree of standardization: missing guardrails and config standards put a heavy burden on security teams that needed to repetitively check deliveries from application developers.
- Poor standardization: varying configuration formats and missing guardrails made it hard to drive standardization which in turn drove maintenance overhead.
- In summary delivery times dropped significantly and the provider was outpaced in feature development by its competitors.
"We had a huge commitment on AWS which wasn’t being used and velocity massively dropped. The Internal Developer Platform we built really got us out of trouble."
CIO
Platform architecture
The organization had already decided to build an Internal Developer Platform. After thorough evaluation of the available options, the team decided to use Humanitec to ensure the highest enterprise standards. Key reasons were the significantly lower total cost of ownership, as well as the speed and predictability in delivery.
Given the regulatory background the provider had to remain cloud agnostic. They designed their platform following Humanitec’s multi cloud reference architecture across Openshift and AWS, which allowed them to reduce dependencies on either player. Given the size of the engineering organization, the platform had to support several CI, registry and secret providers, The configuration of the Orchestrator was done as code, using the Humanitec Terraform Provider. The platform team configured the baseline-configurations for applications and built reusable infrastructure templates using Humanitec’s open source drivers. The IaC approach of choice was Terraform, although some teams used Cloudformation and Pulumi, which had to be supported too.
To cater to different types of teams and users, the platform team exposed an array of interfaces to developers and left them with the choice on a workload to workload level. The primary interaction was code-based using the workload specification Score to keep developers in their tested git-push workflow. A service catalogue with Atlassian “Compass” acted as a user interface on top of the Internal Developer Platform.
Key improvements
Building an enterprise-grade IDP with Humanitec allowed the platform and operations teams to define clear golden paths and enable developer self-service. Rather than having to navigate dozens of different file formats and tools, developers were requesting and managing workloads and resources through unified interfaces. Staying code-first prevented long onboarding cycles.
Developer self-service led to a significant reduction in tickets to operations. The first 50 AWS projects created had generated a total of 1,000 tickets from developers in the first year. With a target of 500 AWS projects, the amount of tickets would have become unmanageable . Building the Internal Developer Platform with Humanitec led to a decrease in tickets from developers. While the cloud footprint grew 10x, tickets only increased to 2,000, or an 80% reduction.
- 80% reduction of repetitive tickets to operations.
- 30% reduction in lead time.
- 4 times faster migration by application.
“I’m sometimes wondering what would have happened if we hadn’t done this.”
SVP Operations
Timeline and evaluation
- POC: 3 months
- Evaluated against a self-built setup 12-24 months.
- Total Migration: 5 months
- Onboarding per new developer: 30 minutes
