US Bank

Infrastructure and tooling setup

While their cloud transformation is still ongoing, the majority of workloads are now containerized (73% at the time of writing). The organization runs decentralized and doesn’t “dictate” the use of technologies to individual teams. Clusters are running on-prem with Openshift and in the cloud with Azure AKS as compute. Teams run several different CI setups, the majority leveraging Azure DevOps, Jenkins, Travis and CircleCI. Image registries and binary storage are predominantly done with JFrog, with minor usage of Harbour, ACR and Docker Registry. DNS is done with Cloudflare, APM predominantly with New Relic. Databses include mysql, MariaDB, MongoDB Atlas. S3 is used for file storage.  They use Azure Service Bus. The setup was represented as infrastructure as code with Terraform leveraging libraries such as Terragrunt.

‍

Key challenges

Due to the competitiveness of the financial sector it became increasingly hard to find enough Ops talent to deal with the growing setup complexity. Developers were overwhelmed by the amount of tools they needed to navigate just to do a single deployment. Rate of innovation was stalling and management started asking questions.

• Kubernetes numbness: developers were overwhelmed trying to understand how to operate the setup on Kubernetes at scale.
• Blocked developers: engineers waited up to one week for a fresh environment or infrastructure. This led to significant delays in deployment.
• Frustrated Ops: the Ops team was completely overwhelmed by the accumultaing overhead and request backlog from application developers.
• Slow deployments: the bank’s deployment rate was lagging. as dependencies on several people in the process slowed everyone down.

“At the end of every quarter we realized that we wasted our time doing ticket-ops for internal app developers. “Frustrating” describes it well.”

Head of Infrastructure

Key improvements

By building their Internal Developer Platform with Humanitec, the customer managed to streamline the provisioning of infrastructure resources and application configurations to enable developer self-service at scale. Golden paths helped security teams enforce best practices, while unblocking developers.

• Unlock Ops: by automating frustrating, repetitive ticket workflows, the team could concentrate on optimizing their setup to reach true DevOps.
• Unblock developers: thanks to self-service, developers can go fast without breaking things. App configurations work by default.
• Scale securely: nudging teams to stay on the “golden path” enforces best practices around secrets management and parameterization.

“I was always a believer of “you build it, you run it”. With Humanitec this is a reality for the first time in my career”.

EVP Delivery

Humanitec erased bottlenecks and dependencies, reduced pressure on operations, simplified maintenance and reduced waiting times. Deployment frequency skyrocketed and the change failure rate dropped.

Technical deep dive

Infrastructure orchestration before and with Humanitec

Before building their Internal Developer Platform with Humanitec, the customer’s setup was static. If a developer required a new infrastructure component, they had to request that from a central Ops team. Ops had to handle a whole array of different IaC tools. This approach led to bottlenecks and frustrated both Ops and developers. After building their IDP with Humanitec, developers at the bank self-serve the tech they need, following clear governance. Rather than executing IaC statically, developers request resources through the self-service UI or CLI. The Ops team codify what resources are used using the Platform API. Their existing IaC setup is wrapped into Open Source Drivers that are executed by the Platform API to put the respective resource into the correct state and wire it up to the correct microservice.

“The idea of providing golden paths rather than golden cages is key. Humanitec helped us to build a platform that didn’t restrict developers but enabled them. “

CTO

App config management

Before Humanitec the team used Helm charts to manage application configurations. They ended up with 140 microservices with 140 materially different configurations. As the application life-time increased, the versioning became harder to handle. Roll-backs took ages and the change failure rate increased. The team lost a significant amount of time trying to build a GitOps setup but was frustrated by the amount of complexity involved in debugging and understanding failed deployments.

With Humanitec, the Ops team sets baseline templates that contain any default the securtiy team wants to enforce. Developers can apply changes to these templates through the CLI or UI. At deployment time, the platform API creates a fresh set of manifests including the environment specific elements (DBs, DNS, etc.), saves them to the repo in Github and executes them against the AKS API. Manifests are versioned, increasing visibility and allowing for easy rollbacks or diffs

“When we started looking at Humanitec we thought we had a huge replatforming ahead of us. It was a matter of weeks in the end with immedeate returns.”

CTO

Final setup

Individual teams still use their respective CI setups but those are signaling builds to the Platform API. All image registries are wired up to the Platform API as well. The API deals with RBAC, creates application configurations per deployment and call the correct open source driver at the correct request. Developers self-serve deployments, resources, logs and more through the developer self-service UI or CLI.

Timeline and evaluation

• POC: 3 weeks
• Evaluated against a self-built setup. Estimated annual cost of maintenance and development: 4 Mio USD.
• Migration: 15 weeks
• Onboarding per new developer: 30 minutes