About the bank
A multi-national bank based out of the USA is a financial institution that operates in multiple countries, with its headquarters located in the United States of America. This type of bank provides a wide range of financial services to both individuals and businesses, including personal and commercial banking, investment management, wealth management, and corporate finance. Due to its global presence, a multi-national bank based in the USA has a diverse customer base and is subject to different regulatory frameworks in each country where it operates. This requires the bank to have a strong compliance and risk management culture to maintain its reputation and mitigate potential risks.
Infrastructure and tooling setup
The provider was in the middle of a enterprise wide transformation from on-prem with Openshift into the public cloud. While the provider had a wide range of cloud resource types in use, the main applications were hosted on GCP with GKE. They primarily use Jenkins for CI and JFrog Artifactory as a container registry. They operate a wide range of databases from MongoDB to Postgres. The primary APM provider was Datadog, Hashicorp Vault was used as a secrets manager. Spotify’s Backstage was deployed as a developer portal. Cloudflare was the primary provider of DNS. Before adopting Humanitec, the provider managed their applications as configuration as code, primarily with Helm. All resources were represented as “IaC” with Terraform for their Infrastructure configurations.
Key challenges
The development team at the bank faced a significant problem with their developer experience. The team was incredibly frustrated with the management and operation of a wide array of different tools, standards, and interfaces, which led to their inability to release software. The team had to repeatedly ping operations teams for help, and the overall productivity heavily decreased. These problems also had a major impact on the lead time and slowed innovation.
The underlying problem was the lack of standardization in configurations, tools, and interfaces. This lack of consistency led to a situation where the development team struggled to maintain the various systems and tools they used, leading to a loss of productivity and increased frustration.
The bank failed to focus on standardizing their configurations, tools, and interfaces, and the development team continued to face these challenges. The overall performance of the bank was affected. It was crucial that the bank took immediate steps to address this problem and ensure that their development team had access to standardized tools and interfaces they needed to work efficiently.
- Interface and standard sprawl: missing guardrails and config standards put a heavy burden on developers.
- Bad documentation: the reality in financial institutions is that heavy alterations to the provider configurations are necessary. If those changes aren’t documented properly this is making it hard for teams to operate.
- Poor standardization: varying configuration formats and missing guardrails made it hard to drive standardization which in turn drove maintenance overhead.
- In summary delivery times dropped significantly and the provider was outpaced in feature development by its competitors.
"We had a huge commitment on AWS which wasn’t being used and velocity massively dropped. The Internal Developer Platform we built really got us out of trouble."
CIO
Platform architecture
The organization made the decision to build an Internal Developer Platform and after a thorough evaluation of available options, the team chose to use Humanitec’s products to bind their existing tech and tools into a consistent platform. The key reasons for choosing Humanitec were the degree of standardization that the Platform Orchestrator brings to configurations as well as the community of Score as a workload specification. In addition to that the lower total cost of ownership, as well as the speed and predictability in delivery played a role.
To comply with the regulatory background, the provider had to remain cloud-agnostic. They followed Humanitec's multi-cloud reference architecture across Openshift and GCP to reduce dependencies on either player. Using Dynamic Configuration Management the Platform Orchestrator is now able to create the contextualized configurations following the latest policies.. As the engineering organization was sizeable, the platform had to support several CI, registry, and secret providers. The Orchestrator configuration was done as code using the Humanitec Terraform Provider. The platform team configured baseline configurations for applications and built reusable infrastructure templates using Humanitec's open-source drivers. Although the IaC Terraform approach was the choice, Cloudformation and Pulumi were also supported.
To meet the needs of different types of teams and users, the platform team exposed various interfaces to developers and left them with the choice on a workload to workload level. The primary interaction was code-based, using the workload specification Score to keep developers in their tested git-push workflow. A service catalog with Spotify's "Backstage" acted as a user interface on top of the Internal Developer Platform.
Key improvements
Humanitec's IDP enabled the platform and operations teams to establish clear golden paths unifying the developer experience primarily against Score as a workload specification. this allowed developers to manage workloads and resources through unified interfaces, without having to navigate multiple file formats and tools. By remaining code-first, the IDP prevented long onboarding cycles.
This developer self-service approach led to a significant decrease in tickets to operations. In the first year, the first 50 GCP projects generated a total of 1,000 tickets from developers. If the target of 500 GCP projects was achieved, the amount of tickets would have been unmanageable. However, building the Internal Developer Platform with Humanitec resulted in a reduction in tickets from developers. Despite a 10x increase in cloud footprint, tickets only increased to 4000, which is an 64% reduction.
- 64% reduction of repetitive tickets to operations.
- 28% reduction in lead time.
- 3 times faster migration by application.
“Let’s say “it was a good idea.”
SVP Operations
Timeline and evaluation
- POC: 3 months
- Evaluated against a self-built setup 12-24 months.
- Total integration: 5 months
- Onboarding per new developer: 30 minutes
