Spotify Backstage: Service Catalogs Explained
We talk to hundreds of engineering teams and organizations of all sizes every month. Lately, service catalogs have been coming up in conversations more and more, especially when we speak with mid or large size enterprise accounts. If you too work in a large dev organization (>300 developers), this probably comes as no surprise.
With a growing number of tools requested by different development teams and an ever-expanding base of services, big enterprise setups are characterized by an increasing lack of transparency and visibility. It’s consistently becoming harder to have a full picture of what service is running on which infrastructure, who operates it or who owns it. It’s also extremely difficult to map out similar if not identical services to avoid duplication and prevent engineers from reinventing the wheel over and over again across multiple teams.
Service catalogs like Spotify’s Backstage are establishing themselves as the best answer to these issues. By making services and their metadata easy to understand and reuse throughout the entire organization, service catalogs bring back a level of transparency and observability that most enterprise teams have long dreamed of regaining. In this blog post, we’ll discuss what these service catalogs are and how they can help your team. We’ll also look at how top performing engineering organizations use service catalog functionalities as one part of their Internal Developer Platform (IDPs) to provide their engineers with an end-to-end development and deployment experience of the highest quality.
What is a service catalog
First of all, it’s worth clarifying what we mean exactly when we are talking about a service catalog. In the DevOps and software infrastructure realm there are a few examples of similar yet different service catalogs:
- In the context of global hyperscalers like GCP and AWS, a service catalog represents the sum of all services that are available in the respective consoles a.k.a. the overwhelming amount of options you are presented with every time you open your console.
- In the Kubernetes universe there is an extension API called Service Catalog, which can be used to integrate managed services from service brokers.
- The new kid on the block: Backstage.io, an open-source project by Spotify that allows organizations to establish their own service catalog.
For the purpose of this article, we’ll discuss service catalogs like Spotify Backstage (but the same is applicable to products like LeanIX and Port), which enable enterprise teams to create an organized and curated collection of all business and information technology services and applications within an enterprise.
We define a service catalog as a means of centralizing all services that are important to the stakeholders of an organization that implements and uses it. Given its digital implementation, the service catalog acts, at a minimum, as a digital registry and a means for highly distributed enterprises to see, find, invoke, and execute services regardless of where they exist in the company. Crucially, this means that people in one part of the world can find and utilize the same services that people in other teams use on the other side of the world/enterprise, eliminating the need to develop and support local services.
Zooming in, every service catalog should have some version of these four core elements.
Ownership information and other metadata
A good service catalog contains a range of information about each service in the enterprise. This includes information such as ownership (typically pointing to a specific individual or team), programming language, source code, current version, last update, documentation. Depending on the company, additional information may be essential. This view is especially interesting for the developer or the product manager. It allows anyone in the enterprise to find out very quickly whether a certain required service is already available to then coordinate directly with the respective responsible team.
Service templating
Ops teams also use service catalogs as a way to define templates and blueprints for the rest of the engineering organization to use. This allows developers to get coding right away, using a predefined service design and language framework like Golang, Node.js, etc.
Service usage
A service catalog answers the question around which service (or fork of it) is consumed by which applications. This view is especially interesting for the team owning said service, as it makes it easy to learn about any missing functionalities or potential new features.
Service versioning
Finally, the service catalog allows Ops teams to know at a glance which versions of a particular service are used by which applications and in which environments. This is specifically useful in the event vulnerabilities are found in a given service version, as teams can be warned and only the affected environments or apps can be shut down/rolled back.
Spotify Backstage
In March 2020 Spotify announced they were releasing an open source version of their own internal service catalog, called Backstage, used by over 280 engineering teams to manage 2,000+ backend services, 300+ websites, 4,000+ data pipelines, and 200+ mobile features.
Backstage gives teams a very straightforward method to unify all of your infrastructure tooling, services, and documentation under a single, easy-to-use interface. Built around the concept of metadata YAML files, Backstage makes it easy for a single team to manage tens of services and allows a company to easily manage thousands of them. Because the system is practically self-organizing, it requires considerably less oversight from a centralized Platform team than a normal catalog would. Developers can get a uniform overview of all their software and related resources (such as server utilization, data pipelines, pull request status), regardless of how and where they are running, as well as an easy way to onboard and manage those resources.
Spotify actually said they reduced onboarding time by more than 50% since introducing Backstage internally. It is no wonder then that ever since the open source announcement, Backstage has quickly become the go-to framework for most enterprises looking to build a service catalog.
Use cases range from making documentation easier to create and consume by allowing for Markdown files alongside the actual code, all the way to better cloud cost control through enhanced visibility into each developer and team’s resource usage. Any engineer in the organization can now easily search all existing services through Backstage, consume what they need or spin up a new service with a predefined architecture and design, using the 10s of available plugins to document it, track its resource consumption and overall health or identify its dependencies.
Service catalogs and Internal Developer Platforms
Service catalogs and Backstage in particular provide enterprise teams with an incredibly useful pane of glass on top of their apps and services. At Humanitec, we often get asked how this functionality compares to that of Internal Developer Platforms (IDPs). Although some people seem to think they are mutually exclusive, IDPs and service catalogs (or Humanitec and Backstage) actually complement one another quite well.
The latest Gartner report “A Software Engineering Leader’s Guide to Improving Developer Experience” distinguishes between these two categories and explains how they interact with one another:
“Internal developer portals serve as the interface through which developers can discover and access internal developer platform capabilities.”
A service catalog like Backstage allows you to easily search all your services and immediately create an application from a template. The new service comes with a predefined design and set of metadata, depending on the specifics of your Ops or Platform team. You can get going with the coding right away, fantastic! From there, you might have a CI pipeline with GitHub Actions and JFrog Artifactory in place.
What such a setup does not allow you to do however, is running your service. The service doesn’t come with dependencies to databases, routing, storage, secrets and everything else you need to actually deploy a set of services or applications to your infrastructure. That’s where the Platform Orchestrator of the IDP comes in.
The Platform Orchestrator is the configuration engine at the core of the IDP. It lets Ops and platform teams design golden paths and enforce a Declarative Application Model. Developers use this model to specify environment agnostic configurations for their workloads (e.g. env variables, dependencies, etc.), valid across all environments and stages of the delivery pipeline.
Developers can self-serve any tech they need, like DBs, ingress, file storage and all other dependencies their apps require to run. They can also manage their own deployments, doing roll-backs and diffs, versioning configurations the same way they do with code in Git.
Combining Backstage for service discovery with Humanitec for platform orchestration, teams can achieve a new degree of Ops automation and developer self-service on all levels. Engineers can now not only one-click create a new service with all required metadata attached to it, but also one-click deploy it to a new environment, provisioned with the resources they need. And all in a context where a central Platform team can set predefined rules and golden paths for all other app development teams to operate within. Git-push or a single CLI command are the equal alternatives, depending on the preferred workflows of developers.
If you want to dive deeper check this demo on how to architect your dynamic IDP with Backstage and Humanitec:
If you have questions, talk to our sales.
Sources
“A Software Engineering Leader’s Guide to Improving Developer Experience” by Manjunath Bhat, Research VP, Software Engineering Practice at Gartner.
https://www.gartner.com/document/4017457
Disclaimer
GARTNER and HYPE CYCLE are registered trademarks and service marks of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved.
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
