Service catalogs and Internal Developer Platforms: how can they work hand-in-hand?

It seems that every high-performing engineering organization is building entire suites of internal tools to improve their workflows. Internal developer platforms (IDPs), Value Stream Management (VSM) tools, and service catalogs are becoming more commonplace as tools to improve transparency and developer experience. 

But what does it all mean? What’s the difference between an IDP and a service catalog, and how do they both relate to value streams? Do you need all of these concepts to ship code reliably? 

Dominik Rose, VP of Product at LeanIX, and Kaspar von Grünberg, CEO at Humanitec, joined us in a recent webinar to answer these questions and teach us how service catalogs and IDPs can be used in tandem to enhance the development environment. 

Building software right matters

In recent years, there’s been a major evolution in software engineering concepts. More than ever before, the industry is placing greater emphasis on developer experience, internal platforms, and mapping the customer journey to technology. 

That’s because, as corporations like Amazon and new industries like fintech take up an increasingly large chunk of their respective markets, smaller businesses either need to step up their technology offerings or risk being left behind. 

However, it’s not just about improving an organization’s tech stack. Of course, investing in more modern technologies and practices are vital to compete in today’s technological economy, but cultural change is just as important to empower developers. As Dominik explains: 

“This is clearly where we need to form empowered development teams. This is something we try to follow at LeanIX on the business side. You need to think in terms of problems, not solutions, which is very much a cultural thing.” 

What is a service catalog? 

Service catalogs represent all of the services available within the development environment. Within each service “profile”, the catalog should display who owns that service, which team manages it, which projects it’s used within, and any available documentation. 

Service catalogs fit within the wider methodology of value stream management by providing a central platform that acts as a single source of truth. Ideally, these catalogs will also allow for further implementation of developer experience topics like templating, creating new applications, and automating other processes.

What is an Internal Developer Platform?

IDPs allow developers to build, deploy, and operate applications across the entire life cycle. So, service catalogs focus on categorizing, sorting, and documenting services within the development process, while IDPs work more on operating applications and systems. Kaspar describes IDPs like this: 

“IDPs are the sum of everything that gives developers the capacity to work on and operate their applications in self-service, following the ‘you build it, you run it’ mantra to reach true DevOps by reducing dependencies between development and operations teams without overwhelming cognitive load.”

With that in mind, service catalogs exist as a component of an Internal Developer Platform to provide additional functionality and further improve the developer experience. 

Both of these components are vital in the modern software industry as software engineering becomes increasingly complex. Individual apps consume 25 times more components than 10 years ago, and organizations are themselves consuming a vast amount of specialized tools to meet scale in a global environment. 

On top of this, many organizations are still binding the components needed in an application with static scripts that are, more often than not, very buggy and require a greater degree of ongoing maintenance. 

Instead, modern platform APIs and operators take those static components, template them, and dynamically create configurations that allow organizations to operate applications across the entire life cycle. Kaspar elaborates: 

“I truly believe that this is where the industry is going. The key beauty is that it eliminates case and complexity by letting developers operate applications from idea to production against a single API. It’s about how we model configurations, manage infrastructure, and how we provision and deploy everything. We’re fusing application bits together dynamically with every single deployment, and further down the life cycle, we can add visibility, locks, environments, management, and the ability to audit applications.” 

How service catalogs and IDPs fit together

In modern platforms, organizations will have a developer workflow that doesn’t materially change. There will be an IDE for coding (such as Visual Studio), a Version Control System (VCS) like GitHub or Bitbucket, and the CI pipeline to bring everything together and test the application. The Platform API then composes a model of the application dynamically at every deployment, creates configuration files, and is responsible for executing and operating the application. 

Connected to the Platform API, organizations will have certain interfaces, and this is where the service catalog exists. This allows the service catalog to execute certain automations, such as pushing pre-defined templates into the platform API and receiving the returned metadata for it to catalog and document. 

Following that, there is a self-service UI that allows the developer to operate applications. There’s also a self-service CLI to automate deployments. 

It’s worth thinking about a service catalog as a tool that forces developers to think about what services they’re using in their workflow. As Dominik explains: 

“We can assume while we have a decent catalog in place, it makes it easy for developers to jump in and select their technology. Then the platform API kicks in to provide that technology, deploy and operate the application, and then, ideally, bring it back into the business view. This means we can meet all the requirements from the business side, like understanding who owns what resource, and measuring development performance through DORA metrics and other flow data like JIRA tickets that allow us to continuously improve in a data-driven way.” 

Using a Service Catalog within an IDP 

With LeanIX’s value stream manager, developers can see from the product view who is responsible from the business side. Moving deeper into the relations explorer, developers can then link the particular microservices the product is utilizing, which allows them to quickly see the status of that service and applicable life cycle information. 

Developers can also add additional information to each service, such as which teams own each service, where that team is located, or whether they need more information regarding the people responsible for that service. 

LeanIX’s VSM also offers a development view that allows engineers to explore the services that are in use directly from the cluster. Similar to the product view, these services can also be mapped in relation to each other. 

While this doesn’t give the most in-depth view of the product, it does make this overview as easy as possible to digest by visually showing relationships between applications, services, and deployments.

LeanIX also provides a software, API, and technology catalog that allows engineers to see which items are owned by which team and what core attributes of those services are in use. The service catalog ideally should allow engineers to launch a GitHub repository with a template to start running that service, if applicable.

In this example, the engineer uses the service catalog to launch a node.js application from a template. 

This template contains all of the information that Humanitec’s Platform API needs to launch the microservice, including YAML files that define the provisioning of the infrastrucuture, DNS records, which port the application should be exposed through, and what configurations should be applied.

When we run the workflow, it will build the Docker files, push an image to the registry, execute the baseline configurations, create a new namespace, a DNS record, and configure the Platform API to set up deployment automation. This is all done automatically, abstracting a good amount of complexity away from the engineer.

Once the application is running (in this example we called it servicecatalog), then the engineer can use the UI to check the workload configuration.

In this case, you can see that the Platform API has automatically created a DNS resource and domain, which allows the engineer to review and test the running application.

Kaspar tells us: 

“Now you’ve got this fully running application - imagine how much time you’d spend going to operations and asking them to run this for you. Now the process is automated. While it needs to be indexed and documented, the platform API can signal that this new microservice has been created so it will be automatically added to the service catalog”. 

From there, the service catalog can run a static template GitHub action that gathers information covering who owns that service, the team that handles it, and what product it’s attached to. 

How service catalogs add value to an IDP

Service catalogs allow organizations to leverage technology to understand how different elements in the development process are connected together. By using a service catalog in your IDP, you can effectively create a joint business and development view of a product that allows both engineering and stakeholders to understand what components are involved in the development process. 

Not only that, but a service catalog within an IDP can use the platform API to gather data during development to produce DORA metrics and other workflow data to improve transparency within the development pipeline. By analyzing and learning from this data, organizations can improve the developer experience and continuously optimize the software engineering process. 

Thanks again to Dominik Rose and Kaspar von Grünberg for demonstrating how to use a service catalog within an IDP. If you haven’t watched the video yet, check it out here. In addition, if you want to learn more about service catalogs and how they can empower your development team, head over to our overview of Spotify’s service catalog for more information.