One of the biggest things that 2021 taught us is that DevOps is absolutely vital in today’s technological landscape. Even though the DevOps methodology has been around for over 10 years, many companies are only just starting to introduce this to their engineering teams - and there really is no time like the present.
As technology advances, so does the complexity we load onto our engineers, but with the friction they face throughout the development pipeline there’s little wonder that organizations lose talent to competitors with a greater emphasis on developer experience.
However, by enhancing our DevOps setups and using Internal Developer Platforms (IDPs) to help provide full developer self-service, with abstractions to take away cognitive load and allow engineers to focus on the things that truly matter, we can start to
Steve Pereira, the founder of Visible, joined us in December 2021 to talk about how using an Internal Developer Platform, or IDP, can help organizations to take their DevOps setup to the next level.
The State of Play
In 2021, organizations globally started to introduce more microservices into their development pipelines, particularly using platforms like Kubernetes and cloud providers to help them to manage their dependencies. However, this hasn’t been without its pitfalls.
As complexity has increased, so has the amount of friction and waste within organizations. Engineers are increasingly having to make tough decisions about which tools to use and when or reinventing the wheel every time they can’t access a service or tool they need. Instead of being able to see the big picture, engineers aren’t given the chance to see how their work has meaning.
“I refer to this as Tactical Myopia - it’s not being able to see the bigger picture and understand what you’re doing is helping to move things forward in a meaningful way”, explains Steve.
On the flip side, many organizations are choosing to forego external tools in favor of building and hosting everything in-house. While this can have its benefits, the majority of the time, it leaves engineers feeling frustrated as their time is taken up developing internal tools than working on meaningful projects.
So, where do organizations go from here? And, more importantly, how can using an IDP help to supercharge DevOps so these problems can be eliminated?
Avoiding Vanity DevOps
The biggest mistake most organizations make is that they believe that DevOps is everything within their development pipelines. In these cases, organizations become so focused on building and introducing a DevOps setup that they don’t take the time to understand where bottlenecks exist, where automation is needed, and whether this methodology will have any meaningful impact on their engineers.
The truth is that people will always want to work with tools and platforms that they already know. It makes it easier to estimate deadlines because they know where those bottlenecks lie. When a new platform or tool is introduced, the unknowns make it difficult to provide those estimates.
This is a common pitfall when you introduce an IDP, but it can be easily circumvented.
Steve says “You need to show them something so easy and beneficial that it’s worth the cost of switching”. Whenever organizations introduce a new tool into the engineering environment, there will always be a cost associated with switching over. Engineers will have to learn how to use this new tool in their workflow and solve any issues that come with adjusting the tech stack to use something new.
Not only is it important for the benefits of an IDP to outweigh that cost, but engineers also need to trust that the inevitable initial struggle will be worth it. To do this, organizations need to get influential people to use the platform so they can talk about how it works from a first-hand perspective. Social proof is a powerful tool, and it’s one that organizations need to get teams on board with using their IDP.
Complexity
“The more we can help people understand how to do something, the more we can give people an easy way to get stuff done. Then, they can spend their cognitive load on real problems instead of figuring out how to do their job,” explains Steve.
By using an IDP, organizations can connect and integrate every tool within their DevOps setup in a meaningful and productive way. Engineers can see every tool they need in a specific, purpose-built interface that follows a set standard, so they don’t have to reinvent the wheel every time they need to get something done with a specific tool or platform.
Friction
Another major stumbling block in DevOps is automating the CI/CD pipeline and neglecting all other parts of the value stream. While automation in these areas is extremely valuable, it can often lead to organizations failing to tackle roadblocks up or downstream of that specific pipeline.
“Something I see all the time is that constraints aren’t always in the place we expect them. When you have data, though, you can see where things are slowing down, where quality is being impacted, and where you’re spending too much time on less valuable tasks. By using an IDP, you can give teams that information and let them do what they want with that” adds Steve.
The truth is that constraints will move all the time throughout the development process, so using data helps teams to understand what’s influencing those bottlenecks and what needs automating to reduce friction throughout the pipeline.
Not only does using an IDP make it easier to abstract away complexity, but it can also provide valuable data to demonstrate whether those abstractions are helping or hindering the development pipeline.
Scale
In DevOps, it’s common for shared services teams to become a bottleneck, particularly as organizations scale and add more product teams. If every product team is reliant on a shared services team, that team will likely spend all of their time fixing everyone’s problems instead of developing new tools and improving the development process.
Many organizations believe that true DevOps lies in having a DevOps engineer on every product team, but that’s an expensive endeavor. In comparison, investing in building an IDP that gives every engineer the ability to run some services on their own is significantly cheaper and carries far more benefits for the organization.
However, Steve cautions, “Don’t think about throwing everything away and building everything new.” Instead, to help organizations scale, he says they should consider this: “Connect your pipeline and infrastructure, automate everything you can, and make provisioning new infrastructure so easy that no-one needs to learn the language that automation is run in.”
Not only does using an IDP for automation make things easier for engineers, but it gives organizations a lot more control over how infrastructure is provisioned. IDPs allow managers to set limits and control who can access what resources, as well as enforce tagging of resources so the network doesn’t get overwhelmed.
Value
For engineers, it’s vital to know what tasks are valuable so they know how much energy to invest into completing that task. Otherwise, they end up gold-plating custom-built tools that won’t be utilized, particularly if third-party tools offer the same features executed in a far more polished way.
Similarly, adds Steve, “we have to try and convince people to use things in DevOps that we haven’t proven to be valuable, and it can be painful”.
Using an IDP as part of a DevOps setup helps engineers to understand both what value they bring to the business and the customer, and what tasks are worthy of gold-plating to bring that value. However, as Steve mentions, IDPs need to be of significant value to the engineers for them to have any meaningful impact on the developer experience.
This means building easy-on ramps so engineers don’t have to struggle to fit it into their workflow, learning resources to teach them how to navigate the platform, and building in the ability to scale so it can grow as the platform team takes on feedback from engineers.
Most importantly, it also means treating an IDP as a valuable product within DevOps and giving it the same consideration the organization would to products for external customers.
5 Reasons that IDPs are Next-Level DevOps: In Summary
As we move into 2022, we need to be aware of the ever-present trap of vanity DevOps and understand that introducing this methodology for the sake of saying it’s there can often cause more harm than good.
Fortunately, if you’ve fallen into this hole, IDPs offer the solution. Instead of staying stuck with an inefficient DevOps setup, building and introducing a meaningful IDP can transform your DevOps setup into a high-performance pipeline.
Steve says that “managing complexity is a superpower of IDPs”, which is why they’re crucial to any high-performing DevOps setup.
IDPs aren’t just about providing an interface for managing abstractions. A well-managed IDP greatly improves developer experience by reducing cognitive load and providing data to help them manage their workloads.
Not only that, but IDPs can easily scale to add in new tools and abstractions when they’re properly managed by a platform team, allowing you to build a platform that can make engineers’ lives easier for years to come.
Steve has some fantastic insights into using IDPs in DevOps, so make sure you check out the webinar if you haven’t already.
