Testing Hotfixes To Production Without Rolling Back Staging

Imagine you are the head of engineering at a company called TopicShare Inc that allows companies to share internal topics and questions with each other. You are about to release a major new update when a large client in Korea reports that the current version of TopicShare running in production doesn’t render Korean characters correctly. This is a big problem for the client and their users, and makes the product mostly useless to them. You need to fix this bug.

Your current environment setup is a fairly standard development, QA, and production workflow, connected to a CI pipeline. Each of these environments is currently at different versions of TopicShare. You need to get this bug fixed as soon as possible. Ordinarily, you would release the fix to QA and then once your QA team has tested, push the release to production. However your QA team and QA environments are tied up with testing version 2.0.

You have a problem or fix needed for a major production issue, without affecting QA or similar environments first. 

What do you do?

• You could roll back the QA environment to the current production version, but this would involve rolling back (and later redoing) big database migrations and slowing down the new version’s release.
• You could deploy the fix directly to Production and then verify in Production. This is very risky and could cause more problems than the bugfix solves if a new issue was introduced by the fix!
• An alternative option is to use the production environment as the basis for a clone to test the hotfix, and after testing, deploy that clone to production. Think of it as a temporary QA environment.

Humanitec is designed for this kind of process and makes it a matter of clicks.

For example, the screenshot below is a sample organization for TopicShare with a defined Development, QA and Production environment. Each of these environments has external resources that aren’t directly part of the cluster connected to it, such as databases and DNS services. You can define these external resources (and which instances to connect to) in different environment types for your organization. You can also define if Humanitec should deploy an environment type to a new namespace in the same Kubernetes cluster or to a new cluster.

Humanitec separates out the environment specific configuration (e.g. name and credentials for the database) from non-environment specific configuration (e.g. dependencies between services). This is done by parameterizing things like environment variables so that the relevant values for an environment can be injected at deployment time. This allows a developer to take the same material configuration and deploy it into multiple environments, knowing that the right thing will happen in each environment.

In our case, Humanitec allows you to clone the deployment that is running on Production and deploy this clone to a brand new environment. This environment will be materially the same as Production, but without any shared resources, so you can safely use it for debugging.

Behind the scenes, Humanitec creates a new namespace in the configured kubernetes cluster and  creates the resources that the application depends on based on the template defined by the environment type.

The clone at this stage will contain exactly the configuration from Production - including the bug. So it is easy to validate that the bug can be reproduced in this new environment. The version of the code containing the bug fix can be updated and then re-deployed in this new environment. Before every deployment, a diff will show the changes between the previous and the new version.

With these changes deployed to the cloned environment, you can test them, verify the hotfix and check that nothing else has broken. Once testing is complete in the temporary environment, the environment can be "cloned" back to Production. Again a diff will show what the material changes are.

With the hotfix now live in production, all that is left is making sure that the hotfix is incorporated back into your mainline codebase. For this you just follow your normal development workflow.

Fixing production

With the issue identified, fixed, and live, your Korean client has a working product and is much happier. Behind the scenes, your team continued to work on, and test new features, with none of the environments they were using affected. 

Humanitec added a layer of flexibility to the development setup, to allow the team to create and destroy environments based on other environments as needed to suit their needs.

Why don’t you try it yourself? Start a free trial or join a webinar with our DevOps experts.