Automating deployment validation with quality gates
Automatic scoring and grading of builds, artifacts, test results, feature flags, canary deployments, and even full-blown releases must be part of every modern progressive delivery process. These capabilities, which are built into the Keptn continuous delivery process, are called Keptn “quality gates.” While quality gates are a built-in Keptn capability, they can also be used as a standalone feature and thereby integrated with any other existing deployment tool or release process. Read on for my explanation of what a quality gate is, what’s needed to define a quality gate, and how to use a quality gate.
What is a quality gate?
A quality gate is basically the scoring and grading of an artifact based on predefined quality criteria. These criteria can be, for example, response time, failure rate, or throughput of an application or microservice. In the process of releasing a new version of your microservice, a quality gate makes sure that the predefined quality criteria are satisfied, and if not, the new version is not pushed further through the deployment pipeline.
When you consider that quality gates must be reevaluated with each new deployment of a release (or on an ad-hoc basis), it’s clear that automation is key to success. Look at the image below taken from Thomas Steinmaurer’s Continuous Performance Validation — you can see that extensive evaluation, spanning multiple metrics, is performed daily. This allows metrics to be compared not just to fixed thresholds, but also to single or multiple previous runs, allowing for more sophisticated analysis.
How to define a quality gate?
In Keptn, defining a quality gate is a simple matter of configuration, starting with the definition of Service-Level Indicators (SLIs), which are the metrics you want to evaluate. Common SLIs include Response Time, Throughput, or Failure Rate of a service. More advanced monitoring tools such as Dynatrace also provide architectural or scalability metrics such as Number of Service Calls or Number of Database Calls. These metrics can be split by service endpoints, business transactions, or test transaction names. SLIs in Keptn are an abstraction from the actual data provider, which makes them easy to use and understand without having to get into the details of where the data comes from. The actual implementation has been influenced by other projects such as Kayenta from Google and Netflix, the Indicator protocol from Pivotal, and Pitometer (now called Lighthouse), which was based on the Unbreakable Pipeline from Andi Grabner. We’ll dig deeper into the technical details in a follow-up article.
Once you know your important SLIs, you can specify your Service Level Objectives (SLOs) for the SLIs. Some companies have hard-coded SLOs, for example, requiring the response time for a specific microservice to be faster than 100 ms. Another example might be that the number of database queries for the Login transaction should never increase between two releases of the software. Specific SLOs are highly dependent on the services that they’re used for, and Keptn gives you the means and extended capabilities to define SLOs.
Note that SLOs for a service can include multiple objectives, and you’re not limited to validation against a single metric. In fact, Keptn gathers all SLO metrics automatically and performs a grading based on the scores for each metric. This means that if a single objective doesn’t meet the defined quality criteria, but all other objectives are fine, the service might be still able to pass the quality gate. This is just a matter of configuration and might differ substantially between different microservices. As an example, you might want to define that the response time must be faster than 100 ms, and the memory consumption should be lower than 200 MB, but unless memory consumption exceeds the boundary of 350 MB, the service can still pass the quality gate. The bottom line is that defining a quality gate is quite easy.
How to evaluate the quality gate?
The SLIs and SLOs you’ve defined can now be leveraged by Keptn. You can either use the Keptn CLI or the Keptn REST API, depending on the use case. If you’re a developer and want an ad-hoc evaluation of a new version of a microservice, the Keptn CLI might be the right choice. Alternatively, to integrate the quality gates with an already existing pipeline or other external tools, the Keptn API might be the better choice.
Either way, triggering a Keptn quality gate starts an evaluation of the given SLIs in a defined timeframe. As mentioned above, the quality gate, in terms of SLOs, can comprise multiple metrics to be evaluated, scored, and graded. The result can then be used to decide if the microservice should be promoted. Not only does Keptn give you the final “pass” or “fail” result that’s been calculated, it returns a detailed score sheet including all evaluated SLIs together with the results of their evaluation.
For example, the following image shows the evaluation results for two different builds retrieved from the Keptn bridge (aka, the Keptn UI), which provides full observability into Keptn internals, not just the evaluation of quality gates.
As you can see, the quality gate on the left for build number 1.2.3 passed with a total score of 100 % as all SLOs meet the criteria.
For the new build 1.2.4, the quality gate evaluation shows a total score of 0 % and therefore the evaluation result is “fail.” A closer look at the evaluation criteria shows that both criteria were violated, as the measured values exceed the thresholds. With this quality gate in place, we can ensure that builds that do not meet our quality criteria won’t make it into production.
Getting started!
You can get started quite easily! Keptn provides tutorials for all its use cases. Head over to the Quick Start section of Keptn help to set up your Keptn installation and begin validating so that your deployments never crash production again.
