SHARE THIS POST

So often I hear someone mention how they would like to be able to release on a dime. Their team completed a feature and they want to get it to the customer right away to get feedback! Unfortunately, in some organizations, they have to wait until a scheduled release to get their work out. For some agile organizations, that means every two weeks. It’s certainly better than the days of quarterly releases (ouch that hurt to even type and sorry if that’s still you) but not where they’d like it. The dream is for most organizations to be able to successfully master what I’m describing in my opening sentence: Continuous Integration, Continuous Delivery (CI/CD). CI/CD probably isn’t a concept I’m introducing to you for the first time. There are quite literally thousands of articles, books, and blogs on CI/CD. That’s just it though, why do CI and CD get all the fame? What about its lesser-known brother, Continuous Testing (CT)? I want to spend this blog talking about Continuous Testing and how you, as testers, can help define and implement CT as a part of your DevOps strategy at your organization.

Before we start swirling around and throwing out CIs, CDs, and CTs (oh my!) let’s start with common definitions for the three. Continuous Integration is described simply as developers making small code changes and frequently checking those into a shared repository. Continuous Delivery is an extension of CI. When the code is checked in, automated processes test and push the code to the defined environments. Continuous Testing is the enabling process of executing your automated tests throughout your CI/CD pipeline to ensure code should be pushed through. The days of you battling the “its a small one-line change, no testing required” fight are over!

What Does Continuous Testing Actually Look Like For a Tester?

In a Continuous Testing world, a developer would make a code change and push it to your source repo where unit tests would automatically run. However, instead of someone manually deploying that code to a testing environment (without any knowledge of the state of that build), another set of automated tests would run; if they pass, it deploys! If they don’t, alert the masses! This code-build-test-deploy process could be done nightly with automation in place. When you come into the office (or, because quarantine, pull up to your home office desk) each day you know there is a freshly deployed build waiting for you that has gone through a pre-defined level of validations. Call me biased but, CT is the glue that holds all of it together.

I want to pause and talk to my manual testers for a brief moment. If you’re reading this and thinking, that’s a lot of mentions of automation, where do I fit in? Let me assure you. In this world, manual testers are even more important: automation can’t (and shouldn’t) mimic the clever tricks you use to find bugs. They can’t ad-hoc or exploratory test. Automation is meant for a set of mundane, calculated scripts that run on command. Moving forward, where varying sets of automated scripts give the go-ahead for the code to progress, it’s even more important that manual testers are involved and trusted to test new features coming through and help to maintain a solid regression suite.

Continuous Testing enables you, the tester, to get feedback to your developers as quickly as you can. Traditionally, testing is completed as the final step, before deployment. Continuously validating code throughout the lifecycle allows the team to get feedback quickly and find bugs earlier in the development cycle.

How can a Quality Engineer Impact A Continuous Testing Strategy?

No matter the title you have, we can all be leaders – just look at you here, reading blogs and learning stuff. Leaders help mold and drive strategy and as QA leaders it’s our job to align QA strategy with the business’s overall goals. Spoiler alert, getting new features to customers as quickly as possible is the goal, and rightfully so.

If your DevOps team spends countless hours automating every portion of your delivery pipeline only to realize manual testing of critical test paths is still required, well, you may have some disappointment on your hands. Ensuring quality is key; its why we are here and valued. Let’s dive in and discuss how you might be able to better prepare your CT strategy so when CI/CD comes a-knocking, you’re happy to let it in.

Here are some points your continuous test strategy should account for:

• Has the performance of your application met your baseline performance standards? Performance degradation can creep up over time. Accounting for it in each deployment is paramount in ensuring you avoid this. Regular performance tests should still occur. This isn’t a replacement for them, but more like a small check-in.
• Have your business-critical test paths been accounted for? Can someone select a product and give you money for it? Can they sign up for your newsletter? Whatever your business-critical paths are, ensure they’re covered.
• Is the new code being deployed accounted for? Don’t release a new feature without adequate coverage!
• Do your automated scripts manage for test data?
• Do you have the appropriate test environments set up to account for this testing?
• Does your pipeline have adequate alerting if a defect is found?
• Do your tests act as a stop gate and limit bad code from getting further down the pipe?
• How will you maintain your scripts? False failures could be a killer for continuous anything!
• Do your scripts run in a sufficient amount of time to enable your company’s CI/CD dream to become a reality? The key is understanding the delicate balance of what needs to be tested in each phase. If you go all in and execute 2500 scripts, your continuous testing is more like regression.
• Do upfront research on what tools are right for your organization and QA team. Be prepared to present those to your leadership group to ensure QA is accounted for when planning the CI/CD strategy. Open source automation tools such as Selenium and Appium allow you to quickly test across platforms but there are an array of tools that might make a better fit.

Conclusion           

If your IT leaders came to you today and told you they wanted to start delivering software to customers daily, would QA be an enabler or a blocker of that initiative? In the end, it’s about quality, or it should be. Don’t sacrifice a quality product to implement a CI/CD strategy your QA team isn’t prepared for. Start strategizing now; what will you do in the coming months to better prepare your organization for this?

Share your thoughts in the comments below:

• Has your organization already implemented a similar strategy? If so, share some lessons learned to the group. How has this changed your role as a manual, automation, or performance engineer?
• What steps can you take now to ensure you’re ready for this shift?
• What other things do you think your CT strategy should account for?

About the Author

Johanna South has been actively testing or evangelizing software testing concepts for a little over a decade. She has worked in various roles throughout the Software Development Lifecycle, including manual and automation testing, systems analyst, and leading full-stack software development teams. Her broad understanding of delivering quality software gives her a unique perspective in which to pull ideas.  Johanna is currently a Sr. Director of Engineering, Mobile Applications for TwinSpires.com. Also, she leads her organization’s QA Center of Excellence and is responsible for ensuring application performance across multiple platforms.

Her passion for both encouraging everyone to understand the value of quality, and learning new techniques to be a successful tester, have led her to author a blog: theQAconnection.wordpress.com. You can find Johanna on LinkedIn and Twitter @theQAConnection).