Organizations are striving to release features faster than ever, to meet growing customer demands, and stay relevant in the industry. The market is highly competitive and teams are looking into lean approaches to deliver high-quality products. Various agile approaches have been adopted to support this effort and one of them that has gained popularity in recent years is DevOps.
What is DevOps?
Historically, development and testing efforts were viewed as two separate entities. This model proved to have several bottlenecks. Some of which included:
- Gaps in communication between the groups, leading to defects
- Incorrect understanding of the implemented features
- Not having clarity on test coverage
- No visibility into each other’s work
- Unnecessary friction due to lack of collaboration
- Unstable features pushed to QA
And most of all, no one had any idea of how the customers were using the deployed product. They did not communicate directly with the operations team monitoring the application in production, and vital statistics that could further improve the product weren’t gathered. This wasted incalculable amounts of time, effort, and cost for organizations and ultimately led to a delay in releasing the features on time and resulted in a decline in customer satisfaction.
DevOps was introduced to mitigate the above problems and reintroduce the mentality that quality is everyone’s responsibility. The main focus of this approach is to increase collaboration between the developer, tester, and operations teams, so they can work as one single unit to deliver high-quality products.
Role of QA in DevOps
In other agile approaches, the developer is in charge of coding and creating the feature and writing and running unit tests; testers perform automated and manual tests on this feature. DevOps purposefully blurs the lines between these responsibilities, forcing collaboration. The developers start building the mindset of continually checking for errors in their code. The testers increase their responsibilities from validating the application, to ensuring it is deployable at all times. They may even fix code as needed. All these pieces work together to ensure rapid delivery of features. The focus on the customer drives the work of the entire team.
In DevOps, the testing perspective changes from “completed testing on this module” to “What are the business risks that have been mitigated in the release candidate?” They move beyond manual testing and focus on scalable deployments, automation right from the beginning of the development process, standardizing QA environments, aligning QA tasks with Continuous Testing, and CI/CD pipelines.
A high level of collaboration between developers, testers, and the operations team enables testers to monitor the application in production and use the data to help stakeholders make informed decisions.
How to integrate QA into a DevOps environment?
DevOps does not replace the existing QA process. Instead, it shifts testers’ viewpoint from the traditional way of testing, such as mapping test cases to requirements, executing test cases manually, and reporting defects; to a more agile way of delivering software, where QA is part of a large team instead of being a separate entity.
Integrating QA into DevOps
Here are some ways to integrate QA into the DevOps environment:
Re-align team structure
First, organizations should remove the boundaries between developers, QA, and operations teams. This would involve restructuring the existing QA team to align with the DevOps goals.
Teams have to merge into one technical team that focuses on various activities to deliver the features on time. These technical teams have to communicate, collaborate and optimize their efforts to find defects fast and fix them before deployment. Also, testers should get involved early in the development process, starting at the requirements phase (where they can flesh out any ambiguity in requirements) by ensuring the whole team has clarity on how the feature will be developed, tested, deployed, and monitored post-launch. Continuous testing becomes part of everyone’s responsibility.
Automate as much as possible
Secondly, DevOps and automation go hand-in-hand—without one, the other won’t work. This is where smart people and tools can help in bringing shorter and more dependable release cycles. Everything that can be automated should be automated. This may include unit, deployment, integration, smoke, regression, security, and performance tests. Testers play an integral part in the DevOps process by starting automation early in the process which finds defects sooner than later.
Also, for automation to be effective, you should modularize your test suites and run them at various phases of the development process based on the features that have to be tested. Finally, as new features are added to the product, the test suites should be flexible enough to scale according to complexity. This may include the need for additional software, hardware, and skilled resources to support the effort.
Use metrics to track progress
Another important factor determining the success of QA integration into the DevOps process is having metrics that can quantify the value. A survey conducted by Forrester Research found 75 common metrics that DevOps teams use to measure the success of the adoption. Some of them that directly relate to the quality of the products are as follows:
- Test case coverage
- Unit test coverage
- Pass/fail rate
- Number of tests executed
- API pass/fail rate
- Requirements covered by tests
- Blocked test cases
- Percentage of automated test cases
- Successful code builds
- New defects
- Critical defects
- Automated tests prioritized by risks
- Code coverage
- Release deadlines
- Total number of defects discovered in test
- New API defects found
Using these metrics, stakeholders can make informed decisions about how QA can effectively collaborate with the developers, operations team, and other roles in the project.
Run tests in parallel
Testers run tests in parallel to various activities in the DevOps process. For parallel testing to be effective, QA engineers need to set up standardized test environments and have sufficient hardware and software resources to run many tests as fast as possible. They may also have periodic audits to ensure the testing process is optimized to finish within the software delivery time.
Continuous monitoring
DevOps involves multiple tools, processes, automated tests, and environments. Data is extracted from different stages of the development process and analyzed to ensure the system is performing at the optimum level at all times. Usually, the operations team works with developers and testers to build monitoring and analytics capabilities right into the developed application. It is vital for testers to know how to use tools like Splunk, Kibana, Graphite, and Firebase that are used for this purpose to support the operations team.
Have a common set of processes and tools
One of the biggest problems in organizations is each team follows a different process and uses a distinct set of tools to accomplish the same task. When this goes unnoticed, it leads to unnecessary wastage of cost, time, and effort for the organization.
With DevOps, the teams collaboratively have to agree upon a common set of processes and tools everyone will follow. Once decided, each team member will know what process to follow and tools to use at any given time in the development process.
Continuous feedback
Organizations must emphasize the importance of receiving continuous feedback from the team during the implementation. Daily standups, team meetings, retrospective meetings, and planning meetings are great examples of touchpoints to gather feedback. Based on this, teams will modify their approach for the next iteration. This continuous cycle of build->measure->learn (validated learning) helps to improve the DevOps process.
Validated Learning
Increase visibility into tasks and processes
For a successful DevOps implementation, teams have to collaborate and have visibility into each other’s work. At any instant in time, teams should be able to visually see the progress of the development process. Organizations can use tools and dashboards that highlight vital metrics such as team velocity, defects logged/fixed, number of tests executed, test coverage, and other KPIs important to the team.
Sufficient training
Testers’ skills need to adapt to the DevOps process. It starts by training everyone to do full-stack test automation, which includes writing automated tests such as unit tests, API tests, UI tests, and back-end tests. Manual testing will still be relevant, but the focus will shift from only executing test cases manually to doing more risk-based testing and exploratory testing. This is performed in parallel to automated tests.
Test strategy and test design become even more critical in the DevOps process. Since testers are working along with developers and operation teams, their test strategy will have to include tasks related to QA and areas that cut across development and the operations teams. A good approach to come up with a solid test strategy and design tests is for different roles to work together collaboratively. This helps the team gain a common understanding of what needs to be tested, who will be performing them, and how the tests will be executed.
Finally, to successfully transition into DevOps, organizations should implement three essential processes: continuous testing, integration, and delivery.
The three pillars of DevOps
The main goal of DevOps is to increase collaboration. This is not achievable without having Continuous Testing, Continuous Integration, and Continuous Delivery. Testers play a vital part in each of these processes.
1. Continuous testing (CT)
CT is all about testing an application from its early stage and automating tests throughout the lifecycle of the development process. This helps the team to analyze the quality of the product at every stage of the process and enables stakeholders to make informed decisions.
CT also reduces the risk of shipping features that break in production, saving a considerable amount of time, cost, and effort for the company. The key is to test as early as possible and find defects as soon as possible.
Here are some things to consider when integrating CT with your current testing process:
Have sufficient resources
If testing has to be performed in each stage of the development process, the teams need to have sufficient resources: hardware, software, and skilled engineers. The team collaboratively decides on a test strategy and procures the necessary resources before implementing CT.
Align with the Shift-Left paradigm
CT aligns with Shift-Left testing, focusing on starting testing early right from the requirements phase and continuing throughout the development and release process. A part of this alignment is to expose developers and testers to different activities to empower them. For example, a developer would pay more attention to writing better unit tests, and testers can get involved in development activities. This helps to speed up the development cycles and builds cross-functional teams that can adapt to any environment.
Identify gaps in the existing test process
Before implementing CT, do an audit of your current test strategy. Get a handle on your current test coverage, the type of testing performed – manual scripted test execution, automated and exploratory tests. Identify your current bottlenecks, analyze trends in production bugs and figure out how you can mitigate them with CT.
Use automation wisely
One of the common problems teams face when implementing CT is trying to automate everything, even if it does not provide value and could prove to be a maintenance nightmare in the long run. For example, automating features with a high state of flux could result in inefficient use of resources as they may spend more time maintaining the test than running them. Use automation when repeating mundane tasks, getting faster feedback on builds, deploying builds to different environments, and performing various kinds of testing that would otherwise be hard to do or consume time when done manually.
Have the right automation tools
For effective implementation of CT, teams have to invest in using the right tools for various activities. This could include tools for unit testing, UI testing, API testing, performance testing, security testing, and accessibility testing.
2. Continuous Integration (CI)
CI is a primary DevOps best practice of automating the integration of code changes into a central repository where it builds and runs several times a day. One of the key benefits of implementing CI is detecting errors quickly and locating them more easily.
There are various benefits for QA in implementing a CI pipeline:
- Remove dependency on code check-ins
Any number of people can check in code multiple times a day into the version control system. There is no dependency, and each check-in has timestamps to know who wrote the code exactly and pushed it to a particular branch. This means testers and developers can both write application and automation code and check it into different branches without any interference.
- Faster feedback
Once the code is checked in, tests automatically run to get quick feedback on the build, so errors can be found quickly and fixed as soon as possible. It enables testers to quickly figure out what the problem is, report it, and take the necessary action to address and fix the defect.
- Increased collaboration
CI improves overall engineering communication and accountability, enabling greater collaboration between development, testing and operations in a DevOps team. Pull requests are tied into the CI process, helping to gain visibility into code from other team members. This enables developers and testers to collaborate on feature branches as the feature progresses through the CI pipeline. The automated tests have to follow the same pull request process.
- Automated regression testing
As part of the CI process, testers configure regression tests to run when the build is deployed to different environments automatically. This ensures the new changes do not break existing functionalities. These tests are set up to run as quickly as possible to get faster feedback on the deployment.
- Extensive documentation
When automated tests run in every stage of the CI pipeline, extensive logs are generated with timestamps for future reference. This is especially useful for organizations that have to adhere to regulatory compliance. The logs serve as a rich source of information during routine audits to ensure there is no breach of policies and procedures.
3. Continuous Delivery (CD)
One of the integral parts of DevOps is to deliver features rapidly. CD enables this to happen by giving teams the ability to release the application at any instant of time. Beyond making sure your application passes automated tests, it has to have all the configuration necessary to push it into production. CD encompasses everything from ideation to build to readiness to delivery to production.
The role of QA in a CD pipeline is to ensure continuous testing happens throughout the development process. Testers use various tools for smoke testing, regression testing, API testing, UI testing, performance, and security testing. Tools free up their time to do exploratory testing and risk-based testing to mitigate identified risks. Also, having a CI environment ensures automated checks happen in various phases of the build process, giving faster feedback on the quality of the features.
Key tools needed for DevOps implementation
Since DevOps includes continuous testing, integration, deployment, and delivery, many tools have to be used to get quick feedback about the application.
Below are some commonly used tools in a DevOps pipeline:
- Source code repository: Git, Assembla, CloudForce, TFS, Subversion
- Build server: SonarQube, Jenkins, Artifactory, Travis CI, Circle CI, Kubernetes, Docker
- Feature flags: Rollout, LaunchDarkly
- Unit testing: JUnit, Nunit, RSpect, TestNG, Jasmine, Mocha
- Configuration management: Puppet, Ansible, Salt, Chef
- Test automation: Selenium, Appium, Watir, Ranorex, Cypress, Espresso
- Test management: TestRail, Jira, Zephyr, Xray, PractiTest
- Monitoring tools: Splunk, Kibana, Graphite, Crashlytics
- Virtual infrastructure: Amazon Web Services, Microsoft Azure, VMware vCloud
- Non-functional security tests – Mittn, Gauntlt, BDD Security
- Static analysis security testing tools – Kiuwan, Coverity, Checkmarx and other paid tools. Open-source options include Reshift, Brakeman, JSHint, NodeJsScan.
- Security Scanners – Netspaker, Acunetix, Wireshark, Nessus, OWASP ZAP (Open Source)
As you can see, various tools can be used for different purposes in the DevOps pipeline. A good approach to selecting the best tools that work in your project context is by keeping the below factors in mind:
- Choose tools that increase collaboration
Almost every tool used in the DevOps pipeline should facilitate effective collaboration between different team members. For example – Version control is a basic necessity in any agile project, including one that follows DevOps. The tool used for this purpose should help in the seamless integration of code, configuration files, scripts, and other artifacts across teams and different systems.
- Tools should have open APIs
As technology advances rapidly, new tools come into the picture to solve existing problems. So, teams tend to shift to these tools. Migrating to new frameworks and tools should be assumed normal in teams following a DevOps process.
It is important to use tools that have open APIs so that even if the team migrates to a newer tool, teams can still use the existing scripts by calling the APIs with minimal disruption. Don’t get stuck with a single tool; instead, think about how easily any tool you choose can integrate with other tools through the use of APIs.
- Ease of use
A DevOps team consists of people in different roles, including developers, testers, operations teams, business analysts, product managers, project managers, and other stakeholders. Any tool you choose should be able to be used by everyone without a steep learning curve. If a tool becomes too tricky to learn, people get intimidated and stop using them. When collaboration is the key aspect of DevOps it is critical to pay attention to how easy it is to use by people of various technical and domain backgrounds.
QA is an integral part of implementing a robust DevOps strategy. Faster releases and delivery cycles will not happen without a comprehensive QA-DevOps transition plan. The role of QA teams has shifted in the world of DevOps. Instead of individual contributors, they have become QA consultants. Testers will have to shift their mindset from providing traditional testing services to helping deliver quality software by mitigating risks across roles, including the development and operations team. They have more opportunities to teach people about the importance of quality as they will work closely with different roles. There will be no longer a separate QA team; instead, there will be technical teams that combine different roles.
QA teams will be forced to look for new ways to add value which will help to bring out their critical thinking skills. This could mean taking ownership and making some strategic decisions that they may not have done when working in traditional QA teams. There will be more focus on thinking of the software from an end-user perspective and how customers would interact with it daily.
One of the biggest obstacles to DevOps implementation is the process itself. It involves shifting the attitude of not only QA but all roles in a team. This takes a considerable amount of effort, and higher standards, and involves more tools and resources. Implement DevOps the right way and reap the benefits (that far outweigh the initial struggles!).
