This is a guest post by Rachel Kibler.
First things first: If you move to DevOps, testing requires a lot more communication and collaboration. People skills are important, and asking questions is important.
There are some other practices that are good ideas for testing in DevOps too, and if your company wants to be successful with DevOps, you should try to do most of them. Follow these steps and you’ll be on the right track.
1. Set the framework
- Determine your definition of a bug and policy for bugs. Are bugs everything found in production that deviates from expectations? Will they be fixed first or prioritized? How will bugs that require immediate engagement be decided?
- Make sure your rollback process is easy.
2. Get the whole team engaged in writing stories
- Split the stories into the smallest releasable (and testable) chunks. API calls that aren’t hooked up to the front end are fine and can be tested on their own before the UI is done.
- Review the acceptance criteria carefully. Make sure the problem being solved is actually understood. Look for, as Elisabeth Hendrickson says, absolute statements, such as software “always” or “never” doing something. These are potential landmines. Make sure everyone is on the same page and understands the approach.
- Ask questions about risk — risk to the user, security, business, whatever you can think of.
- Figure out what needs to be answered with the code change. Write some charters or other kinds of testing notes. Ask for feedback on this. Giving the developers and product owner an idea of how the change will be tested can lead to better code. By “better,” I mean cleaner, more testable code, and a smoother process for the people who need to use it and maintain it.
3. While the code is being written, there are a few things you can do
- Pair with developers. If you can get a greater understanding of the code and what the changes touch, you’ll do better testing in the end. You can also raise issues during pairing, giving your developers a better idea of what you’ll be looking at.
- Set up your test environment, if it needs it. Get some scenarios set up, and get users in the right state.
- Start writing your automated testing, if applicable.
4. When the code is ready for you, it’s time to do what everyone thinks we always do
- Review the code changes. Make sure the developer wrote whatever automated tests they’re expected to write, whether unit, integration or UI. If there are no tests, ask why.
- Using all the notes you’ve collected and all the conversations you’ve had, test the scenarios you’ve set up.
- Make sure you touch all the areas of risk. Stuff can change during development, so talk with your developer to see what areas of code the change touches.
- Run all the applicable automated tests.
- If you don’t already know this, ask your developer if they want all the bugs at once or one at a time. Depending on how they feel about context switching, they may want you to do as much testing as possible first, then tell them about the issues, or they may want to follow your progress and fix stuff as you find it.
- Add your automated tests, either in a separate branch or the same one. Make sure you’re following the same code process that your developers are held to.
5. When the code is released, you’re not done
- Check production to make sure the change got there successfully. Pipelines can be weird, and sometimes stuff gets stuck or doesn’t release properly. Depending on the stability of your pipeline, you might not see this much, but it’s worth checking. You don’t need to reproduce all your testing again; just a simple check will do it.
- Make sure you’re reviewing error logs to see if there are spikes in errors caused by the change. The amount of time you look at the logs depends on the possibility of intermittency of errors. If you released a change to login, you probably don’t need hours of review. If you released a change to a flow that users see only a little bit, set a query, and come back to it later.
There it is! DevOps is fun and worth it, and with these proven practices, you can set yourselves up for success with testing in DevOps.
- Announcing TestRail 6.6 with Enhanced Administration
- Announcing TestRail 6.5: New Plugins, Enhanced Integrations & Searchable Drop-downs
- TestRail Leads in the Fall 2020 G2 Grid Report