TestRail defines itself as a test management tool that allows you to centralize all testing activities like test design, test planning, test execution, and reporting, all in one place. On top of that, TestRail has out-of-the-box integration with a wide range of issue trackers and other mechanisms you can use to create and customize integrations with other tools.
This article will explore how you can leverage TestRail by using UI Scripts and the TestRail API to fully streamline your automated tests execution and debugging from inside TestRail. For demonstration purposes, we will use Pytest as the test automation framework, GitHub for source code management, Jenkins as the test automation orchestrator, and the TestRail CLI as an interface to the TestRail API to upload automated test results. We will give a step-by-step overview of all the necessary blocks to reach the end goal, followed by a series of articles with more detailed instructions on achieving each building block’s end result.
Step 1: Setting up a Pytest automated tests project
We’ll start by exploring a Pytest automated tests sample project, installing it, and executing the tests contained in it in your local machine. This project is hosted on TestRail’s Automation Frameworks Integration GitHub repository, so we will be using GitHub as our Source Code Management platform, from where you can fetch the project source code on demand. You should be able to quickly achieve the workflow on the image below by just installing Python on your system and executing a couple of commands to install and execute the project on your terminal. After completing this set up, you can execute the automated tests project as many times as you wish on your machine and customize it to your needs.
Step 2: Orchestrating an automated tests project with Jenkins as your CI tool
Executing automated tests on your local machine can be enough when you’re just starting, but it is not scalable and has downsides such as creating team dependencies on one person, being unable to reproduce the same project execution environment, or simply lacking visibility foryour organization. To avoid these downfalls, you should automate the orchestration of your automated tests project using a CI tool, in this case, Jenkins. We will create a job that fetches the source code from GitHub and executes the tests in it using a shell script, as you would on your local machine, ending up with a workflow similar to the first one but on a proper CI system.
Step 3: Sending automated test results to TestRail
Now that you have your tests successfully running on your CI tool, it’s time to bring your test results to light by integrating them into TestRail. To do that, we’ll leverage the TestRail CLI by using its ability to parse JUnit reports and upload their results to TestRail. First, you’ll need to configure Pytest to generate a JUnit style report, which is supported out-of-the-box and achieved through a simple command line argument. Now that Pytest is configured to generate the JUnit report, you just need to install the TestRail CLI and you can send your test results to TestRail with one simple command. You will then have achieved the workflow on the image below.
As a result of this workflow, each time the Jenkins job runs, the TestRail CLI creates a new test run with your test results on TestRail. You can see these results in your TestRail’s Test Runs & Results page and act on them as you would on any other test run, dig into test results to debug failing tests, change test results, add comments, etc.
Step 4: Triggering the Jenkins job from TestRail using UI Scripts
Lastly, we’ll use UI Scripts to close the circle by adding a widget on TestRail to trigger your Jenkins job and retrieve build information to quickly understand what has been going on with your latest builds. We provide a Jenkins Builds UI Script that is hosted on our TestRail Customizations GitHub repository. You simply change the configuration variables at the beginning of the script according to your needs and add it through the Administration > Customizations > UI Scripts page on your TestRail instance.
In TestRail, you will see that if you navigate to the Test Runs & Results page, there’s a new widget on the right-hand panel with a trigger button for the configured job and the latest builds. This widget allows you to trigger the job, check the latest builds status through the color code, and navigate to the builds by clicking on the respective links.
Final takeaways
With this workflow in place, triggering and accessing automated tests information becomes part of TestRail. Any user who is performing his daily activities in TestRail can start an automated tests suite, and receive the test results as a test run as soon as it is finished on the CI tool, without ever needing to navigate outside TestRail.
Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing information on reports. By doing so, you can get solid insights on your app’s test coverage and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would for your manual test results.
For more detailed instructions on how to implement this workflow, please see the next articles in this blog series:
- Setting up a Pytest automated tests project
- Orchestrating an automated tests project with Jenkins as your CI tool
- Sending automated test results to TestRail
- Triggering the Jenkins job from TestRail using UI Scripts
Documentation references:
