The Azure DevOps integration plugin provides an out-of-the-box integration of step with the Azure Test Plans module of Azure DevOps.
With the Azure DevOps plugin you create, design and manage your automated test cases in Azure Test Plans, run them on step’s Execution Engine, and evaluate the run results and test reports at the same place in Azure Test Plans. This plugins is therefore the perfect solution for users requiring the best of the 2 worlds: the unique power of the step Execution Engine with the test management and reporting features of Azure Test Plans.
The test case design relies on the standard plain text syntax of STEP (The same as used by the other integration plugins like ALM and Jira).
Test Case Definition in Azure DevOps
Test Cases can be defined directly in Azure DevOps, using step’s plain text syntax.
In the “Expected result” column, you can enter assertions which are expected to hold.
Running Tests in step
To execute tests stored in Azure DevOps, start by selecting the “Azure DevOps” menu entry in the hamburger menu:
You will be presented with a window where you can browse repositories, plans, suites, and test cases.
From there, you can launch entire Test Suites, or individual Test Cases.
Run Evaluation in Azure DevOps
After tests were run in step, the results are exported back to Azure DevOps, where they can be found under the “Runs” menu item:
Installation and Configuration
In order to interact with Azure DevOps, step must have the permission to do so. This is achieved by creating an Access Token which has at least full read/write access to the “Test Management” functionality:
In step, the integration is configured using the step.properties file on the controller.
There are two kinds of entries to create.
This configuration key contains a space-separated list of organization/project combinations that can be interacted with.
For example, if your project’s root URL on Azure devops is https://dev.azure.com/myorganization/myproject , the corresponding configuration entry is:
repository.azuredevops.projects = myorganization/myproject
If you wish to access multiple projects (possibly even in a different organization), add additional entries, like so:
repository.azuredevops.projects = myorganization/myproject myorganization/otherproject anotherorg/testproject
For each organization that is to be accessed, add a configuration entry with the organization name and the respective access token.
repository.azuredevops.token.myorganization = thetokenformyorganizationabcdefghijlkmnopqrstuvwxyza
Starting with step version 3.17.2, a number of further configuration options are available. These include:
- Specifying whether, and how, the numerical DevOps IDs of test plans, suites, and cases, are to be displayed while browsing them in the step UI (default: disabled)
- Specifying the name that will be appended to test runs in the DevOps UI (default: STEP)
- Specifying the format of the attached reports (HTML and/or plain-text, default: HTML)
- Controlling whether the original test step formatting is to be preserved in reports (default: false)
Please refer to the
step.properties file for further documentation on configuring these options.
Disabling reporting for executions
By default, all executions of test suites or cases defined in Azure DevOps are also reported back to DevOps as Test Runs. This may not be desirable in all cases.
Starting with step version 3.17.2, it is therefore possible to define an execution parameter which allows to control the reporting.
The execution parameter MUST have the ID
azure_devops_reporting and have a boolean value (true/false). Here is an example on how to define it:
Known Issues and Limitations
There are a number of limitations imposed by Azure DevOps. Below, we list the ones we are currently aware of:
Access Tokens are bound to a specific Azure DevOps User. Executions will always be shown as belonging to that user. It is therefore recommended to use a technical user for that purpose, if possible.
Access Tokens have a maximum allowed lifetime of 12 months. After the token expires, a new one must be created and the configuration must be updated accordingly.
Running tests, evaluating test results
It is unfortunately not possible to directly launch test runs in step from Azure DevOps. In other words, tests must be initiated from within step using the “DevOps browser” functionality.
Test runs are always reported as “Manual”. To mitigate this, we append “(STEP)” (or the value configured in the configuration file) to the names of the runs which were executed in step.
Reporting in Azure DevOps is limited in a number of ways.
Test steps which were run repeatedly can only show up once in the report.
Only a very short “comment” field without support for formatting is available.
For these reasons, step attaches reports (in a configurable format, by default HTML) to each Test Case that was run, providing more detailed information.
- Executing nested test suites recursively is not supported.
- “Shared steps” are not supported.