Integrate with Azure DevOps

Run an Azure DevOps task to get Octomind test results into your pull request comments

CI/CD pipeline integration

1. Create an API key

The option to create your API key can be found within the settings menu.

Account settings menu icon, 5/2025

Click on Create an API key.

Create an API key button, 05/2025

You then will be navigated to our auth provider where you can create a new organization-level API key.

Create a new Octomind API key in the settings, 05/2025

Copy the Octomind API key

You can only copy the API key once, so make sure to do that now. You should not share it with anyone.

Add the API key to your CI provider as a secret to run test reports from the CI.

2. Copy your test target ID

You can get your test target ID from the octomind landing page once signed-in. If you are greeted with the 'Setup your autonomous end-to-end tests' message, please sign-in first

The option to copy your test target ID can be found within the settings menu.

project settings, 08/2024

Click on the copy test target id.

Copy test target ID

3. Install Azure DevOps pipeline extension

Install our Azure DevOps pipeline extension from the Marketplace.

Install from here: pipelines extension

Octomind Azure install, screenshot 06/2023

4. Include the Azure DevOps Task

Include the following Azure DevOps Task into your CI/CD build pipeline on pull request. We assume you are running branch deployments so that every change can be tested in isolation. You will have to inject the publicly accessible URL of your branch deployment and the API key via the azure task.

- task: automagicallyexecute@2
  inputs:
    url: <publicly accessible url to your deployment>
    token: ${{ secrets.AUTOMAGICALLY_TOKEN }}
    testTargetId: <your testTargetId that you also get from us>
    blocking: <if you want your pipeline to wait for all tests to pass>
    environmentName: <if you want to run tests on an environment other than default>

Please make sure to integrate the AzureDevOps Task AFTER your deployment step.

The Azure DevOps Task is published here: source code

5. Set user permissions

Make sure that your build service user also has permissions to comment to pull requests. To do so, navigate to the Project Settings > Repos > Repositories > Security tab and allow the ‘contribute to pull request’ flag.

Octomind Azure pipeline permissions, screenshot 06/2023

6. Allow comments

During installation, we’ll ask you to grant access to your repository with write access to pull request threads, so we are allowed to comment back into the pipeline. It will list the test results with additional links to more details, looking like this ↓

Example of Octomind test results in a pull request comment, screenshot 07/2023

We won’t need any other permissions. We won’t be able to see your code.