SpotCheck

Maintain a screenshot baseline on your web application, right from Azure DevOps through the SpotCheck Azure DevOps Extensions. This extension is publicly maintained on GitHub.

Once installed, it enables an extra task within pipelines, as well as an extra UI tab for those pipelines. Through some minimal configuration, maintaining a visual basline of E2E tests becomes very accessible.

Usage

Generate screenshots through cypress or Robot Framework.

Use cypress-image-diff-js

describe('Visuals', () => {
  it('Visit Google', () => {
    cy.visit('www.google.com')
    cy.compareSnapshot('home-page', 0.1)
  })
})

Or a custom keyword in Robot Framework

*** Test Cases ***
Visit Example
  Go To  http://www.example.com
  Wait Until Element Is Enabled  //h1
  Compare Snapshot  example_homepage  0.1

*** Keywords ***
Compare Snapshot
  [Arguments]  ${FileName}  ${Threshold}=0.1
  Capture Page Screenshot  ${FileName}.png

Generate the Cypress or Robot report within a pipeline, then use SpotCheck to process the report.

Process Robot report:

- task: SpotCheck@1
  inputs:
    input: '$(Build.Repository.LocalPath)/Automation/robot-output.xml'

Process Cypress Image Diff report:

- task: SpotCheck@1
  inputs:
    input: '$(Build.Repository.LocalPath)/Automation/cypress-image-diff.json'
    mode: 'cypress'

Process Robot report with screenshots in a subfolder without posting comments on the related pull request.

- task: SpotCheck@1
  inputs:
    input: '$(Build.Repository.LocalPath)/Automation/robot-output.xml'
    baseDir: '$(Build.Repository.LocalPath)/Automation/screenshots'
    skipFeedback: true

Differences against the baseline are reported through a dedicated threadon the PR. When a difference is resolved, the thread will automatically resolve.

The visual differences are presented on an extra tab in the build pipeline, to which the PR will link as well.

It only requires a few clicks to update the baseline, if a change is found to be legitimate.