Visual Regression testing with

09 Nov, 2017
Xebia Background Header Wave

Cypress focusses on writing front-end tests for web applications. Cypress can verify the contents and functionality of your application, but it cannot pixel-by-pixel tell if all things rendered as expected. For some visually complex components, in our case graphs, visual regression testing is something we need as well.

How we solved this

Cypress can create a recording of the actions you perform or on test failures. Screenshots are created manually by using the command cy.screenshot() at any point in your test script. Cypress will save this screenshot for feedback after the tests are complete, but unfortunately, it has no built-in image compare to test for visual regression.

We wanted a simple, small npm package that could do a few things:
1. Run in our Jenkins pipeline.
2. Block-out certain areas of the picture that we want to discard in our comparison.
3. Set a threshold.
4. Provide a screenshot with the difference on failure.

We choose the tool ‘Blink-Diff‘ that does all of the above. It can run from the command line, but also in a script, which made it easy to add in our CI.

blink-diff cypress visual regression

We use Jenkins for our CI-tool and make use of the Jenkinsfile functionality so that all our config is in one place. Our Integration Test ‘stage’ looks something like this:

jenkinsfile cypress visual regression

The ‘image-compare-ci’ script basically performs the standard blink-diff command ./node_modules/blink-diff/bin/blink-diff ${ignoreBlocks} --verbose --output ${diff} ${base} ${input}

Creating reliable visual regression tests

Each image you take needs to be exactly the same as the stored base-image. Image compare is often a difficult topic since these tests tend to be brittle. When no changes were made, you want to make sure that each time you run the test the screenshots you make during that specific run are exactly the same as last time. Brittle image compare tests are often the result of not having consistent, unchanging data in your app.

Cypress can help in this area as well. In my case, I needed to make screenshots of graphs. These graphs have an hour, day, week and month view. I need my test to still pass in two months time. Therefore I needed control over two things:

  1. My application needs to think it’s always the same date/time so it will render the data it receives on the same axis.
  2. Mocks with the expected data.

In a Cypress test this looks like this:

cypress visual regression test

The cy.route together with the cy.clock function make it possible to very easily mock the needed test data and bring the application in the correct state. Using this method we have created a very stable way of visual regression testing.

Updating the base images

Base-images need updating when functionality changes. In our case, we decided that this should not happen often so we perform a manual update of our base image when needed. We have three folders that we use:

Within our cypress base folder we have created the following folders:

  • screenshots (standard cypress folder)
  • screenshots_base
  • screenshots_diff

Cypress will put it’s screenshots in the screenshots folder when the tests are run. After the Cypress tests are complete blink-diff will treat this folder as it’s new images folder and compare these images against the ‘screenshots_base’ folder. Please note: the screenshots folder is in our .gitignore since we want to avoid have any file conflicts thereby pushing temporary screenshots to our CI by accident. When there are differences found, these are placed in the ‘screenshots_diff’ folder which we can access through our Jenkins workspace after our tests are complete.

Getting feedback using the Cypress dashboard

One of the cool features of Cypress that is related to the screenshots is the Cypress dashboard. Using this dashboard you can easily see the detailed results of your tests and act on them. The dashboard will show the screenshots of any failures that have occurred and it will also show other screenshots that you’ve made during your test run.

Cypress Dashboard

Future improvements

The Cypress team has just released version 1.0. With this release, Cypress is now open-sourced. This means we can now add our own functionalities to the Cypress project. Having visual regression testing build into Cypress will be very cool, but until this is the case, please enjoy this simple solution.

I am a specialist at Qxperts. We empower companies to deliver reliable & high-quality software. Any questions? We are here to help!

Erik Zeedijk
As an Agile Test Consultant Erik always has the drive to improve the quality of software being delivered. This includes setting-up test automation and using CICD to get the fastest possible feedback on quality. It also means introducing the mindset change that is needed to think differently about testing and the way we measure quality.

Get in touch with us to learn more about the subject and related solutions

Explore related posts