Java | Software Development | Testing
Mutation testing with Pitest 
Jan Vermeir 12 Feb, 2020
Share 
Cypress focusses on writing front-end tests for web applications. Cypress can verify the contents and functionality of your application, but it cannot pixel-by-pixel tell if all things rendered as expected. For some visually complex components, in our case graphs, visual regression testing is something we need as well.
How we solved this
Cypress can create a recording of the actions you perform or on test failures. Screenshots are created manually by using the command cy.screenshot() at any point in your test script. Cypress will save this screenshot for feedback after the tests are complete, but unfortunately, it has no built-in image compare to test for visual regression.
We wanted a simple, small npm package that could do a few things:
1. Run in our Jenkins pipeline.
2. Block-out certain areas of the picture that we want to discard in our comparison.
3. Set a threshold.
4. Provide a screenshot with the difference on failure.
We choose the tool ‘Blink-Diff‘ that does all of the above. It can run from the command line, but also in a script, which made it easy to add in our CI.
We use Jenkins for our CI-tool and make use of the Jenkinsfile functionality so that all our config is in one place. Our Integration Test ‘stage’ looks something like this:
The ‘image-compare-ci’ script basically performs the standard blink-diff command ./node_modules/blink-diff/bin/blink-diff ${ignoreBlocks} --verbose --output ${diff} ${base} ${input}
Creating reliable visual regression tests
Each image you take needs to be exactly the same as the stored base-image. Image compare is often a difficult topic since these tests tend to be brittle. When no changes were made, you want to make sure that each time you run the test the screenshots you make during that specific run are exactly the same as last time. Brittle image compare tests are often the result of not having consistent, unchanging data in your app.
Cypress can help in this area as well. In my case, I needed to make screenshots of graphs. These graphs have an hour, day, week and month view. I need my test to still pass in two months time. Therefore I needed control over two things:
- My application needs to think it’s always the same date/time so it will render the data it receives on the same axis.
- Mocks with the expected data.
In a Cypress test this looks like this:
The cy.route together with the cy.clock function make it possible to very easily mock the needed test data and bring the application in the correct state. Using this method we have created a very stable way of visual regression testing.
Updating the base images
Base-images need updating when functionality changes. In our case, we decided that this should not happen often so we perform a manual update of our base image when needed. We have three folders that we use:
Within our cypress base folder we have created the following folders:
- screenshots (standard cypress folder)
- screenshots_base
- screenshots_diff
Cypress will put it’s screenshots in the screenshots folder when the tests are run. After the Cypress tests are complete blink-diff will treat this folder as it’s new images folder and compare these images against the ‘screenshots_base’ folder. Please note: the screenshots folder is in our .gitignore since we want to avoid have any file conflicts thereby pushing temporary screenshots to our CI by accident. When there are differences found, these are placed in the ‘screenshots_diff’ folder which we can access through our Jenkins workspace after our tests are complete.
Getting feedback using the Cypress dashboard
One of the cool features of Cypress that is related to the screenshots is the Cypress dashboard. Using this dashboard you can easily see the detailed results of your tests and act on them. The dashboard will show the screenshots of any failures that have occurred and it will also show other screenshots that you’ve made during your test run.
Future improvements
The Cypress team has just released version 1.0. With this release, Cypress is now open-sourced. This means we can now add our own functionalities to the Cypress project. Having visual regression testing build into Cypress will be very cool, but until this is the case, please enjoy this simple solution.
I am a specialist at Qxperts. We empower companies to deliver reliable & high-quality software. Any questions? We are here to help! www.qxperts.io
by Erik Zeedijk
As an Agile Test Consultant Erik always has the drive to improve the quality of software being delivered. This includes setting-up test automation and using CICD to get the fastest possible feedback on quality. It also means introducing the mindset change that is needed to think differently about testing and the way we measure quality. 
