It's the year 2023. We now live in a world of AI Chat , AI image composition and other impacting technologies. The world is moving faster and faster. And somehow, we as Test Engineers or Developers have a role to play, which often has to do with giving our customers software they can rely on. It is as simple as that and therefore we bring you new insights to regularly challenge your fast feedback loop.
To rely on the software we build, we test. To scale these tests and to make them repeatable (read: not affected by the absence of caffeine) we automate them.
Within the world of the test automation frameworks, Cypress promises super fast and helpful feedback on the tests. Because Cypress can run in 'UI-mode', you get live coverage on how your test is running. CNN couldn't have done it better ;) After each test command the 'before state' and the 'after state' of the DOM can be inspected, which helps a lot when you want to inspect the application under test.
From my experience I can say that while Cypress has a speedy testrunner, there can still be reasons why your tests run slow. This article covers some reasons and how to mitigate them, so you can receive the fast feedback you deserve.
So 👇 a list of reasons why your Cypress Testsuite could be slower than needed. Use this list as a checklist to see where you can improve.
- [ ] Latest Cypress Version
- [ ] Analyse (keep your project in shape)
- [ ] Automatic waiting
- [ ] Parallelisation
- [ ] Loading of plugins
- [ ] Cypress Greb (smoke test)
Cypress VersionMake sure you are always running latest Cypress Version. I have seen projects where this was not the case and projects where more than 1 major version 'behind', which already can make a real difference when talking about a fast feedback loop! In some of these cases, a major - upgrade was responsible for observing faster tests because of fixes that had to do with the startup time of Cypress.
One suggestion here to make is to make use of a 'bot' that automates all your dependency updates for you, ie. Renovate or Dependabot, which takes care of updating your Cypress dependencies within your package.json as well as the Cypress Docker image you should be using in your pipeline definition file.
Analyse (keep your project in shape)
While analysing your test output can be a whole blog on itself (note to self: write a blog about test analysis), let's cover a short version here.
In my experience, I have seen many times that when the test code has been written once, people do not look at it anymore. They focus on what is at hand, the feature they are working on, and forget that there is a whole world of code, pipeline output or metrics that are not learned from. This can 'harm' your fast feedback loop in such a way that it might be even faster then it already is.
The focus I try to bring into a team is to plan time (lets say 1 hour a week) to analyse the projects output (pipelines, metrics, test output) to learn if things are still working as expected.
Circling back to the Cypress tests; this means going back to the spec files colleagues wrote and see if it still makes sense. What do I see ? Do I still understand what is going on within the tests? (if not: what can I learn from this?) Are the right 'best' practices in place? How fast is my test suite running? Could it run faster? Analyse, Analyse, Analyse to understand.
Maybe there is a spec file that has a cy.wait command, but now that you learned more about your application or about Cypress you are able to approach the test in a different way which could prevent you using the cy.wait command. And now, every time the test runs (may it be in the Cypress GUI or in CI), there is no more need to wait a fixed time.
Oh, and did you know you can change the timeout of tests, commands and your complete testsuite?
The default waiting time for a Cypress command is 4000 milliseconds. The example below shows a specific command timeout being overwritten to 10000 milliseconds.
Some prefer extending the default timeout within a specific test. This can be done like this:
It is possible to set the default command timeout in the Cypress Config file. In this way the whole testsuite will be affected. Except of course the timeout settings you set within a specific spec file
Another part of analysing the tests is to check in the Cypress Cloud whether tests have become flaky. When the same tests often depend on the retry-ability of Cypress, you could call this is 'smell'. The test could be flaky because of a regression in the code. The Cypress Cloud has some nice features for the analyses, like 'slowest tests', 'top failures' and 'flaky tests'.
Mocking / Stubbing
With the Cypress Command cy.intercept you can mock a call to an API. By mocking your api you do not have to have a running API, which is often a backend service with a database, and you do not have to seed a database for your tests. This saves you a lot of time while running your tests and therefore you maintain a fast feedback loop
An example of such a mock:
This example intercepts a post to an api and returns a body object with a given key / value. This call will never hit the actual api, so there is no need to set up that API (service and database) for testing purposes and you can resume your test with the returned response.
When stubbing your API calls, one should be aware that there should be tests in place that covers the connection with the API. When stubbing the API, your tests are not aware of any API changes that can occur. Therefore you can have tests in place that detects these changes.
There are different approaches that covers the topic on how to implement a proper test strategy. Here are some links to that:
- Testing Trophy - Kent C Dodds
- Gleb Bahmutov
- Filip Hric - Testing Circle
Automatic WaitingCypress has a build in feature called Automatic Waiting. Unlike the
cy.wait()command where you have to wait a static amount of time to continue with your test, the automatic waiting works like a polling mechanism. It retries the command and when the command resolves, it will continue.
Often, this automatic waiting is used with the cy.intercept mechanism. You can add an alias to your intercept and then wait for that alias to resolve, and then continue your test.
*note: in this example the interception did not mock the network-call.
Enabling the Cypress Cloud will give your testsuite even more superpowers. The Cypress Cloud will enable you to run your tests in CI in parallel mode, reducing the total run time of your test suite. Not to mention all the insights you could gain from all the other features of the Cypress Cloud.
Checkout a talk I did at the Dutch Cypress Community (Meetup recording) to find out more about how to set up Cypress parallelisation.
When you run the spec files from your testsuite in parallel and you use the Cypress Cloud, the Cloud Dashboard will inform you on how many CI - runners you need to configure for making your parallelised testruns even faster:
The Cypress Cloud has some nifty features which, together with the Parallelisation will give you your feedback even faster. The feature is called 'Run failed Tests First' (RFTF).
- A note on the Parallelisation feature of the Cypress Cloud though. If you want to use it on the free tier, your project has to be served on a public repository of your CI provider.
There has been some online discussions about the prices of the different Cypress Cloud Tiers. An interesting Twitter thread can be found here: [Thread]
Loading of pluginsEvery time Cypress starts to run a spec file, first all the plugins described in your plugins file wil be loaded and secondly the spec file itself will be loaded.
When you have plugins installed to be used for specific spec-files there is no need to put them in your 'global' plugins file so they will only be loaded when needed. This can save you quite some time! Talking about "keep your testsuite in shape" !
Cypress Grep is a plugin which adds some extra features to the Cypress Testrunner. No more selecting spec files by using globs, or even selecting specific folders for a smoke test. With this plugin you can label your tests and use that label to define which tests to run in a particular CI/CD job (or even locally).
Lets say you want to label some of your tests as
smoke (for smoke tests), you would be able to label entire spec files by labelling a 'describe' like this:
or particular tests by labelling an 'it'
Now, when you start your test in CI with
Only the tests will run that have been tagged with, in this case, 'smoke'.
There is a set of points of attention that can help you challenge your fast feedback loop to keep your testsuite in shape.
Use mocking and stubbing together with Cypress Aliases to make your tests run really fast. Use Cypress Grep to run only the tests in CI you have selected. Make use of the Cypress Cloud's parallelisation to lower your CI runtime using the latest Cypress version for the testrunner. Use the plugins only where it is really needed to make your testsuite slim. And last but not least: analyse, analyse, analyse to understand.
Are you interested in a thorough training on using Cypress in your projects, becoming a developer or tester who is able to write great tests, setup an E2E testsuite with Cypress, check out this training: <Link> If you are interested in reading another article I wrote about using Gitlab's parent-child pipelines to trigger Cypress E2E tests, you can find my post here: link