Covered in this doc
Automatically-supported CI services
Advanced: using "finalize all" for full control
How Percy handles parallel build environments
Parallelism is a common way to speed up test suites by running multiple tests concurrently and then combining results at the end.
Percy has built-in support for complex test suites that run in parallelized CI services or parallel test runners. There are two different ways that you can setup Percy for parallel test suites: across many machines or running parallel tests on the same machine.
Percy is designed to support any parallelized test environment. Usually this happens automatically for most CI providers, but you can also configure Percy for more custom setups with just two environment variables.
Once set up, snapshots are pushed from your tests to Percy and grouped in the same Percy build, no matter if your tests are run in different processes or even on different machines.
Troubleshooting builds stuck in "Receiving..."
If your parallel setup is not configured correctly, you may see what look like duplicate Percy builds that get stuck in the "Receiving..." state. If so, this doc is for you.
If your tests are running in parallel on the same machine, Percy won't require any
PERCY_PARALLEL_* environment variables being set. Setting up parallel tests for this use case is much easier!
In this case you should use the
percy start CLI command (rather than
percy exec) to start a local Percy server. This will create a Percy build and wait for snapshots to be sent to the Percy process (which happens while tests are running &
percySnapshot is called). Percy only requires a single server to be running and accepting snapshots. If the server is accessibile to all parallel tests (on the same machine rather across many different machine), using
percy start is the preferred way to go.
Once your tests have completed you then will need to call
percy stop to finalize the build (or by terminating the Percy process). This should be run only after all the tests have completed & exited to ensure all snapshots have been sent.
A step by step break down of this could look like:
- CI starts a build
- Application build/dependencies install/etc (your CI process)
percy startis called (and creates a Percy build)
- Tests run in parallel, on the same machine
- Tests complete
percy stopis called to finalize the Percy build
Supporting parallel tests in a visual CI service is complex---unlike traditional CI services, different builds in Percy influence each other by design. Snapshots rendered in one build might become the new base for visual diffs in a later build. Because of this, we must ensure that all snapshots from your test suite are grouped correctly in a single Percy build.
Percy's API provides a simple mechanism to associate snapshots to the same Percy build, no matter if the snapshots are pushed from different processes or even different machines. Two params are needed: a "parallel nonce" (any unique identifier for the build) and the total number of build nodes. Having a nonce that is shared across build nodes lets us link together snapshots created in different processes to the same Percy build. The client sends both of these with every API call to create a build.
On the server-side, we use a transaction to coordinate between all the parallel API calls and only create one build per nonce. If a pending build has already been created for the given nonce, the build is simply returned to the client.
To tell when a parallelized build is finished, we count the number of "finalize" API calls that have been received and only truly finalize the build when the count equals the total number of nodes we expect a response from. This lets us avoid many race conditions by design since we don't rely on when build nodes start or how long they run.
With this mechanism, the client-side logic remains very simple and all the complexities of transactionality and coordination between nodes happens in our service.
Depending on your CI service, you may need to do some simple configuration to support parallel tests.
- Automatically supported when using the Percy client libraries:
- Travis CI
- CodeShip parallel threads
- Buildkite parallelism
- Manual configuration:
- parallel_tests gem
- Jenkins, Drone, others
Two environment variables must be configured to support parallel test suites: a nonce, and the total number of parallel nodes.
PERCY_PARALLEL_NONCE: A unique identifier for this build. This can be anything, but it must be the same across parallel build nodes. Usually, this is just the CI build number or a shared timestamp.
PERCY_PARALLEL_TOTAL: The total number of parallel build nodes.
These are the only variables needed for Percy to correctly support your parallelized test suite.
For example, if you are using the parallel_tests gem with RSpec, you may run your tests like this:
$ rake parallel:spec 4 processes for 100 specs, ~ 25 specs per process ...
We can modify this to export the two environment variables:
$ export PERCY_PARALLEL_NONCE=`date +%s` $ export PERCY_PARALLEL_TOTAL=4 $ rake parallel:spec 4 processes for 100 specs, ~ 25 specs per process ...
Or, all on one line:
$ PERCY_PARALLEL_NONCE=`date +%s` PERCY_PARALLEL_TOTAL=4 rake parallel:spec
In this example, we use
date +%s to generate a Unix timestamp to use as the parallel nonce. As long as different test runs never start at the exact same second, this will work nicely as an identifier for this build across all four parallel processes.
Percy's SDKs try to automatically determine the correct paralleization settings per environment, but sometimes it's easier to take full control and handle it all yourself. This can help in complex CI parallel configurations where the number of parallel nodes changes regularly, or can't be automatically determined.
Percy Agent, our core SDK, provides a way to opt-in to manually telling Percy when all the parallel nodes have finished processing. We call this "finalize all", and it's available as this command:
$ percy finalize --all
This command requires that you have set the following environment variable in all the places that Percy snapshots might run:
(Notice it is the value negative 1, "-1").
The general setup you'll be aiming for is the following:
Your test suite kicks off and it divides the work of running the tests up into multiple test runners. Those test runners will finish at different times. Once they have all finished you will run a "Finalize All" step, which tells Percy the build is completely finished at this point and no new snapshots for this build will arrive.
PERCY_PARALLEL_NONCE: The nonce is an identifier that must be the same over each of the steps in the workflow shown above, but different if you were to kick off the whole workflow again. Note: if you are using any of our fully supported CI services, you do not need to set this as we have figured it out for you here.
PERCY_PARALLEL_TOTAL: this must be set to
-1. This tells Percy we don't know how parallelized this run is, but we'll consider your Percy build fully wrapped up when we hear the "Finalize All" step.
PERCY_TOKEN: This is the token obtained from your project settings in Percy.
You can run the
percy finalize --all command by using
npx percy finalize --all
Updated 4 months ago