Percy Developer Hub

Welcome to the Percy Developer Hub. You'll find comprehensive guides and documentation to help you start working with Percy as quickly as possible, as well as support if you get stuck. Let's jump right in!

Parallel test suites

Running complex test suites with Percy

Covered in this doc

Automatically-supported CI services
Advanced: using "finalize all" for full control
How Percy handles parallel build environments

Parallelism is a common way to speed up test suites by running multiple tests concurrently and then combining results at the end.

Percy has built-in support for complex test suites that run in parallelized CI services or parallel test runners. Percy is designed to support any parallelized test environment. Usually this happens automatically for most CI providers, but you can also configure Percy for more custom setups with just two environment variables.

Once set up, snapshots are pushed from your tests to Percy and grouped in the same Percy build, no matter if your tests are run in different processes or even on different machines.

Troubleshooting builds stuck in "Receiving..."

If your parallel setup is not configured correctly, you may see what look like duplicate Percy builds that get stuck in the "Receiving..." state. If so, this doc is for you.

How it works


Supporting parallel tests in a visual CI service is complex---unlike traditional CI services, different builds in Percy influence each other by design. Snapshots rendered in one build might become the new base for visual diffs in a later build. Because of this, we must ensure that all snapshots from your test suite are grouped correctly in a single Percy build.

Percy's API provides a simple mechanism to associate snapshots to the same Percy build, no matter if the snapshots are pushed from different processes or even different machines. Two params are needed: a "parallel nonce" (any unique identifier for the build) and the total number of build nodes. Having a nonce that is shared across build nodes lets us link together snapshots created in different processes to the same Percy build. The client sends both of these with every API call to create a build.

On the server-side, we use a transaction to coordinate between all the parallel API calls and only create one build per nonce. If a pending build has already been created for the given nonce, the build is simply returned to the client.

To tell when a parallelized build is finished, we count the number of "finalize" API calls that have been received and only truly finalize the build when the count equals the total number of nodes we expect a response from. This lets us avoid many race conditions by design since we don't rely on when build nodes start or how long they run.

With this mechanism, the client-side logic remains very simple and all the complexities of transactionality and coordination between nodes happens in our service.

Setup


Depending on your CI service, you may need to do some simple configuration to support parallel tests.

  • Automatically supported when using the Percy client libraries:
  • Manual configuration:

Manual configuration with environment variables

Two environment variables must be configured to support parallel test suites: a nonce, and the total number of parallel nodes.

  • PERCY_PARALLEL_NONCE: A unique identifier for this build. This can be anything, but it must be the same across parallel build nodes. Usually, this is just the CI build number or a shared timestamp.
  • PERCY_PARALLEL_TOTAL: The total number of parallel build nodes.

These are the only variables needed for Percy to correctly support your parallelized test suite.

Example: parallel_tests gem


For example, if you are using the parallel_tests gem with RSpec, you may run your tests like this:

$ rake parallel:spec[4] 4 processes for 100 specs, ~ 25 specs per process ...

We can modify this to export the two environment variables:

$ export PERCY_PARALLEL_NONCE=`date +%s`
$ export PERCY_PARALLEL_TOTAL=4
$ rake parallel:spec[4] 4 processes for 100 specs, ~ 25 specs per process ...

Or, all on one line:

$ PERCY_PARALLEL_NONCE=`date +%s` PERCY_PARALLEL_TOTAL=4 rake parallel:spec[4]

In this example, we use date +%s to generate a Unix timestamp to use as the parallel nonce. As long as different test runs never start at the exact same second, this will work nicely as an identifier for this build across all four parallel processes.

Using "finalize all"


Percy's SDKs try to automatically determine the correct paralleization settings per environment, but sometimes it's easier to take full control and handle it all yourself. This can help in complex CI parallel configurations where the number of parallel nodes changes regularly, or can't be automatically determined.

Percy Agent, our core SDK, provides a way to opt-in to manually telling Percy when all the parallel nodes have finished processing. We call this "finalize all", and it's available as this command:

$ percy finalize --all

This command requires that you have set the following environment variable in all the places that Percy snapshots might run:

PERCY_PARALLEL_TOTAL=-1

(Notice it is the value negative 1, "-1").

Setup finalize all

The general setup you'll be aiming for is the following:

Your test suite kicks off and it divides the work of running the tests up into multiple test runners. Those test runners will finish at different times. Once they have all finished you will run a "Finalize All" step, which tells Percy the build is completely finished at this point and no new snapshots for this build will arrive.

Required environment variables

  • PERCY_PARALLEL_NONCE: The nonce is an identifier that must be the same over each of the steps in the workflow shown above, but different if you were to kick off the whole workflow again. Note: if you are using any of our fully supported CI services, you do not need to set this as we have figured it out for you here.
  • PERCY_PARALLEL_TOTAL: this must be set to -1. This tells Percy we don't know how parallelized this run is, but we'll consider your Percy build fully wrapped up when we hear the "Finalize All" step.
  • PERCY_TOKEN: This is the token obtained from your project settings in Percy.

Docker Image

If Docker is available to you in CI, you can use our minimal pre-built Docker image which has Percy Agent and the percy command already setup so you can run percy finalize --all as an entry point.

Custom Configuration

If Docker isn't available to you, you can run the percy finalize --all command by using npx:

npx percy finalize --all

CircleCI

We recommend using our CircleCI Orb integration.

Updated about a year ago

Parallel test suites


Running complex test suites with Percy

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.