Percy Developer Hub

Welcome to the Percy Developer Hub. You'll find comprehensive guides and documentation to help you start working with Percy as quickly as possible, as well as support if you get stuck. Let's jump right in!

Parallel test suites

Running complex test suites with Percy

ūüĎć

Covered in this doc

Automatically-supported CI services
Advanced: using "finalize all" for full control
How Percy handles parallel build environments

Parallelism is a common way to speed up test suites by running multiple tests concurrently and then combining results at the end.

Percy has built-in support for complex test suites that run in parallelized CI services or parallel test runners. There are two different ways that you can setup Percy for parallel test suites: across many machines or running parallel tests on the same machine.

Percy is designed to support any parallelized test environment. Usually this happens automatically for most CI providers, but you can also configure Percy for more custom setups with just two environment variables.

Once set up, snapshots are pushed from your tests to Percy and grouped in the same Percy build, no matter if your tests are run in different processes or even on different machines.

ūüďė

Troubleshooting builds stuck in "Receiving..."

If your parallel setup is not configured correctly, you may see what look like duplicate Percy builds that get stuck in the "Receiving..." state. If so, this doc is for you.

Parallel tests on the same machine

If your tests are running in parallel on the same machine, Percy won't require any PERCY_PARALLEL_* environment variables being set. Setting up parallel tests for this use case is much easier!

In this case you should use the percy start CLI command (rather than percy exec) to start a local Percy server. This will create a Percy build and wait for snapshots to be sent to the Percy process (which happens while tests are running & percySnapshot is called). Percy only requires a single server to be running and accepting snapshots. If the server is accessibile to all parallel tests (on the same machine rather across many different machine), using percy start is the preferred way to go.

Once your tests have completed you then will need to call percy stop to finalize the build (or by terminating the Percy process). This should be run only after all the tests have completed & exited to ensure all snapshots have been sent.

A step by step break down of this could look like:

  • CI starts a build
  • Application build/dependencies install/etc (your CI process)
  • percy start is called (and creates a Percy build)
  • Tests run in parallel, on the same machine
  • Tests complete
  • percy stop is called to finalize the Percy build

Parallel tests across many machines

How it works


Supporting parallel tests in a visual CI service is complex---unlike traditional CI services, different builds in Percy influence each other by design. Snapshots rendered in one build might become the new base for visual diffs in a later build. Because of this, we must ensure that all snapshots from your test suite are grouped correctly in a single Percy build.

Percy's API provides a simple mechanism to associate snapshots to the same Percy build, no matter if the snapshots are pushed from different processes or even different machines. Two params are needed: a "parallel nonce" (any unique identifier for the build) and the total number of build nodes. Having a nonce that is shared across build nodes lets us link together snapshots created in different processes to the same Percy build. The client sends both of these with every API call to create a build.

On the server-side, we use a transaction to coordinate between all the parallel API calls and only create one build per nonce. If a pending build has already been created for the given nonce, the build is simply returned to the client.

To tell when a parallelized build is finished, we count the number of "finalize" API calls that have been received and only truly finalize the build when the count equals the total number of nodes we expect a response from. This lets us avoid many race conditions by design since we don't rely on when build nodes start or how long they run.

With this mechanism, the client-side logic remains very simple and all the complexities of transactionality and coordination between nodes happens in our service.

Setup


Depending on your CI service, you may need to do some simple configuration to support parallel tests.

  • Automatically supported¬†when using the Percy client libraries:
  • Manual configuration:

Manual configuration with environment variables

Two environment variables must be configured to support parallel test suites: a nonce, and the total number of parallel nodes.

  • PERCY_PARALLEL_NONCE: A unique identifier for this build. This can be anything, but it must be the same across parallel build nodes. Usually, this is just the CI build number or a shared timestamp.
  • PERCY_PARALLEL_TOTAL: The total number of parallel build nodes.

These are the only variables needed for Percy to correctly support your parallelized test suite.

Example: parallel_tests gem


For example, if you are using the parallel_tests gem with RSpec, you may run your tests like this:

$ rake parallel:spec[4] 4 processes for 100 specs, ~ 25 specs per process ...

We can modify this to export the two environment variables:

$ export PERCY_PARALLEL_NONCE=`date +%s`
$ export PERCY_PARALLEL_TOTAL=4
$ rake parallel:spec[4] 4 processes for 100 specs, ~ 25 specs per process ...

Or, all on one line:

$ PERCY_PARALLEL_NONCE=`date +%s` PERCY_PARALLEL_TOTAL=4 rake parallel:spec[4]

In this example, we use date +%s to generate a Unix timestamp to use as the parallel nonce. As long as different test runs never start at the exact same second, this will work nicely as an identifier for this build across all four parallel processes.

Using "finalize all"


Percy's SDKs try to automatically determine the correct paralleization settings per environment, but sometimes it's easier to take full control and handle it all yourself. This can help in complex CI parallel configurations where the number of parallel nodes changes regularly, or can't be automatically determined.

Percy Agent, our core SDK, provides a way to opt-in to manually telling Percy when all the parallel nodes have finished processing. We call this "finalize all", and it's available as this command:

$ percy finalize --all

This command requires that you have set the following environment variable in all the places that Percy snapshots might run:

PERCY_PARALLEL_TOTAL=-1

(Notice it is the value negative 1, "-1").

Setup finalize all

The general setup you'll be aiming for is the following:

Your test suite kicks off and it divides the work of running the tests up into multiple test runners. Those test runners will finish at different times. Once they have all finished you will run a "Finalize All" step, which tells Percy the build is completely finished at this point and no new snapshots for this build will arrive.

Required environment variables

  • PERCY_PARALLEL_NONCE: The nonce is an identifier that must be the same over each of the steps in the workflow shown above, but different if you were to kick off the whole workflow again. Note: if you are using any of our fully supported CI services, you do not need to set this as we have figured it out for you here.
  • PERCY_PARALLEL_TOTAL: this must be set to -1. This tells Percy we don't know how parallelized this run is, but we'll consider your Percy build fully wrapped up when we hear the "Finalize All" step.
  • PERCY_TOKEN: This is the token obtained from your project settings in Percy.

Custom Configuration

You can run the percy finalize --all command by using npx:

npx percy finalize --all

Updated 28 days ago

Parallel test suites


Running complex test suites with Percy

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.