Parallel test suites

Running complex test suites with Percy

👍

Covered in this doc

Parallel suites across many machines or containers
Parallel suites on the same machine
Troubleshooting

Introduction

Parallelism is a common way to speed up test suites by running multiple tests concurrently and then combining results at the end.

Percy has built-in support for complex test suites that run in parallelized CI services or parallel test runners. There are two different ways that you can setup Percy for parallel test suites: across many machines or running parallel tests on the same machine.

Percy is designed to support any parallelized test environment. Once set up, snapshots are pushed from your tests to Percy and grouped in the same Percy build, no matter if your tests are run in different processes or even on different machines.

Parallel suites split across different machines

If your tests are split across multiple machines or containers, there are two key steps to make it work with Percy:

  • Pass a --parallel flag to the Percy CLI (percy exec --parallel -- [test command]).
  • After all tests have completed, call percy build:finalize to finalize the entire Percy build.

Example config

This is not a working example, but it illustrates the steps to take.

version: 2.1

default_test_steps: &default_test_steps
  docker:
    - image: circleci/node:14-browsers
  steps:
    - checkout
    - run: npm ci

jobs:
  test_1:
    <<: *default_test_steps
    steps:
        - npx percy exec --parallel -- npm run test
  test_2:
    <<: *default_test_steps
    steps:
        - npx percy exec --parallel -- npm run test
      
  finalize_percy:
    <<: *default_test_steps
    steps:
        - npx percy build:finalize

workflows:
  version: 2.1
  test_and_release:
    jobs:
      - test_1
      - test_2
      - finalize_percy:
          requires:
            - test_1
            - test_2

Advanced details

Percy’s parallel support works by looking for two different environment variables to be set. If your CI provider is supported by Percy, PERCY_PARALLEL_NONCE will automatically be set for you.

  • PERCY_PARALLEL_TOTAL - The number of parallel nodes being ran
  • PERCY_PARALLEL_NONCE - A unique ID for this parallel build. Percy uses this to group many builds into a single build (with matching nonce’s) . It can only be used once.

Depending on the value of PERCY_PARALLEL_TOTAL, there are two different ways the Percy API behaves.

If PERCY_PARALLEL_TOTAL is set to a specific number, the API will wait for that many finalized builds to permanently finalize the Percy build. That means builds can no longer be created or added with that nonce once the number is reached. With this approach, you will not call percy build:finalize

If PERCY_PARALLEL_TOTAL is set to to -1 this tells the API to disregard the number of builds created and finalized. The API will then wait for a finalize all builds call, which will close the build for that nonce. The Percy CLI sets the total to -1 when the --parallel flag is passed.


Parallel tests on the same machine

If your tests are running in parallel on the same machine or container, the path to setting up Percy for parallel builds can be easier (as there's no environment variables to setup).

Percy's SDKs works by starting a locally running server (by default on http://localhost:5883). The SDKs then POST their captured snapshots to this server, which will then process the snapshots and send them to the Percy API. As long as this server is accessible while your tests run, Percy will capture snapshots. There are two different ways this approach can work, which depends on how your tests are setup.

If your test command manages spawning the parallel process, you can keep using percy exec as you were before. For example, a project using Ember Exam won't need to setup any extra configuration since Ember exam opens many browsers on the same machine to split the tests up.

If you are managing the processes on your own, you would use the Percy start CLI command (percy exec:start rather than percy exec) to start a local Percy server. This will create a Percy build and wait for snapshots to be sent to the Percy server (which happens when percySnapshot is called). Percy only requires a single server to be open and accepting snapshots. Once all of your tests have completed, you will need to call percy exec:stop to finalize the build (which closes the running Percy server). This should be run only after all the tests have completed & exited to ensure all snapshots have been sent.

Example config

This is not a working example, but it illustrates the steps to take.

version: 2.1

default_test_steps: &default_test_steps
  docker:
    - image: circleci/node:14-browsers
  steps:
    - checkout
    - run: npm ci

jobs:
  test:
    <<: *default_test_steps
    steps:
      - npx percy exec:start &
      # for example: ember exam --split=3 --parallel
      # opens 3 browsers on the same machine
      - npm run test:parallel
      - npx percy exec:stop

workflows:
  version: 2.1
  test_and_release:
    jobs:
      - test

Troubleshooting

Setting PERCY_PARALLEL_NONCE

If your CI provider is not supported by Percy out of the box, there's one environment variable you will need to set for parallel builds to work: PERCY_PARALLEL_NONCE. This environment variable should be set to an ID that is only used once, ever. Typically using a workflow ID or CI build number works well.

If your CI provider reuses these values on rebuilds, it will cause issues with Percy (and you should find a more unique nonce to use, instead).

Stuck in receiving

If a build is hanging in the "receiving" state, even after tests have finished, that means the build was never fully finalized.

If PERCY_PARALLEL_TOTAL is set to a specific number, make sure there were that many Percy builds finalized in the test run. For example, if PERCY_PARALLEL_TOTAL=4 and there were only 3 builds ran/finalized, Percy will hang in a receiving state. This is because Percy is waiting for 4 finalized builds.

If PERCY_PARALLEL_TOTAL is set to-1 (using --parallel), make sure percy build:finalize was called. If it was, make sure PERCY_PARALLEL_NONCE is the same for the tests and the finalize step.


Still need help? Open a GitHub discussion on the Percy CLI repo.