Replay checks outside GitHub

The replay-check Action is a wrapper around one command, pyrula ci run. Anything that can run that command with a token and a Python environment can run the check: GitLab CI, CircleCI, Jenkins, a Makefile. What a pass means, the outcomes and the exit codes are on Replay checks in CI; this page covers only what changes off GitHub.

Inputs

Action input	pyrula ci run
suite	--suite NAME
project	--project ID
environment	--environment NAME (default production)
token	The environment variable PYRULA_TOKEN. Never a flag
fixture-store-url	The environment variable PYRULA_FIXTURE_STORE_URL
cloud-url	--cloud-url URL or PYRULA_CLOUD_URL, when Cloud is not the hosted service
timeout	--timeout S
idempotency-suffix	--idempotency-key KEY (see The run key)
client-version	Your own pip install pin
outputs verdict, json-path, junit-path	--json PATH, --junit PATH, --summary PATH, and the exit code

Set up the suite, the CI token, the fixture store URL and the [tool.pyrula.ci] target map the same way as for GitHub (Set up). Install your project first, since the check imports your code, then pyrula[cloud], which brings the ci command.

GitLab CI

This job has not been run on GitLab. The flags in it and in the shell form below are checked against the command’s own parser, and the key and exit-code behaviour is read from its source.

Store the token and the fixture store URL as masked CI/CD variables named PYRULA_TOKEN and PYRULA_FIXTURE_STORE_URL, and the project id as PYRULA_PROJECT. The job fails on any exit code other than 0.

replay-check:
  image: python:3.13
  rules:
    # Same-project merge requests only: this job runs the change's code with your token.
    - if: $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_SOURCE_PROJECT_PATH == $CI_PROJECT_PATH
  script:
    - pip install -e . "pyrula[cloud]"
    - >
      pyrula ci run
      --suite checkout-regressions
      --project "$PYRULA_PROJECT"
      --idempotency-key "gitlab-$CI_PIPELINE_ID-$CI_JOB_ID"
      --json report.json --junit report.xml --summary report.md
  artifacts:
    when: always
    paths: [report.json, report.md]
    reports:
      junit: report.xml

artifacts:reports:junit shows each case in the merge request. The JSON report is pyrula.replay-ci/1, as on GitHub (JSON report).

Any shell

export PYRULA_TOKEN=...              # from your CI's secret store, never a flag
export PYRULA_FIXTURE_STORE_URL=...
pip install -e . "pyrula[cloud]"
pyrula ci run --suite checkout-regressions --project proj_123 \
  --idempotency-key "$BUILD_ID" \
  --json report.json --junit report.xml --summary report.md
echo "exit code $?"

Before the first pipeline, you can see the command’s first refusal locally. With no [tool.pyrula.ci] targets configured it exits 2 before any network call:

status=0
pyrula ci run --suite checkout-regressions --project proj_123 --idempotency-key demo || status=$?
test "$status" -eq 2

The exit code is the verdict: 0 every case compatible, 1 a failure, 2 usage or configuration, 3 a case could not be checked, 4 infrastructure, 5 timeout. Gate on non-zero, and use 3 to 5 to tell a broken pipeline from a broken change.

The run key

Cloud holds one suite run per key. On GitHub the command builds it from GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT and GITHUB_JOB. Off GitHub it reads none of them: without --idempotency-key every invocation is a new suite run, and --idempotency-suffix has nothing to append to.

• Pass a key that is stable for one attempt and different for matrix legs: pipeline id plus job id above. A job retried in GitLab has a new job id and starts a new suite run.
• Reuse a key and the command finds the first run: a key already reported exits 3 with the run’s link. Use this to resume a job that was cut off before it reported.
• A key is at most 128 characters. A suite run counts against the plan’s evaluation runs per month.

What GitHub adds that these do not

The command records the repository, commit, ref, pull request number and run URL from GitHub’s GITHUB_* variables, and nothing from GitLab’s or any other runner’s, so a run started elsewhere carries none of them. The Action also masks the secrets in the log, writes the job summary and annotates failing cases; in another runner, print report.md or publish the JUnit file instead.