Skip to content

Container runtime tests

Container repositories can provide tests/smoke.sh to test their final images. The shared container workflow runs the script on the Linux CI runner, outside the image build, so test scripts, fixtures, databases, and startup side effects stay out of image layers.

Build, test, then publish

For every active track, both PR and production jobs follow this order:

Production jobs first reserve a draft release and its image revision, before the steps below. This supplies the build's exact PACKAGE_VERSION. PR jobs prepare a preview package version without creating releases or consuming revisions.

  1. Build every configured platform without layer cache and export one local OCI archive. BuildKit records provenance and SBOM attestations in that archive. This step does not push an image to a registry.
  2. Load each requested platform from the archive into the local Docker daemon using Skopeo. Verify the archive contains exactly one matching platform image and that the loaded image has the expected OS and architecture.
  3. Check the default runtime user, then run tests/smoke.sh once for each platform, when the hook exists. QEMU enables execution of non-native platforms on the GitHub runner. A failing hook fails the track; there is no automatic retry of the entire test.
  4. Validate release metadata, including OS identification from the first tested platform image, before any upload. PRs also run one release-lifecycle rehearsal against that local image and a fake GitHub API, exercising reservation, draft finalization, and metadata generation without publishing anything.
  5. Log in to GHCR and copy the original archive by digest with skopeo copy --all --preserve-digests. This preserves the multiarchitecture index, platform manifests, annotations, provenance, and SBOMs. There is no rebuild between testing and publication.
  6. Publish PR preview tags or finalize the production draft's notes with the tested artifact metadata, then publish image tags and the GitHub release.

An unsuccessful build or test never uploads image manifests or layers to GHCR. BuildKit's GitHub Actions cache may still be written; it is not a published container image. A failed production build or test retains its revision-reserving draft. A later publication failure can also leave an untagged registry artifact, as described in Container releases. This order applies to each track independently, so one failing track does not prevent another from completing.

Fork PRs use the same local build and test path but never log in or upload. Do not use pull_request_target to run these tests or publish PR code.

The shared helper records a successful test receipt next to the archive and checks its checksum before uploading. Missing receipts or modified archives fail publication. These checks guard against accidental misuse; they cannot protect against malicious code already executing on the runner.

Shared runtime-user validation

Before running a repository hook, container-artifact test checks every loaded platform image with its default user and no network, overriding only the entrypoint to /bin/sh. Images must provide /bin/sh and id. The default requirement is user and group tuxcrate, both with numeric ID 1000. The shared check verifies all four values: UID, GID, username, and group name. It also runs when the repository has no smoke hook.

Applications that require root must document the reason and declare it in the Containerfile. For example:

# ZFS administration requires root access to the host's ZFS devices.
LABEL com.tuxcrate.runtime-user="root"

This policy requires UID/GID 0:0 and user/group names root; it does not skip validation. The optional label value tuxcrate declares the default explicitly. Other nonempty values fail validation. The label does not change the configured user or grant host privileges.

The shared validator owns runtime identity assertions. Keep application startup, directory permissions, required tools, version/API checks, and shutdown behavior in repository hooks. Do not repeat the four identity assertions once shared validation is deployed, or start a service during an image build to test it.

Shared test containers and repository hooks use com.tuxcrate.smoke-test. Cleanup recognizes only this label; there is no legacy-prefix fallback.

Hook interface

Create a regular file at the repository root's tests/smoke.sh, regardless of a track's build context. It does not need an executable bit: the runner calls it with Bash from the repository root:

bash tests/smoke.sh IMAGE APP_VERSION PLATFORM
Argument Meaning
IMAGE Unique local Docker image reference loaded from the built archive; use it without pulling or rebuilding
APP_VERSION Expected upstream application version; for APT tracks, CI omits the Debian package revision
PLATFORM Requested platform, such as linux/amd64 or linux/arm64

The hook also inherits PACKAGE_VERSION from the workflow. When an application exposes packaging metadata, verify it matches this value, independently of APP_VERSION. This catches missing metadata, an incorrect revision, or a package author link that the application does not actually read. Sonarr, Radarr, and Prowlarr check their authenticated status APIs using the disposable instance's generated API key.

The Containerfile still receives the full APT installation pin through its APP_VERSION build argument and must install and assert that exact package. Do not assume the similarly named hook argument includes the Debian revision.

The status request retries up to four times with a ten-second socket timeout per request, allowing for slow first requests under QEMU. Once a request succeeds, missing or incorrect package metadata fails immediately; it is not retried. Failure diagnostics print the request exit status or the expected and actual package field, without dumping unrelated API response data or the API key.

The same script handles all tracks. Use the expected version argument instead of hard-coding a release. Helpers and fixtures may live elsewhere under tests/; the pipeline does not automatically execute every file in that directory.

The runner supplies TUXCRATE_TEST_ID. Label every test container with com.tuxcrate.smoke-test=$TUXCRATE_TEST_ID so the shared helper can remove it if the hook times out or is interrupted. Each invocation has a five-minute timeout, followed by a 30-second termination grace period. Hooks should use shorter readiness and shutdown limits and their own cleanup traps.

Repositories without a hook keep their installation assertions and receive platform, runtime-user, and OS-metadata validation, but no application-specific runtime smoke test. Repositories can therefore migrate individually; CI prints a notice when the hook is missing.

Writing a test

Run the production entrypoint as the image's configured user. Do not override it to make startup pass. Check readiness through an application endpoint where available, check the installed version, and confirm graceful shutdown. Check application directory ownership, writable configuration directories, licenses, and required tools as appropriate for the application.

Use disposable container writable layers or volumes for generated data. Do not mount the source checkout over application directories, use production data, install test packages into the image, or commit a tested container back into an image. Prefer no host port mappings; docker exec can query a loopback endpoint inside the container. The Sonarr, Radarr, and Prowlarr hooks run with --network none, since their smoke tests require no external services.

A minimal host-side hook looks like this:

#!/usr/bin/env bash
set -euo pipefail

image=${1:?Image required}
version=${2:?Version required}
platform=${3:?Platform required}
container=
cleanup() {
    status=$?
    if [[ -n $container ]]; then
        if ((status != 0)); then
            docker logs "$container" >&2 || true
        fi
        docker rm --force --volumes "$container" >/dev/null || true
    fi
    exit "$status"
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

container=$(docker create --pull=never --platform "$platform" --network none \
    --label "com.tuxcrate.smoke-test=${TUXCRATE_TEST_ID:-manual-$$}" "$image")
docker start "$container" >/dev/null

# Add bounded, application-specific readiness and exact-version checks here.
# Use "$version"; do not treat successful docker start as application readiness.

docker stop --time 30 "$container" >/dev/null
test "$(docker wait "$container")" = 0

Add readiness and version checks to complete this smoke test. The Sonarr, Radarr, and Prowlarr repositories contain working examples. On failure, print the failed check, container state, logs, and relevant file listings before removing the container. Retry transient readiness conditions; version mismatches and failed assertions after startup should fail immediately.

Hooks run host-side with Docker access and have the same contributor trust requirements as Containerfiles. The helper unsets GH_TOKEN and GITHUB_TOKEN for the hook, and the workflow does not log in to GHCR until testing finishes. Never forward CI credentials or mount the Docker socket into an application container. Same-repository build steps can still receive the existing optional GitHub API BuildKit secret; fork builds cannot.

Local validation

For a quick single-platform test, build and load the final image, then invoke the hook from its repository. Substitute the track's real pins from meta.yaml:

docker buildx build --pull --no-cache --platform linux/amd64 --load \
  --build-arg APP_VERSION="$APP_VERSION" \
  --build-arg BASE_IMAGE="$BASE_IMAGE" \
  --build-arg BUILDKIT_DOCKERFILE_CHECK=skip=InvalidDefaultArgInFrom \
  --file Containerfile --tag tuxcrate-local:test .
bash tests/smoke.sh tuxcrate-local:test "$APP_VERSION" linux/amd64

For images that embed packaging metadata, optionally pass --build-arg PACKAGE_VERSION="$PACKAGE_VERSION" and set the same variable when running the hook. Use a clearly local identifier, such as 4.0.20.3014-local, unless testing a specific revision value. Omitting the argument and environment variable uses the local fallback in the Sonarr, Radarr, and Prowlarr images and hooks. These commands do not reserve or publish a release.

To reproduce the shared artifact path on Linux, install Go, Bash, Docker/Buildx, Skopeo, and QEMU/binfmt support for non-native execution. Build the shared Go command once (from the github-actions checkout):

go build -o /path/to/container-artifact ./cmd/container-artifact

Then run the following from the image repository. Use a Buildx builder that supports the OCI exporter, such as the docker-container driver:

docker buildx build --pull --no-cache --platform linux/amd64,linux/arm64 \
  --provenance=mode=max --sbom=true \
  --build-arg APP_VERSION="$APP_VERSION" \
  --build-arg BASE_IMAGE="$BASE_IMAGE" \
  --build-arg BUILDKIT_DOCKERFILE_CHECK=skip=InvalidDefaultArgInFrom \
  --file Containerfile --output type=oci,dest=/path/to/image.tar .
/path/to/container-artifact test \
  /path/to/image.tar linux/amd64,linux/arm64 "$APP_VERSION"

These commands do not publish anything. The helper retains the first loaded image for OS-metadata validation and removes the other temporary image tags. The first image tag is available as image in GITHUB_OUTPUT when that variable is set; remove it with docker image rm when local validation is finished. Remove the archive and its .tested receipt when they are no longer needed.

Migrating an image repository

For the runtime-user policy, first migrate account names, explicit root declarations, and smoke-test labels in callers. Deploy the shared validator before removing duplicate identity checks from repository hooks. Keep these changes separate when necessary so existing validation remains available during the rollout.

Publish the shared workflow and helper changes before migrating callers. For pinned workflows, keep uses and with.shared-ref on the same published ref. Do not disable existing startup tests until the shared hook is available.

Convert the old build-time test to a host-side Docker test. Remove the Containerfile's smoke-test COPY and RUN instructions, and remove any test-only dependencies or fixtures from the final image. Keep useful installation assertions. No new workflow inputs or meta.yaml fields are needed. Require successful PR validation on both configured architectures before merging the migration.