Container runtime tests¶
Container repositories can provide tests/smoke.sh to test their final images.
The shared container workflow runs the script on the Linux CI runner, outside
the image build, so test scripts, fixtures, databases, and startup side effects
stay out of image layers.
Build, test, then publish¶
For every active track, both PR and production jobs follow this order:
Production jobs first reserve a draft release and its image revision, before the
steps below. This supplies the build's exact PACKAGE_VERSION. PR jobs prepare
a preview package version without creating releases or consuming revisions.
- Build every configured platform without layer cache and export one local OCI archive. BuildKit records provenance and SBOM attestations in that archive. This step does not push an image to a registry.
- Load each requested platform from the archive into the local Docker daemon using Skopeo. Verify the archive contains exactly one matching platform image and that the loaded image has the expected OS and architecture.
- Check the default runtime user, then run
tests/smoke.shonce for each platform, when the hook exists. QEMU enables execution of non-native platforms on the GitHub runner. A failing hook fails the track; there is no automatic retry of the entire test. - Validate release metadata, including OS identification from the first tested platform image, before any upload. PRs also run one release-lifecycle rehearsal against that local image and a fake GitHub API, exercising reservation, draft finalization, and metadata generation without publishing anything.
- Log in to GHCR and copy the original archive by digest with
skopeo copy --all --preserve-digests. This preserves the multiarchitecture index, platform manifests, annotations, provenance, and SBOMs. There is no rebuild between testing and publication. - Publish PR preview tags or finalize the production draft's notes with the tested artifact metadata, then publish image tags and the GitHub release.
An unsuccessful build or test never uploads image manifests or layers to GHCR. BuildKit's GitHub Actions cache may still be written; it is not a published container image. A failed production build or test retains its revision-reserving draft. A later publication failure can also leave an untagged registry artifact, as described in Container releases. This order applies to each track independently, so one failing track does not prevent another from completing.
Fork PRs use the same local build and test path but never log in or upload.
Do not use pull_request_target to run these tests or publish PR code.
The shared helper records a successful test receipt next to the archive and checks its checksum before uploading. Missing receipts or modified archives fail publication. These checks guard against accidental misuse; they cannot protect against malicious code already executing on the runner.
Shared runtime-user validation¶
Before running a repository hook, container-artifact test checks every loaded
platform image with its default user and no network, overriding only the
entrypoint to /bin/sh. Images must provide /bin/sh and id. The default
requirement is user and group tuxcrate, both with numeric ID 1000. The shared
check verifies all four values: UID, GID, username, and group name. It also runs
when the repository has no smoke hook.
Applications that require root must document the reason and declare it in the Containerfile. For example:
# ZFS administration requires root access to the host's ZFS devices.
LABEL com.tuxcrate.runtime-user="root"
This policy requires UID/GID 0:0 and user/group names root; it does not skip
validation. The optional label value tuxcrate declares the default explicitly.
Other nonempty values fail validation. The label does not change the configured
user or grant host privileges.
The shared validator owns runtime identity assertions. Keep application startup, directory permissions, required tools, version/API checks, and shutdown behavior in repository hooks. Do not repeat the four identity assertions once shared validation is deployed, or start a service during an image build to test it.
Shared test containers and repository hooks use com.tuxcrate.smoke-test.
Cleanup recognizes only this label; there is no legacy-prefix fallback.
Hook interface¶
Create a regular file at the repository root's tests/smoke.sh, regardless
of a track's build context. It does not need an executable bit: the runner calls
it with Bash from the repository root:
| Argument | Meaning |
|---|---|
IMAGE |
Unique local Docker image reference loaded from the built archive; use it without pulling or rebuilding |
APP_VERSION |
Expected upstream application version; for APT tracks, CI omits the Debian package revision |
PLATFORM |
Requested platform, such as linux/amd64 or linux/arm64 |
The hook also inherits PACKAGE_VERSION from the workflow. When an application
exposes packaging metadata, verify it matches this value, independently of
APP_VERSION. This catches missing metadata, an incorrect revision, or a package
author link that the application does not actually read. Sonarr, Radarr, and
Prowlarr check their authenticated status APIs using the disposable instance's
generated API key.
The Containerfile still receives the full APT installation pin through its
APP_VERSION build argument and must install and assert that exact package.
Do not assume the similarly named hook argument includes the Debian revision.
The status request retries up to four times with a ten-second socket timeout per request, allowing for slow first requests under QEMU. Once a request succeeds, missing or incorrect package metadata fails immediately; it is not retried. Failure diagnostics print the request exit status or the expected and actual package field, without dumping unrelated API response data or the API key.
The same script handles all tracks. Use the expected version argument instead
of hard-coding a release. Helpers and fixtures may live elsewhere under tests/;
the pipeline does not automatically execute every file in that directory.
The runner supplies TUXCRATE_TEST_ID. Label every test container with
com.tuxcrate.smoke-test=$TUXCRATE_TEST_ID so the shared helper can remove it if
the hook times out or is interrupted. Each invocation has a five-minute timeout,
followed by a 30-second termination grace period. Hooks should use shorter
readiness and shutdown limits and their own cleanup traps.
Repositories without a hook keep their installation assertions and receive platform, runtime-user, and OS-metadata validation, but no application-specific runtime smoke test. Repositories can therefore migrate individually; CI prints a notice when the hook is missing.
Writing a test¶
Run the production entrypoint as the image's configured user. Do not override it to make startup pass. Check readiness through an application endpoint where available, check the installed version, and confirm graceful shutdown. Check application directory ownership, writable configuration directories, licenses, and required tools as appropriate for the application.
Use disposable container writable layers or volumes for generated data. Do not
mount the source checkout over application directories, use production data,
install test packages into the image, or commit a tested container back into an
image. Prefer no host port mappings; docker exec can query a loopback endpoint
inside the container. The Sonarr, Radarr, and Prowlarr hooks run with
--network none, since their smoke tests require no external services.
A minimal host-side hook looks like this:
#!/usr/bin/env bash
set -euo pipefail
image=${1:?Image required}
version=${2:?Version required}
platform=${3:?Platform required}
container=
cleanup() {
status=$?
if [[ -n $container ]]; then
if ((status != 0)); then
docker logs "$container" >&2 || true
fi
docker rm --force --volumes "$container" >/dev/null || true
fi
exit "$status"
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
container=$(docker create --pull=never --platform "$platform" --network none \
--label "com.tuxcrate.smoke-test=${TUXCRATE_TEST_ID:-manual-$$}" "$image")
docker start "$container" >/dev/null
# Add bounded, application-specific readiness and exact-version checks here.
# Use "$version"; do not treat successful docker start as application readiness.
docker stop --time 30 "$container" >/dev/null
test "$(docker wait "$container")" = 0
Add readiness and version checks to complete this smoke test. The Sonarr, Radarr, and Prowlarr repositories contain working examples. On failure, print the failed check, container state, logs, and relevant file listings before removing the container. Retry transient readiness conditions; version mismatches and failed assertions after startup should fail immediately.
Hooks run host-side with Docker access and have the same contributor trust
requirements as Containerfiles. The helper unsets GH_TOKEN and GITHUB_TOKEN
for the hook, and the workflow does not log in to GHCR until testing finishes.
Never forward CI credentials or mount the Docker socket into an application
container. Same-repository build steps can still receive the existing optional
GitHub API BuildKit secret; fork builds cannot.
Local validation¶
For a quick single-platform test, build and load the final image, then invoke
the hook from its repository. Substitute the track's real pins from meta.yaml:
docker buildx build --pull --no-cache --platform linux/amd64 --load \
--build-arg APP_VERSION="$APP_VERSION" \
--build-arg BASE_IMAGE="$BASE_IMAGE" \
--build-arg BUILDKIT_DOCKERFILE_CHECK=skip=InvalidDefaultArgInFrom \
--file Containerfile --tag tuxcrate-local:test .
bash tests/smoke.sh tuxcrate-local:test "$APP_VERSION" linux/amd64
For images that embed packaging metadata, optionally pass
--build-arg PACKAGE_VERSION="$PACKAGE_VERSION" and set the same variable when
running the hook. Use a clearly local identifier, such as 4.0.20.3014-local,
unless testing a specific revision value. Omitting the argument and environment
variable uses the local fallback in the Sonarr, Radarr, and Prowlarr images and
hooks. These commands do not reserve or publish a release.
To reproduce the shared artifact path on Linux, install Go, Bash, Docker/Buildx,
Skopeo, and QEMU/binfmt support for non-native execution. Build the shared Go
command once (from the github-actions checkout):
Then run the following from the image repository. Use a Buildx
builder that supports the OCI exporter, such as the docker-container driver:
docker buildx build --pull --no-cache --platform linux/amd64,linux/arm64 \
--provenance=mode=max --sbom=true \
--build-arg APP_VERSION="$APP_VERSION" \
--build-arg BASE_IMAGE="$BASE_IMAGE" \
--build-arg BUILDKIT_DOCKERFILE_CHECK=skip=InvalidDefaultArgInFrom \
--file Containerfile --output type=oci,dest=/path/to/image.tar .
/path/to/container-artifact test \
/path/to/image.tar linux/amd64,linux/arm64 "$APP_VERSION"
These commands do not publish anything. The helper retains the first loaded
image for OS-metadata validation and removes the other temporary image tags.
The first image tag is available as image in GITHUB_OUTPUT when that variable
is set; remove it with docker image rm when local validation is finished.
Remove the archive and its .tested receipt when they are no longer needed.
Migrating an image repository¶
For the runtime-user policy, first migrate account names, explicit root declarations, and smoke-test labels in callers. Deploy the shared validator before removing duplicate identity checks from repository hooks. Keep these changes separate when necessary so existing validation remains available during the rollout.
Publish the shared workflow and helper changes before migrating callers.
For pinned workflows, keep uses and with.shared-ref on the same published
ref. Do not disable existing startup tests until the shared hook is available.
Convert the old build-time test to a host-side Docker test. Remove the
Containerfile's smoke-test COPY and RUN instructions, and remove any
test-only dependencies or fixtures from the final image. Keep useful
installation assertions. No new workflow inputs or meta.yaml
fields are needed. Require successful PR validation on both configured
architectures before merging the migration.