Snapshots and the compare safety net

deploy-tools has no transactional rollback. Instead it relies on a snapshot of the intended configuration plus a compare command that detects when reality has drifted from it. This page explains how the two fit together.

What the snapshot is

The snapshot is a single file, deployment.yaml, at the root of the deployment area. It is a serialised Deployment — the complete set of Releases (live and deprecated) plus the settings, including each Module’s default version. It records what the deployment area is supposed to contain.

The sync command writes it; sync, validate and compare read it back to work out what has changed since the last sync.

Why it is written before the deploy

During a sync, the snapshot is written before the files are moved into place. This means the snapshot always describes the target state. If the deploy step then fails part way through, the snapshot and the actual files no longer agree, and the next compare reports that discrepancy. Were the snapshot written last, a deploy that died before making any changes could result in compare identifying a broken area as healthy.

How compare works

It reconstructs the de-facto Deployment by walking the deployment area — reading every module.yaml, checking which versions have a live versus a deprecated modulefile link, and reading each .version file — then diffs that against the snapshot. Any mismatch (a missing modulefile, a built module with no link, a wrong default version, a corrupt metadata file) is raised as a clear error showing the differing lines.

compare checks the area’s structure — the modulefiles, links and metadata whose breakage would disable many Modules at once. It does not inspect the contents of large built files such as Apptainer .sif images, so a corrupted .sif in the correct location will not be detected. The risk of corruption is avoided at build time instead: a Module is built on the same filesystem as the deployment area and published by a single atomic rename, so a partial or failed build (including .sif files) is never moved into place.

This is why CI should run compare before every sync: it confirms the area is in the healthy state the last sync claimed to leave it in.

Recovery is manual

With no automatic rollback, recovering from a failed sync is an administrator task: use compare to see what differs, repair the area by hand (or restore the previous snapshot), and re-run compare before another sync. Automatic self-healing was rejected to keep the tool simple for a small development team.

Two facilities help here:

• compare --use-ref <ref> compares the area against the snapshot stored at a previous git commit of the deployment area (e.g. HEAD~1); each sync commits its snapshot. This is helpful when attempting to fix a broken deployment area, as it is often easier to rollback the configuration to a previous state rather than fix the latest deployment.

• compare --from-scratch asserts only that the deployment root exists and is empty — this is the check to run in CI before the very first sync, when no snapshot exists yet.

The git repository in the deployment area exists only to give compare --use-ref this reference point. It deliberately excludes the build area and Apptainer images, and is not intended for reverting the area’s state.

Otherwise, it is expected that the deployment area is properly backed up, preferably with some kind of filesystem-level snapshot system.