Back up and restore

Create one coordinated recovery set and prove it with an isolated restore.

Back up Postgres and object storage from one maintenance point. Redis is a separate policy choice because it contains queues, delayed work, and cache state rather than CMS authority.

Access

You need host, Postgres, object-storage, Redis, backup-destination, and deployment-secret administration. Keep recovery files encrypted and outside the deployment host.

Define the recovery set

Postgres: authoritative CMS, identity, Candidate, Operation, audit, and projection state.
Object storage: originals and generated image results.
Redis: BullMQ and delayed-work state when your recovery policy includes it.
Deployment inventory: trusted release identifier, resolved image IDs/digests, non-secret configuration, and separately protected secrets.

If Redis is omitted, first prove there is no queued, active, or delayed work. Environment operations have a Postgres outbox for enqueue recovery, but Content schedules, Release schedules, and other queues do not all have equivalent replay guarantees.

Create a maintenance boundary

Set the service keys from your production Compose definition, then stop request and background writers. Leave Postgres, Redis, and object storage running for the backup tools.

ADMIN_SERVICE=<admin-service-key>
CORE_SERVICE=<core-api-service-key>
DELIVERY_SERVICE=<delivery-api-service-key>
IMAGE_SERVICE=<image-service-key>

COMPOSE='docker compose -f docker-compose.prod.yaml'

$COMPOSE stop \
  "$ADMIN_SERVICE" "$CORE_SERVICE" "$DELIVERY_SERVICE" "$IMAGE_SERVICE" worker
$COMPOSE ps -a

Keep ingress in maintenance until the recovery point is complete and the restarted services pass smoke tests.

Back up Postgres and objects

The following logical procedure is drilled against the Postgres and object-client images in the production Compose definition. Choose a protected destination and retain its checksums.

mkdir -p recovery/objects
chmod 700 recovery
export OBJECT_INTERNAL_URL=<object-store-internal-url>

$COMPOSE exec -T postgres \
  pg_dump -U postgres -Fc --no-owner --no-acl jetrepo \
  > recovery/postgres.dump

$COMPOSE run --rm --no-deps \
  -e OBJECT_INTERNAL_URL \
  -v "$PWD/recovery/objects:/backup" \
  --entrypoint sh minio-setup -c '
    mc alias set source "$OBJECT_INTERNAL_URL" \
      "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null
    mc mirror --overwrite "source/$BUCKET_NAME" /backup
  '

$COMPOSE images > recovery/images.txt
(cd recovery && find . -type f -not -name SHA256SUMS -print0 \
  | sort -z \
  | xargs -0 shasum -a 256 > SHA256SUMS)
(cd recovery && shasum -a 256 -c SHA256SUMS)

pg_dump is a consistent logical Postgres snapshot. The maintenance boundary prevents the object set from changing while Postgres and objects are captured. Protect the Compose environment and secrets separately; do not put them in a plaintext recovery directory.

Redis choice

Use a Redis-compatible backup procedure only when you intend to retain queued or delayed work. The project drill omitted Redis after proving there were no non-terminal Operations and no BullMQ work keys; a fresh Redis instance then rebuilt queue metadata. Do not generalize that omission to a deployment with pending work.

Restore into an isolated project

Use a different Compose project and network name so the drill cannot attach to production containers, networks, or named volumes.

export JETREPO_NETWORK_NAME=jetrepo_restore_network
RESTORE='docker compose -p jetrepo-restore -f docker-compose.prod.yaml'

$RESTORE config --quiet
$RESTORE build
$RESTORE up -d postgres redis minio minio-setup
$RESTORE ps -a

(cd recovery && shasum -a 256 -c SHA256SUMS)

$RESTORE exec -T postgres dropdb -U postgres --if-exists jetrepo
$RESTORE exec -T postgres createdb -U postgres jetrepo
$RESTORE exec -T postgres \
  pg_restore -U postgres -d jetrepo --no-owner --no-acl \
  < recovery/postgres.dump

export OBJECT_INTERNAL_URL=<restore-object-store-internal-url>
$RESTORE run --rm --no-deps \
  -e OBJECT_INTERNAL_URL \
  -v "$PWD/recovery/objects:/backup:ro" \
  --entrypoint sh minio-setup -c '
    mc alias set target "$OBJECT_INTERNAL_URL" \
      "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null
    mc mirror --overwrite /backup "target/$BUCKET_NAME"
  '

Apply the target release migrations, then reconstruct immutable Resource History for any Backend created before that authority existed:

$RESTORE run --rm migrate
$RESTORE run --rm --no-deps "$CORE_SERVICE" \
  jetrepo-resource-history-bootstrap
$RESTORE up -d
$RESTORE ps -a
$RESTORE logs --tail=200 migrate minio-setup

The history bootstrap is idempotent. Repeating it must report no newly created perspectives.

Verify the restore

Verify all of the following before accepting the recovery set:

Every declared health check is healthy and both one-shot jobs exited 0.
Sign-in and Organization membership resolve correctly. Test the deployment’s OAuth Management connection; if Agent Auth is used, also test a freshly signed request. A historical Agent Auth-only probe does not verify every configured client.
Environment and Alias resolution points to the expected active generations.
Data Types, entries, immutable manifests, Candidates, Operations, and audit evidence match the source.
Published Content returns the same manifest ID/hash and values through REST and Context MCP.
Original objects and one signed image transform return successfully.
Worker metadata completes for a non-sensitive upload.
Pending or scheduled work matches the recorded Redis policy.
A second Resource History bootstrap creates nothing.

The project drill restored a coordinated set into isolated volumes, reapplied all migrations, reproduced the exact Published manifest through REST and Context MCP, read the restored original object, and started every production service healthy. A final authentication probe deliberately exposed a mismatched BETTER_AUTH_SECRET; restoring the source secret then recovered the existing session, Organization membership, encrypted signing key, and a fresh Agent Auth Management MCP initialization without changing Postgres. Physical Postgres backup, live production cutover, encryption tooling, retention, and deployment-specific RPO/RTO remain operator responsibilities.

Restart the source stack

After the backup is complete, restart the source services and repeat health plus functional smoke tests:

$COMPOSE start \
  "$CORE_SERVICE" "$DELIVERY_SERVICE" worker "$IMAGE_SERVICE" "$ADMIN_SERVICE"