Message Ops

Upgrade with approved 1.7.1 image digests and the matching deployment kit. Run the dedicated migration job before activating new API/Worker images, even when the SQL ledger is already current. Do not repeat administrator bootstrap.

Prepare

1. Review the 1.7.1 release notes, your current version’s compatibility requirements and deployment configuration. The 1.7.0 → 1.7.1 patch contains no application or database migration changes.
2. Complete Software Intake and record old/new Web/API/Worker digests. Keep the previous images in ACR.
3. Verify a recent recoverable PostgreSQL backup and matching encryption keys. Perform a restore drill if your evidence is stale. Record acceptable data loss and recovery duration; these are site-specific objectives, not product guarantees.
4. Arrange a maintenance window. Stop new submissions and let in-flight operations settle; reconcile uncertain broker outcomes. Coordinate API replicas, since older releases may have incompatible cookies, session grants or idempotency namespaces.

When upgrading from earlier releases, plan for users to sign in again and restart unfinished SSO flows. Confirm exact allowed provider IDs, immutable SSO identity bindings and acceptable SAML signatures. Review sourced/manual role assignments and telemetry service-name filters. Do not replay pre-upgrade idempotency keys without checking the existing job: older namespace entries are not automatically reused by the current implementation.

Run the new migration image

Preserve the full existing migration execution template, changing only the migrate container’s image. Set RESOURCE_GROUP, MIGRATION_JOB and NEW_API_IMAGE to the intended resources and approved digest:

az containerapp job show -g "$RESOURCE_GROUP" -n "$MIGRATION_JOB" \
  --query properties.template -o json > migration-current.json
jq --arg image "$NEW_API_IMAGE" \
  '.containers |= map(if .name == "migrate" then .image = $image else . end)' \
  migration-current.json > migration-execution.json
EXECUTION=$(az containerapp job start -g "$RESOURCE_GROUP" -n "$MIGRATION_JOB" \
  --yaml migration-execution.json --query name -o tsv)
az containerapp job execution show -g "$RESOURCE_GROUP" -n "$MIGRATION_JOB" \
  --job-execution-name "$EXECUTION" --query properties.status -o tsv

Inspect the generated template before starting it: preserve command/preload, environment, identity, volumes and secret references. Protect and clean up temporary configuration files. Poll the named execution to Succeeded and verify migration results. Stop on any error; repeated application restarts cannot install missing schema.

Roll out and verify

Set API_APP, WORKER_APP, WEB_APP and all three NEW_*_IMAGE variables to the approved resources/digests. Update API first, verify its latest revision, then Worker, then Web:

az containerapp update -g "$RESOURCE_GROUP" -n "$API_APP" --image "$NEW_API_IMAGE"
az containerapp update -g "$RESOURCE_GROUP" -n "$WORKER_APP" --image "$NEW_WORKER_IMAGE"
az containerapp update -g "$RESOURCE_GROUP" -n "$WEB_APP" --image "$NEW_WEB_IMAGE"

Treat each command as a separate gate, not an unattended sequence. Single revision mode shifts traffic after the new revision is healthy. Compare latest and latest-ready revision names, and inspect the exact revision’s image/health. Alternatively, after migration success, update the Terraform digest inputs and review/apply the resulting plan.

Complete installation verification, including scoped sign-in, licensing, queue search, an approved operation and item-level outcomes, audit and telemetry. Reconcile deployed digests into controlled Terraform inputs before the next apply so old values cannot undo the release. Keep deployment-existence flags true.

Rollback limits

Image rollback never reverses a database migration, a published message or a deletion. Even an additive migration is not proof that old images can run: readiness requires the exact application ledger and compatible queue schema expected by that image. Use a container-only rollback only after confirming database compatibility and retained image availability. The 1.7.0 → 1.7.1 source delta adds no runtime/migration changes, but still verify your actual deployed identities and state.

When compatible, redeploy the previous approved digests in API/Worker/Web order, verifying each new revision and the full workflow. Preserve the same credential/trust contract. If compatibility is unknown, stop and obtain a recovery plan; do not edit the migration ledger to make an old image appear ready.

Back up and restore

PostgreSQL is the single application datastore, including jobs, schedules, audit, sessions and throttling. Use Flexible Server backups/PITR with retention and HA chosen for your recovery objectives. Retain encryption keys, deployment configuration and approved images independently. Message Ops does not back up Solace messages; the broker owner must provide that recovery plan.

Perform restore drills into isolated infrastructure with matching images and keys. Verify data, restricted runtime grants, readiness and measured recovery time. Prevent the restored Worker from contacting production brokers during a drill: restored queued work can repeat external effects.

For a production restore:

1. Declare the incident, preserve diagnostic/broker evidence, block new requests and stop Worker processing. Mutation switches alone do not stop all work.
2. Choose and approve the recovery point. Restore PostgreSQL to a new Flexible Server using the site’s PITR/geo-restore procedure.
3. Re-establish private DNS/routes/TLS, separate migration/runtime credentials and matching encryption keys. Select application images compatible with the restored ledger.
4. Reconcile restored jobs and schedules against broker outcomes before enabling Worker. Database restore cannot undo messages already published or deleted; replaying restored work can duplicate them.
5. Review restored account/session/token state: a backup can reintroduce state from before logout or revocation. Reapply required revocations through the controlled recovery process.
6. Validate readiness and approved functional tests, then restore traffic. Record data loss, elapsed recovery time and follow-up actions.

A restore loses application changes after the recovery point, including audit and job history, while the broker may retain their effects. Do not restore the entire database solely to recover one administrator’s access. See Troubleshooting for lockout handling.