Skip to content

Data Backup Strategy

Governance evidence must survive infrastructure failure, accidental deletion, and operator error. Backup frequency, retention, encryption, and restore testing are customer operational responsibilities. This guide defines what to protect and the recommended mechanism for each data store, so a backup strategy can be designed alongside the deployment rather than after it.

For the operational expectations around when to back up and how to validate a restore, see Backup and Restore.

Data storeContentsMechanism
PostgreSQL governance databaseOrganizations, users, projects, policies, controls, declarations, and reviewsDatabase backups / snapshots
PostgreSQL IntegrityServiceDB databaseIntegrity records and certificate metadataDatabase backups / snapshots
Governance object storage (bucket or container)Governance documents and file attachmentsVersioning + replication
Integrity object storage (rootstore)Integrity artifacts and manifestsVersioning + replication
Key management (AWS KMS / Azure Key Vault)DID signing keysDeletion protection (keys are not exportable)
Configuration and secretsvalues.yaml, secrets.yaml, Kubernetes secrets, release manifestVersion control + secret manager

The PostgreSQL databases and the object storage contents reference each other (attachments are stored in object storage, their metadata in the database). Back them up on the same schedule so a restore produces a consistent pair.

Section titled “External Database (Recommended for Production)”

When using a managed database as described in the External Database guide, use the provider’s native backup features:

  • AWS RDS: enable automated backups with at least 7 days retention (--backup-retention-period 7), which also enables point-in-time recovery. Take a manual snapshot before every platform upgrade:

    Terminal window
    aws rds create-db-snapshot \
    --db-instance-identifier governance-platform \
    --db-snapshot-identifier governance-pre-upgrade-$(date +%Y%m%d)
  • Azure Database for PostgreSQL - Flexible Server: configure the backup retention period (7-35 days) and choose geo-redundant backup storage for cross-region recovery

  • GCP Cloud SQL: enable automated backups and point-in-time recovery on the instance

Both platform databases (governance and IntegrityServiceDB) live on the same instance, so instance-level snapshots cover both.

The bundled in-cluster PostgreSQL has no built-in backup mechanism – this is one of the main reasons production deployments should use an external database. If you must run the bundled instance, schedule logical dumps of both databases to object storage:

Terminal window
kubectl exec statefulset/governance-platform-postgresql \
--namespace governance -- \
bash -c 'PGPASSWORD="$POSTGRES_PASSWORD" pg_dumpall -U postgres --clean' \
> governance-platform-$(date +%Y%m%d).sql

Run this from a scheduled job (for example, a Kubernetes CronJob) and copy the output to a versioned bucket. Volume snapshots of the PostgreSQL persistent volume via your CSI driver are a useful complement, but only logical dumps are portable across storage classes and clusters.

Both the governance bucket/container and the integrity rootstore should have:

  • Versioning enabled, so overwrites and deletions are recoverable
  • Replication to a second region or account for disaster recovery
  • Lifecycle rules that expire old noncurrent versions to control cost

Provider specifics:

  • AWS S3: aws s3api put-bucket-versioning --bucket <bucket> --versioning-configuration Status=Enabled, plus a replication rule to a bucket in another region
  • Azure Blob Storage: enable blob versioning and soft delete on the storage account; use object replication or RA-GRS redundancy
  • GCS: enable object versioning; use dual-region buckets or a transfer job to a second bucket

DID signing keys in AWS KMS and Azure Key Vault are not exportable – they cannot be backed up in the conventional sense. The strategy is preventing loss instead:

  • AWS KMS: keys are created with a deletion window (7 days by default in the platform configuration), so a scheduled deletion can be cancelled. Alert on ScheduleKeyDeletion events via CloudTrail
  • Azure Key Vault: enable soft-delete and purge protection on the vault, so deleted keys are recoverable for the retention period and cannot be force-purged

Losing signing keys does not lose governance data, but it breaks signing for existing DIDs. Treat key deletion alerts as critical.

  • Keep values.yaml in version control – it contains no credentials
  • Keep secrets.yaml in an approved secret manager (never in version control)
  • Record the deployed chart version and image tags from the release manifest with each backup, so a restore can redeploy the matching versions

The platform encryption key (platform-encryption-key) deserves special attention: data encrypted with it is unrecoverable if the key is lost, even from a valid database backup. Ensure it is captured in the secret manager copy of secrets.yaml, and verify the stored copy matches the in-cluster secret after any rotation.

ItemFrequencyRetention
PostgreSQL automated backupsDaily (provider-managed)7-35 days
PostgreSQL manual snapshotBefore every upgrade or maintenanceUntil the following successful upgrade
Object storage replicationContinuousGoverned by lifecycle rules
Configuration and secretsOn every changeFull history
Restore testQuarterly-

Tighten frequency and retention to match your organization’s recovery point objectives and compliance requirements.

Restore components in this order so services agree on identity, storage, keys, and data state:

  1. Provision infrastructure (cluster, network, database instance, buckets, vault access)
  2. Restore secrets and configuration (secrets.yaml values, Kubernetes secrets, CA bundles)
  3. Restore the PostgreSQL databases from the chosen snapshot or dump
  4. Restore or re-point object storage (replicated bucket or restored versions) – from the same point in time as the database
  5. Deploy the chart using the values and chart version recorded in the release manifest
  6. Validate sign-in, project access, attachments, reports, indicators, lineage, and certificates before returning the platform to normal use

A backup that has never been restored is not a backup. Quarterly, restore the latest database snapshot and a sample of object storage content into an isolated environment, deploy the chart against it, and run the validation checks above. Record the time taken - that is your actual recovery time objective.