Backup and restore
A systemd timer that takes a Postgres dump and a RustFS mirror every night, and how to restore from them
All Geo-MLOps state lives in two places: Postgres (the DB shared by the app and MLflow) and RustFS (datasets, image layers, model artifacts, build logs, edge uploads). Both exist only in Docker volumes, so anything deleted by mistake — such as a permanently deleted MLflow experiment or a deleted tenant — cannot be recovered without a backup. Compose does not take backups, so install the backup script and systemd timer that ship with the source.
What is backed up, and how
| Target | Method | Retention |
|---|---|---|
| Postgres | pg_dump -Fc inside the container → BACKUP_DIR/postgres/mlflow-<UTC-time>.dump. Each dump is read back with pg_restore --list right away; broken files are not kept | Dumps older than BACKUP_KEEP_DAYS (default 14 days) are deleted |
| RustFS | aws s3 sync per bucket → BACKUP_DIR/objects/<bucket>/ | Additive only (no --delete). Deleted objects stay in the mirror too. Clean up by hand when space runs low |
- It runs every day at 03:30 (with a random delay of up to 15 minutes). A night's backup missed while the server was off is caught up at the next boot.
- It runs at low priority (
Nice=10, idle I/O) so it does not compete with training and requests for the disk. pg_dumpis the one inside the Postgres container, and the S3 copy uses an AWS CLI container. Nothing needs to be installed on the server.
Installation (once, on the server)
-
Install the script and the unit files.
D=/opt/geo-mlops/deploy/backup sudo install -m 0755 $D/backup.sh /usr/local/sbin/geo-mlops-backup sudo install -m 0644 $D/geo-mlops-backup.service /etc/systemd/system/ sudo install -m 0644 $D/geo-mlops-backup.timer /etc/systemd/system/ -
If you want anything different from the defaults, write it in
/etc/geo-mlops/backup.env(the file is optional).sudo tee /etc/geo-mlops/backup.env >/dev/null <<'ENV' BACKUP_DIR=/mnt/backup/geo-mlops BACKUP_KEEP_DAYS=14 ENV -
Enable the timer and run the first backup now to check it.
sudo systemctl daemon-reload sudo systemctl enable --now geo-mlops-backup.timer sudo systemctl start geo-mlops-backup.service journalctl -u geo-mlops-backup.service -n 20 # last line "backup finished" systemctl list-timers geo-mlops-backup.timer # next run time
Values you can change in backup.env
| Variable | Default | Meaning |
|---|---|---|
BACKUP_DIR | /var/backups/geo-mlops | Where backups accumulate |
BACKUP_KEEP_DAYS | 14 | Days to keep DB dumps |
BACKUP_BUCKETS | (empty = all buckets) | Bucket names separated by spaces |
BACKUP_SKIP_OBJECTS | 0 | 1 backs up the DB only |
PG_CONTAINER / PG_USER / PG_DB | geo-mlops-mlflow-postgres / mlflow / mlflow | Postgres container and account |
S3_ENDPOINT / S3_ACCESS_KEY / S3_SECRET_KEY | http://localhost:9000 / rustfsadmin / rustfsadmin | RustFS connection details. If you changed the account in Compose, change these too |
AWS_CLI_IMAGE | amazon/aws-cli:2.34.63 | Image used for the S3 copy |
Restore
Database
Stop the app so nothing writes to the DB, then restore the dump.
docker stop geo-mlops-app
docker exec -i geo-mlops-mlflow-postgres \
pg_restore -U mlflow -d mlflow --clean --if-exists --no-owner \
< /var/backups/geo-mlops/postgres/mlflow-<UTC-time>.dump
docker start geo-mlops-appObjects
Upload one bucket (or a prefix inside it) back from the mirror. The mirror is additive only, so objects deleted after the backup time also come back — upload only the prefixes you need.
docker run --rm --network host -v /var/backups/geo-mlops/objects:/backup \
-e AWS_ACCESS_KEY_ID=rustfsadmin -e AWS_SECRET_ACCESS_KEY=rustfsadmin \
-e AWS_DEFAULT_REGION=us-east-1 -e AWS_REQUEST_CHECKSUM_CALCULATION=when_required \
amazon/aws-cli:2.34.63 --endpoint-url http://localhost:9000 \
s3 sync /backup/datasets/<tenant_id>/<dataset_id>/ s3://datasets/<tenant_id>/<dataset_id>/Restore drill (every quarter)
A backup you do not know you can restore is not a backup. Once a quarter, check somewhere other than the production server that the dump can be read.
BACKUP_DIR=/tmp/geo-backup-drill BACKUP_BUCKETS=build-logs /usr/local/sbin/geo-mlops-backup
docker exec -i geo-mlops-mlflow-postgres pg_restore --list \
< /tmp/geo-backup-drill/postgres/mlflow-*.dump | headNext: Reverse proxy caveats