Operations checklist | Geo-MLOps Manual

The checklist below is the minimum set of habits that keeps a running Geo-MLOps healthy. Each item links to the screen to look at.

Daily (5 minutes)

Who	What to check	Where	If something is wrong
Global administrator	Badges on the system home cards — overall diagnostics verdict, GPU pods waiting, tenant deletions partially failed	System home	Go into that screen and find the cause
Global administrator	Are all dependent components healthy?	System diagnostics	Check again (다시 점검), check the logs of that service
Global administrator	Firing alerts	Alertmanager	During maintenance, silence with narrow matchers
Global administrator	All nodes Ready, Pod Ready ratio	Cluster	Check node conditions, cordon if needed
Global administrator	GPU pods waiting for a long time	GPU slots	Adjust wiring and pool slot counts, ask for stuck training to be cleaned up
Global administrator	Is a package proxy "Dead (죽음)"?	Build cache	If a proxy is dead every build fails — restart the container
Tenant administrator	The three readiness lines, training Jobs running too long or stuck	Admin home, Training runner	Stop (중지) → if that does not work, Force kill (강제 종료)

Who	What to check	Where
Tenant administrator	Anything unfamiliar in last week's role changes and token issuance (`role.`, `mlflow_token.`, `registry_token.`)	Audit log
Tenant administrator	Tokens expiring soon, tokens with no expiry, tokens of people who left	Tokens
Tenant administrator	Memberships of people who left or whose role changed	Members
Tenant administrator	Dataset caches unused for a long time, volumes left as `Released`	Dataset cache
Global administrator	Reclaimable build layers, orphaned weight objects	Build cache, Weights mirror
Global administrator	Volumes whose PVC usage is close to the limit	Storage
Global administrator	Did the backup timer run? — `backup finished` in `journalctl -u geo-mlops-backup.service -n 20`	Server shell
Global administrator	Any new CRITICAL or WARNING in configuration hygiene?	System diagnostics

Restore drill — actually restore a backup. On a development server, make a backup once and check the list of dumps.
```
BACKUP_DIR=/tmp/geo-backup-drill BACKUP_BUCKETS=build-logs deploy/backup/backup.sh
docker exec -i geo-mlops-mlflow-postgres pg_restore --list \
    < /tmp/geo-backup-drill/postgres/mlflow-*.dump | head
```
The DB dumps and the object mirror point to each other, so when restoring, use a dump and mirror from the same day together.
Keep the audit log — download last quarter's Global audit log as CSV and keep it.
Permission review — in Users, check that only people who really need it are global administrators, and that each tenant has two or more ADMINs.
Shared image cleanup — clean up unused shared variants and own variants, and check that no tenant has an empty default ("Defaults missing (기본 누락)").
GPU wiring review — adjust shared pool slot counts and dedicated slots to match each tenant's actual usage.
Retention review — check that the retention periods for notifications, build logs and edge records (GEO_MLOPS_*_RETENTION_DAYS) fit your disk situation. Audit log retention is unlimited by default.