9. Monitoring, drift and retraining
Watch operational metrics and drift, and retrain a model whose performance has dropped
After shipping a model, you watch two things: is the service running well (monitoring), and has the incoming data changed from the training data (drift). If it has, you retrain with new data and go around the cycle again.
Monitoring
- Open Monitoring (모니터링) in the sidebar. At the top right, choose the model, edge device and period (1h, 24h, 7d, 30d). The four tiles at the top are RPS, error rate, p95 latency (95% of requests finish within this time) and data latency; below them are the Training queue (학습 큐) and GPU slots (GPU 슬롯) trends. When you choose a model, Model performance (모델 성능) and input distribution (PSI, KS) charts are added.
Monitoring — operational KPIs, training queue and GPU slots; pick a model for performance and input distribution
Situations that cross a threshold show up in Alerts (경보). If there are too many notifications, see Silence alerts.
Drift
PSI (Population Stability Index) is a value that shows how far the input data distribution has moved from the baseline. The closer to 0, the more similar; the color changes when it crosses the tenant policy's warning threshold (default 0.1) or critical threshold (default 0.2). The KS test checks the same question with a different statistic.
- In the sidebar, open Drift (드리프트) to see each model's latest PSI and KS and its status (Drift (드리프트) or Healthy (정상)). Click a model.
Drift list — latest PSI, KS and status per model - In PSI trend (30 days) (PSI 추이 (30일)) ①, see from when the bars turned yellow (warning) or red (critical). CT trigger history (CT 트리거 이력) (CT stands for continuous training) lists the times a threshold was crossed and a retraining trigger was recorded, and Drift policy (드리프트 정책) holds the thresholds (only admins can edit the policy). To retrain, press Retrain this model (이 모델 다시 학습하기) ②.
Drift detail — ① PSI trend coloured by policy thresholds ② Retrain this model
Around the cycle again
There are two quick ways to retrain.
- Add new data — upload photos collected in the field to the same dataset from 2. Dataset, and the manifest is rewritten automatically. Open the wizard with Train with this dataset (이 데이터셋으로 학습) on the dataset detail.
- Reuse the last settings — press Train again with these settings (이 설정으로 다시 학습) on the last training run's detail, and you go to the review step with all 11 steps filled in. See Train again with the same settings.
If you use the same experiment name and registered model name, the new run piles up in the same experiment and the new version is attached to the same model, so you can compare it with the previous version right away in 5. Compare runs. If it is better, continue again from 6. Promotion request.
Well done. From here, look up common tasks in How-to.