Common failures
Common reasons a training container fails on the platform, and what to do
Training step
| Symptom | Cause | Fix |
|---|---|---|
Exception right at start — active experiment ID does not match | The trainer calls mlflow.set_experiment() itself | Remove the call, or pass os.environ["MLFLOW_EXPERIMENT_NAME"] as is |
| No curves on the training screen | Metrics are only printed, or start_run() is called twice and logs to a different run | Record with mlflow.log_metrics(..., step=epoch), and use only one run |
| No progress bar | The param epochs or the metric step is missing | Add mlflow.log_param("epochs", N) |
| Training completes but model registration is skipped | Only the weight file was uploaded as an artifact and log_model() was not called | Log with mlflow.pyfunc.log_model(...) |
| Fails in the training step, with data-related messages in the log | The dataset does not match the label shape the trainer expects (for example the MNIST example expects meta.label) | Change it to read manifest.json and the annotation files — Paths and environment variables |
| Progress logs appear late, all at once | Output buffering | ENV PYTHONUNBUFFERED=1 in the Dockerfile |
| Checkpoints · reports disappear | Saved only in /geo/work | Upload them with mlflow.log_artifact() |
Serving step (fine during training)
| Symptom | Cause | Fix |
|---|---|---|
| Every inference request fails | Missing signature → inputs are converted to float64 | Build a signature with infer_signature() and pass it |
| The serving container does not start | predictor.py imports training code, or system packages are missing | Make predictor.py self-contained; list metadata.apt_packages |
| The checkpoint does not load | pip_requirements versions differ from the training environment | Pin exactly the training environment's versions; verify with --env-manager virtualenv |
Import · run preparation
| Symptom | Cause | Fix |
|---|---|---|
| Image import rejected | Saved without a tag, architecture mismatch, several images in the archive | Save again with docker save <name:tag>, build for linux/amd64, one image per file |
| Cannot be chosen in the variant list ("No image (이미지 없음)") | The image the variant points to is not in the registry | Check that the import is complete and the address is correct |
Submission rejected — 데이터셋 스테이저 변형이 없습니다 ("no dataset stager variant") | The tenant has no image to stage data | Ask the operator to publish (or share) a stager variant |
| Submission rejected — the runtime variant is for another framework | A variant with a framework set was chosen for training in another framework | Clear the external variant's framework (unrestricted) or choose a matching variant |
Prepare training environment (학습 환경 준비) step fails — namespaces "tenant-…" not found | The tenant namespace does not exist in the cluster | The operator creates the tenant-<tenant code in lower case> namespace |
| Training runs on an unexpected image | The framework has no default variant, so it fell back to the "Unrestricted (제한 없음)" default variant (an external image) | Keep a default variant per framework, or set a framework on the external variant — Make it selectable in the wizard |