Training runner · Dataset cache
Check, stop and force-kill this tenant's training Jobs, and clean up the training dataset cache (PVC)
Required permission: SettingsADMIN
A tenant with the training feature turned on has two screens for operating training. Training is created on the workspace's Training (학습) screen; here you look after what is running and what has piled up.
Training runner
All training runs as Kubernetes Jobs. The Training runner screen shows this tenant's runner state and the training Jobs running or waiting now. Other tenants' Jobs are not shown.
| Field | Meaning |
|---|---|
| Runner · Cluster | Runner type (kubernetes) and whether the cluster is available |
| Running · Waiting | Number of training runs running now and number waiting their turn |
| Concurrency limit · Queue size | How many can run at once and how many are in line |
The Active Jobs (활성 Job) table shows the training run, state, current step, Job name and owner, with control buttons on the right.
| Button | What it does |
|---|---|
| Stop (중지) | The trainer finishes the current epoch, saves a checkpoint (last.pt) and ends safely. If it exceeds the grace period, it is force-killed |
| Force kill (강제 종료) | Deletes the Job right away. Use it only for a trainer that is stuck and not responding |
Dataset cache
Downloading a dataset into the pod before training takes time, so a dataset downloaded once is kept as a cache on a volume (PVC) in the tenant namespace so that the next training run starts right away. In exchange it takes up cluster storage, so clean it up from time to time.
- Usage (사용량) card: total cache size, number of claims and the per-claim limit, and the automatic cleanup rule (for example, cleaned up automatically after 14 days without use). Usage is not measured; it is the sum of the sizes of the staged files.
- Cluster cache (PVC) (클러스터 캐시 (PVC)) table: claim per dataset, usage/limit, last used, and On delete (삭제 시) (the volume reclaim policy).
- Volumes (PV) (볼륨 (PV)) table: volumes that remain after their claim is gone.
- Press Clear (비우기) on the row of a dataset that has not been used for a long time to delete its cache. To delete everything, use Clear all (전체 비우기).
- If a row's On delete is
Retain, deleting the claim leaves the volume asReleased, still taking up disk. Press Delete (지우기) for that volume in the Volumes (PV) table below to actually reclaim it.