Monitoring and Checkpoints
Where: Model Training → Task Management, then click Details on a job card.
Job List
The job list shows each training job as a card:
| Field | Description |
|---|---|
| Job name and state | See the state table below |
| Dataset | The dataset versions used for training |
| Resource queue | The compute queue the job runs in |
| Compute configuration | Instance and GPU specifications |
| Output model | The model name and version produced when training completes |
| Created at | — |
You can search the list by job name and switch to a table view.
Training States
| State | Meaning | Available actions |
|---|---|---|
| Queued | Waiting for available compute | Cancel |
| Training | In progress | Export checkpoint, stop |
| Completed | Ran the full step budget and produced a model version | View model, compare |
| Stopped | Stopped manually; exported checkpoints are kept | View logs |
| Failed | Terminated abnormally | View logs, create a new job |
Metrics
| Metric | Description | How to read it |
|---|---|---|
| Loss curves | Training and validation loss over steps | A steady decline that levels off is normal; a clear rise in validation loss suggests overfitting |
| Learning rate | The actual learning rate schedule | Should match the template |
| Throughput | Samples processed per second | If it's well below past jobs at the same tier, check data loading |
| GPU utilization and memory | Real-time usage per GPU | Persistently low utilization usually means data loading is the bottleneck |
| Steps and elapsed time | Current step / total steps, elapsed time | Use these to estimate the time remaining |
| Logs | Output from the training process | Use these to diagnose errors |
Exporting Checkpoints
You can export the current checkpoint at any time during training:
- On the details page, click Export Checkpoint.
- The platform registers the weights at the current step as a checkpoint model, labeled with the step and progress, for example "Step 50,000 · 62%".
- The checkpoint model appears in Model Management and can be deployed directly for an inference test.
| Rule | Description |
|---|---|
| Training continues | The job keeps running; its state stays Training and progress doesn't go back |
| No duplicates | Exporting the same step twice shows that it already exists |
| Not a release | Checkpoints are for checking direction. A model still needs evaluation and on-site validation before delivery. |
Recommended checkpoint cadence
Export a checkpoint at roughly 25%, 50%, and 75% progress and run an inference test on each. This surfaces data problems early, so you don't discover a wrong direction only after the full run.
Comparing Jobs
Click Compare in the job list and select several training jobs to overlay their loss curves and view their dataset versions, model families, and parameter snapshots side by side. Because the training strategy is unified, any difference reflects the data and the model family.
Stopping a Job
Click Stop on the details page. The job state changes to Stopped, and any exported checkpoint models are kept. A stopped job can't be resumed; create a new job if needed.
Troubleshooting Training
| Symptom | Check first | Resolution |
|---|---|---|
| Queued for a long time | Resource queue usage | Leave it queued; to adjust quotas, contact an administrator |
| Training failed | Errors in the logs | For data format errors, go back to the export step; for resource errors, resubmit |
| Loss isn't decreasing | The share of failed demonstrations in the data and the correctness of action labels | Go back to the data workbench, fix the data, and export a new version |
| Sudden spikes in loss | Anomalous samples | Inspect those samples, exclude them, and export again |

