Skip to content

Monitoring and Checkpoints ​

Where: Model Training → Task Management, then click Details on a job card.

Job List ​

The job list shows each training job as a card:

FieldDescription
Job name and stateSee the state table below
DatasetThe dataset versions used for training
Resource queueThe compute queue the job runs in
Compute configurationInstance and GPU specifications
Output modelThe model name and version produced when training completes
Created at—

You can search the list by job name and switch to a table view.

Training States ​

StateMeaningAvailable actions
QueuedWaiting for available computeCancel
TrainingIn progressExport checkpoint, stop
CompletedRan the full step budget and produced a model versionView model, compare
StoppedStopped manually; exported checkpoints are keptView logs
FailedTerminated abnormallyView logs, create a new job

Metrics ​

MetricDescriptionHow to read it
Loss curvesTraining and validation loss over stepsA steady decline that levels off is normal; a clear rise in validation loss suggests overfitting
Learning rateThe actual learning rate scheduleShould match the template
ThroughputSamples processed per secondIf it's well below past jobs at the same tier, check data loading
GPU utilization and memoryReal-time usage per GPUPersistently low utilization usually means data loading is the bottleneck
Steps and elapsed timeCurrent step / total steps, elapsed timeUse these to estimate the time remaining
LogsOutput from the training processUse these to diagnose errors

Exporting Checkpoints ​

You can export the current checkpoint at any time during training:

  1. On the details page, click Export Checkpoint.
  2. The platform registers the weights at the current step as a checkpoint model, labeled with the step and progress, for example "Step 50,000 · 62%".
  3. The checkpoint model appears in Model Management and can be deployed directly for an inference test.
RuleDescription
Training continuesThe job keeps running; its state stays Training and progress doesn't go back
No duplicatesExporting the same step twice shows that it already exists
Not a releaseCheckpoints are for checking direction. A model still needs evaluation and on-site validation before delivery.

Recommended checkpoint cadence

Export a checkpoint at roughly 25%, 50%, and 75% progress and run an inference test on each. This surfaces data problems early, so you don't discover a wrong direction only after the full run.

Comparing Jobs ​

Click Compare in the job list and select several training jobs to overlay their loss curves and view their dataset versions, model families, and parameter snapshots side by side. Because the training strategy is unified, any difference reflects the data and the model family.

Stopping a Job ​

Click Stop on the details page. The job state changes to Stopped, and any exported checkpoint models are kept. A stopped job can't be resumed; create a new job if needed.

Troubleshooting Training ​

SymptomCheck firstResolution
Queued for a long timeResource queue usageLeave it queued; to adjust quotas, contact an administrator
Training failedErrors in the logsFor data format errors, go back to the export step; for resource errors, resubmit
Loss isn't decreasingThe share of failed demonstrations in the data and the correctness of action labelsGo back to the data workbench, fix the data, and export a new version
Sudden spikes in lossAnomalous samplesInspect those samples, exclude them, and export again

NexCore is evolving rapidly; the latest version prevails.NexCore Docs · Embodied AI Skill Learning Platform by Lumos Robotics