Skip to content

Validation and Evaluation ​

After deploying a model, you need to confirm that it completes the task reliably in the real environment. NexCore offers two ways to do this: on-site validation (inference tests and direct observation) and NexBench evaluation (automatic scoring by execution node and step).

On-Site Validation ​

Define the Test Conditions ​

Set the conditions before you validate, so the results are meaningful:

ConditionExample
Task definitionMove a cube from the tray into the box; success means the cube ends up inside the box
Scene rangeCube placed randomly within a 30 cm × 20 cm area of the tray
Number of trials20 trials per model version
EnvironmentNormal indoor lighting, unchanged background

Run the Validation ​

  1. Click Inference Test on the deployment instance and enter the test scenario and number of trials.
  2. Complete the safety confirmation and start the test.
  3. On-site staff observe each trial and use the emergency stop if needed.
  4. When the test ends, review the execution records in the inference job details.

Record the Results ​

At a minimum, record the test conditions, number of trials, number of successful trials, failure observations, and the model version. Failure segments are kept automatically in inference feedback for review.

NexBench Evaluation ​

NexBench is evaluation built into each inference job. It collects data and scores the run by execution node and execution step, so you can pinpoint where failures occur and compare model versions.

Turning It On ​

  • Turn on NexBench when creating a deployment or inference job. It's off by default.
  • The setting is saved in the deployment record.
  • With NexBench off, execution data still flows back to the platform; it just isn't scored.
Controlled by
Whether a run is scoredThe NexBench toggle
Whether data flows backAlways on; no toggle needed

Metrics ​

When enabled, results are reported by node and execution step:

MetricMeaning
Success rateThe share of trials in which the node succeeded
Cycle timeThe node's average execution time
AccuracyThe node's positioning or placement error
  • Missing nodes are marked Not Evaluated and are not scored as zero.
  • Low-scoring nodes are added to the bad case list automatically and can be returned to curation after review.

Cross-Job Summary ​

Model Evaluation → NexBench in the left navigation provides a summary across inference jobs, where you can:

  • compare node success rates and cycle times across model versions;
  • track how scores for the same task change over time;
  • jump from summary results to the corresponding inference jobs and bad cases.

Using the Results ​

  • Evaluation results are viewed in the platform and used to guide quality improvements.
  • Success rate, accuracy, and cycle time depend on the task definition, data volume, and test conditions; results from different conditions aren't directly comparable.
  • Formal acceptance follows the test conditions agreed by both parties; evaluation results are one input to it.

Recommendation

Turn on NexBench during model validation and use node scores to find where failures occur. On cycle-time-sensitive production lines, enable it in a validation environment first before deciding whether to use it in production.

Common Issues ​

IssueResolution
Inference results have no scoresNexBench wasn't turned on for that inference job
A node shows Not EvaluatedThe node didn't run or no data was captured in that trial; it isn't scored as zero
Two model versions have very different success ratesConfirm the test conditions were the same, then review the bad cases for the failing nodes

NexCore is evolving rapidly; the latest version prevails.NexCore Docs · Embodied AI Skill Learning Platform by Lumos Robotics