Skip to content

Data Pipeline Overview ​

A data pipeline is a set of operators arranged by their dependencies that turns raw recordings into structured data. Each run is a processing job with its own pipeline job ID, progress, logs, and outputs.

When You Need a Pipeline ​

DataPipeline required?Why
RAW (Robot-free Ego, Robot-free UMI)RequiredRAW contains only raw sensor data. Operators must produce structured data such as trajectories, depth, and keypoints before it can be curated or used for training.
HDF5 (Station Lite, MOS)OptionalHDF5 already includes robot state and can be curated directly. Run a pipeline only when you need additional visual or depth information.

Operator Types ​

TypeInputTypical outputs
SLAMStereo video, IMU, calibrationCamera trajectory and pose sequence
VisionVideo framesObject detection boxes, segmentation masks, hand keypoints
DepthStereo or monocular videoPer-frame depth maps
TactileTactile sensor signalsAligned tactile sequences and features
Multi-operatorAny combination of the aboveMultiple output types

R&D engineers upload operators to the project's operator library. You can use the operators provided by the platform or write your own following Developing Operators.

Pipeline Chain ​

Every pipeline is shown as three stages:

text
Prerequisites ──▶ Operator execution ──▶ Post-processing
(decoding, time      (selected operators run      (formatting, writing back to
 alignment,           in dependency order)          the asset pool, generating
 loading calibration)                               previews)

Processing Jobs and Outputs ​

How data assets, curation projects, processing jobs, outputs, and dataset versions relate

  • A processing job is started from a curation project and processes the items selected in that project.
  • Processing is asynchronous, so you can leave the page after submitting.
  • A job completes only when every required operator succeeds. Otherwise the whole batch is marked failed, and you can retry the failed steps.
  • Outputs are written back to the asset pool, tagged with the pipeline job ID. The original data is preserved.
  • Outputs can then be labeled and reviewed in the workbench and exported as a dataset version.

In This Section ​

NexCore is evolving rapidly; the latest version prevails.NexCore Docs · Embodied AI Skill Learning Platform by Lumos Robotics