Skip to content

Data Management Overview ​

Data management turns uploaded data into assets that are traceable, curatable, and ready for training. It has five parts:

ModulePurposeWhere
Data UploadThree upload modes (full, screened, segmented) and two storage formats (HDF5 and RAW)Selected when creating a collection task
Data AssetsThe master list of all ingested data, filtered first by collection taskData Curation → Data Assets
Curation ProjectsAn editable workspace that references a set of dataData Curation → Curation Projects
Data WorkbenchPreview, label, trim, review, and batch-process dataData Curation → Data Workbench
Datasets and VersionsFrozen snapshots exported from curation projects, used for training and deliveryData Curation → Dataset Management

How Data Flows ​

How data assets, curation projects, processing jobs, outputs, and dataset versions relate

  1. Data that finishes uploading and passes integrity checks becomes a data asset with a permanent data ID.
  2. Select data assets to create a curation project.
  3. In the data workbench, flag quality issues, define keep and exclude ranges, and review each item. RAW data goes through a data pipeline first.
  4. When curation is complete, export a dataset version for training or delivery.

Data Curation ​

Curation in NexCore never alters source data:

ActionWhat actually happens
Add a labelA metadata record is created
Mark an excluded rangeAn export instruction is recorded
Exclude an entire episodeAn export instruction is recorded
Run a pipelineNew outputs are generated; the original data is kept
Export a datasetNew files are generated from the recorded instructions

The original data stays unchanged after any action, so you can always return to it and curate again from scratch.

Data States ​

StateMeaningCan be added to a curation project?
Ready to CurateHDF5 data, or outputs that have been through a pipeline✓
Awaiting PipelineRAW data that hasn't been through a pipeline yet✗
Validation FailedUpload integrity checks did not pass✗
ExportedReferenced by at least one dataset version✓

Isolation and Security ​

  • Data is isolated by project; searches never return data from other projects.
  • Exports, deletions, reviews, and similar actions record who performed them and when.
  • Data is stored on the storage nodes in its region. A private deployment is available if data must stay on premises.

NexCore is evolving rapidly; the latest version prevails.NexCore Docs · Embodied AI Skill Learning Platform by Lumos Robotics