Skip to content

Upload Modes and Storage Formats ​

Upload mode, storage format, and device SN are three separate decisions made at different levels:

DimensionDetermined byWhat it controls
Upload modeSelected when creating the taskHow data is sent back to the platform
Storage formatSet automatically by device modelThe format in which data is ingested
Device SNSelected when creating the taskWhich physical device the data comes from

Three Upload Modes ​

ModeHow it worksAdvantageTrade-offBest for
Full uploadUploads every episode in fullMost complete dataHighest bandwidth useSmall batches of high-value tasks
Screened uploadThe client filters out clearly invalid segments before uploadingSaves bandwidth and storageRelies on collector judgment; valid data may be droppedHigh-volume tasks with experienced collectors
Segmented uploadSplits recordings by duration and uploads while recordingSuited to long sessions; little is lost if interruptedA segment boundary may cut through an actionAll-day operation and continuous operation data

All three modes are available for every device and are independent of the device model.

Full Upload ​

  • Each episode is uploaded in full when it ends.
  • Failed uploads are retried automatically. If the retries also fail, the episode is flagged in the task details and can be re-uploaded manually.
  • No extra configuration is needed.

Screened Upload ​

  • Before uploading, the client applies screening rules to filter out clearly invalid segments, such as all-black video or large gaps in joint data.
  • Screening checks every frame, so overall upload is slower than full upload.
  • Screening rules are configured and validated by R&D engineers for each customer scenario. The platform does not ship generic thresholds.

Screening is not a substitute for curation

Screening is only a coarse filter. Data uploaded with screening still needs quality labeling and human review in a curation project.

Segmented Upload ​

  • Long recordings are split into fixed-length segments and uploaded one by one. The minimum segment length is 5 minutes.
  • Uploads resume after interruptions, and duplicate or out-of-order uploads never create duplicate samples.
  • When an action spans two segments, the platform stitches it back into one sample during processing.

Choosing a Mode ​

text
Does the task run all day?
├─ Yes → Segmented upload
└─ No → Is this a small batch of high-value data?
        ├─ Yes → Full upload
        └─ No → Are the collectors experienced and consistent?
                ├─ Yes → Screened upload
                └─ No → Full upload

Two Storage Formats ​

Storage formatDevicesCharacteristicsCurate directly?
HDF5Station Lite, MOS (teleoperation)Robot state (joints, grippers, end-effector poses) and multiple camera streams, aligned by timestamp✓
RAWRobot-free Ego, Robot-free UMIRaw sensor data (images, IMU, and so on) without robot state✗ Pipeline required
  • The storage format is set automatically by the device model. No format option is offered when creating a task, and it can't be changed afterward.
  • The format column in the asset list shows only HDF5 or RAW.
  • HDF5 is also one of the dataset export formats, but the two belong to different stages: the storage format is determined at collection, while the export format is chosen when exporting a dataset.

See Data Formats for file layouts.

Bandwidth Guidelines ​

These are typical figures for a single device collecting continuously. Actual values depend on the number of cameras, resolution, and compression settings.

DeviceData per 1-minute episode (approx.)Recommended uplink
Station Lite150–250 MB≥ 20 Mbps
MOS200–350 MB≥ 30 Mbps
Robot-free Ego100–200 MB≥ 20 Mbps
Robot-free UMI80–150 MB≥ 10 Mbps

When bandwidth is insufficient, data is cached on the device and queued for upload, so collection itself isn't affected. Task progress counts only completed uploads.

NexCore is evolving rapidly; the latest version prevails.NexCore Docs · Embodied AI Skill Learning Platform by Lumos Robotics