Skip to content

Egocentric Data Collection ​

Egocentric data collection doesn't involve a robot. The demonstrator wears or holds a capture device and performs the task by hand. The devices are lightweight and can be taken into far more real-world environments. Data is stored as RAW and must be processed by a data pipeline into structured data before it can be curated or used for training.

Supported Devices ​

ModelForm factorWhat is captured
Robot-free EgoHead-mounted first-person stereo cameraLeft and right video, IMU, frame timestamps, stereo calibration
Robot-free UMIHandheld capture deviceStereo video, IMU, frame timestamps, calibration

See Data Formats for the RAW directory layout.

Prerequisites ​

RequirementDetails
Device registeredThe device model and SN are in the device inventory
Valid calibrationStereo calibration ships with the device and is updated during periodic recalibration
Collection taskA collection task bound to the device exists
Operators readyThe project's operator library contains the operators needed for processing. See Developing Operators.

Collecting Data ​

  1. The collector claims the task in Collection Tasks and clicks Start Collecting.
  2. Wear or hold the device and make sure both hands and the target objects are fully in view.
  3. Finish the demonstration and end the episode; the data uploads automatically.
  4. For long continuous sessions, use segmented upload: the recording is split into 5-minute segments and uploaded while recording, minimizing loss if the session is interrupted.

Capture Guidelines ​

GuidelineDetails
Keep hands in viewKeep both hands and the objects in frame throughout the key parts of the task
Avoid rapid head turnsFast head or device motion causes motion blur, which degrades SLAM and keypoint results
Stable lightingAvoid strong backlighting and frequent changes in brightness
Sufficient textureTextured surroundings help visual localization; accuracy drops near plain, uniform walls
Hold still before startingKeep the device still for about one second before each episode to aid initialization

After Collection: Pipeline Processing ​

Egocentric data is ingested with status Awaiting Pipeline and can't be selected for curation directly. The processing path is:

text
RAW ingested (Awaiting Pipeline) → Create curation project → Pipeline processing (SLAM / vision / depth, etc.) → Outputs ingested (Ready to Curate) → Curation and export

For the full walkthrough, see Tutorial A: From Raw Data to Annotated Data.

Common Issues ​

IssueResolution
Items can't be selectedRAW data must be processed by a pipeline first
SLAM trajectory driftsCheck for fast motion or low-texture scenes during capture; recollect if needed
Keypoints are lostCheck whether the hands left the frame or were occluded

NexCore is evolving rapidly; the latest version prevails.NexCore Docs · Embodied AI Skill Learning Platform by Lumos Robotics