Data Collection Workflow

For recording data, we provide several tools. The envisioned workflow follows a standard data collection and annotation pipeline. On this page we explain the different ways in which you can gather the data and launch the annotation tool. On the next page, we present the interface of the annotation tool and the annotation process in more detail.

We support three workflows:

In-the-loop collection and annotation: You can integrate our tool directly in your evaluation loop. The tool will open a browser prompt and ask you to directly annotate the episode with success/failure information after the robot completes an episode.
Bulk collection and annotation: If annotating in the loop does not fit your work style, you can also simply save your data using our EpisodeRecorder and launch the annotation tool as a stand-alone script after you are done with your evaluation runs on the robot.
Custom collection and bulk annotation: If your setup is incompatible with our EpisodeRecorder, or if you have already collected data and simply want to format it into the Oopsie Data format for annotation and submission, please see the detailed information on data formatting and data conversion. The web annotator tool can work with any dataset that is saved in the specified format. We keep a growing list of scripts to convert common data formats such as RLDS in our github repository.

For workflows 1. and 2., we provide example scripts in examples/inference_examples. These are derived from evaluation scripts in popular frameworks. If the one you would like to use is missing, please feel free to create an issue or pull request to add it.

Recording data with our toolkit

1. In-the-loop collection and annotation

For interactive annotation after each rollout using the browser UI, you can use the following code snippet as a template.

In-the-loop code template. Click to expand!from oopsie_tools.annotation_tool.rollout_annotator import WebRolloutAnnotator
from oopsie_tools.utils.robot_profile import *

robot_profile = load_robot_profile(<path_to_robot_profile>)
rollout_annotator = WebRolloutAnnotator(
    robot_profile=robot_profile,
    data_root_dir=<path_to_data_save_dir>,
    port=<port_for_web_annotator>,
    wait_for_annotation=<halt_robot_on_annotation>,
    resume_session_name=<optional_existing_session_name>, # Optional: None for new session
    operator_name=<robot_operator_name>, # The name of the person running the experiment
    annotator_name=<optional_annotator_name>, # Optional: Defaults to operator_name if None
)
rollout_annotator.start()

# policy initialization code
# ...

for _ in range(num_eval_episodes):
    # policy reset code
    # ...


    rollout_annotator.reset_episode_recorder()
    instruction = rollout_annotator.wait_for_task()

    for obs, action in run_policy(env, policy, instruction):
        # step policy and robot environment
        # ...

        rollout_annotator.record_step(
            observation={
                "image_observation": {
                    "<camera_name>": rgb_array,
                },
                "robot_state": {
                    "<state_key>": state_array,
                },
            },
            action={
                "<action_key>": action_array,
            },
        )


    annotation = annotator.finish_rollout(
        instruction=instruction,
    )
    # annotation is saved automatically in the hdf5 file, we simply return 
    # it for optional further use by the user

2. Bulk collection

Bulk collection will only record your session data in the oopsie-data format without launching the annotator. You can still annotate the collected data later by launching the annotation tool manually. This is detailed on the annotation tool instruction page.

Bulk collection. Click to expand!from oopsie_tools.annotation_tool.episode_recorder import EpisodeRecorder
from oopsie_tools.utils.robot_profile import *

robot_profile = load_robot_profile(<path_to_robot_profile>)
episode_recorder = EpisodeRecorder(
    robot_profile=robot_profile,
    data_root_dir=<path_to_data_save_dir>,
    resume_session_name=<optional_existing_session_name>, # None for new session
    operator_name=<robot_operator_name>,
)

# policy initialization code
# ...

for _ in range(num_eval_episodes):
    # policy reset code
    # ...
    episode_recorder.reset_episode_recorder()
    for step in rollout:
        # step policy and robot environment
        # ...
        episode_recorder.record_step(
            observation={
                "image_observation": {
                    "<camera_name>": rgb_array,
                },
                "robot_state": {
                    "<state_key>": state_array,
                },
            },
            action={
                "<action_key>": action_array,
            },
        )

    # wrap up policy rollout
    # ...

    episode_recorder.finish_rollout(instruction=instruction)

ToolKit API

`WebRolloutAnnotator`

WebRolloutAnnotator(
    robot_profile: RobotProfile,
    data_root_dir Path,
    operator_name: str,
    port: int = 5001,
    annotator_name: str | None = None,
    wait_for_annotation: bool = True,
    open_browser: bool = True,
    resume_session_name: str | None = None,
)

Parameter	Type	Default	Description
`robot_profile`	`RobotProfile`	—	Robot and policy metadata (see Robot Profile)
`data_root_dir`	`Path`	—	Directory where episode HDF5 and video files are written
`operator_name`	`str`	—	Name of the person running the evaluation
`port`	`int`	`5001`	Port for the local Flask annotation server
`annotator_name`	`str \| None`	None	Name of the person performing annotations (if it is different from the operator name)
`wait_for_annotation`	`bool`	`True`	Block `finish_rollout()` until a human annotation is submitted
`open_browser`	`bool`	`True`	Automatically open the annotation UI in the default browser
`resume_session_name`	`str \| None`	`None`	Resume a previous session by name instead of starting a new one

To collect important robot and policy specific metadata, the EpisodeRecorder and WebRolloutAnnotator require a RobotProfile config dataclass. This dataclass is automatically generated from the robot profile YAML file with the load_robot_profile function.

`record_step(observation, action)`

Append one rollout timestep to the in-memory buffers. No data is written to disk; all buffered data is only persisted later by calling save().

Parameters

observation: dict[str, Any]

Top-level observation payload for a single timestep. Must contain:

Key	Type	Description
`"robot_state"`	`dict`	Proprioceptive state of the robot (see sub-keys below)
`"image_observation"`	`dict`	Per-camera frame data (see sub-keys below)

observation["robot_state"] — required keys are determined by robot_profile.robot_state_keys:

Key	Type	Description
`"cartesian_position"`	array-like	End-effector pose as `[x, y, z, <rotation>]` (single-arm) or doubled (bimanual). Rotation is converted to a quaternion format automatically based on the information in `robot_profile.orientation_representation`.
`"gripper_position"`	array-like	Current gripper opening width
`"joint_position"`	array-like	Per-joint angular positions

observation["image_observation"] — required keys are determined by robot_profile.camera_names. For each camera cam, the frame is looked up under any of the following candidate keys (first match wins):

Candidate key	Example
`cam`	`"wrist"`
`"image_{cam}"`	`"image_wrist"`
`"{cam}_image"`	`"wrist_image"`

Frames must be uint8 RGB arrays of shape (H, W, 3).

action: dict[str, np.ndarray]

Please ensure that the actions are provided as absolute, non-normalized, single-step vectors. Processing action chunks or normalization across different embodiments is difficult, so make sure you record every timestep of the robot execution together with the executed actions.

Dictionary of actions commanded at this timestep. All keys are optional; missing keys default to None and are stored as empty HDF5 datasets.

Key	Shape	Description
`"cartesian_position"`	`(3 + ROT)` or `(2 x [3 + ROT])`	Target end-effector pose. Same as with the robot state, the rotation component is automatically transformed into a quaternion representation
`"cartesian_velocity"`	`(6,)` or `(12,)`	End-effector Cartesian velocity
`"joint_position"`	`(N,)`	Target joint angles
`"joint_velocity"`	`(N,)`	Target joint velocities
`"base_position"`	`(3,)`	Mobile base position command
`"base_velocity"`	`(3,)`	Mobile base velocity command
`"gripper_position"`	`(1,)`	Continuous gripper position target
`"gripper_velocity"`	`(1,)`	Gripper velocity command
`"gripper_binary"`	`(1,)`	Binary open/close gripper command

Raises

ValueError — if observation is not a dict, required observation keys are missing, robot_state is missing profile-required keys, image_observation is missing expected camera keys, or all action values are None.
ValueError — if cartesian_position is present but not shaped (7,) or (14,) (after rotation conversion).

Possible Issues

If you are running your policy evaluation script inside a docker or singularity container, and you want to use the WebAnnotator tool, please make sure that you have forwarded the relevant ports to your main machine.