Oopsie ToolKit
You can download the Oopsie ToolKit from our github. The toolkit provides utilities for the main workflow:
Data Collection: Recording the data in the expected format
Annotation Tool: Annotating recorded data during data collection or after in bulk to capture failure information
Upload: Uploading it to the project submission server
In addition we provide utilities to reformat pre-gathered data to the format recognized by the annotation tool and the upload server. To simplify this, we document:
Data Format: The dataset format with all required and optional fields
Dataset Conversion: Existing tools for converting common formats into the Oopsie Data annotation and submission format.
Workflow
Our recommended workflow is to integrate our tools directly into your real-world policy evaluation pipeline. That way the data will be recorded in the correct format and the annotation tool will be automatically prompting you via a browser window to provide relevant information such as a failure description.
Integrating the recorder and annotator is described in detail in the data collection instructions.