Introduction

A turn-based data structure is essential for organizing and managing conversational data in a structured manner. This approach segments interactions into distinct turns, making it easier to annotate, analyze, and process multimodal data. Each turn encompasses all relevant modalities—text, audio, image, and video—along with associated goals, behaviors, and actions.

Importance of Turn-Based Data Structure

Implementing a turn-based data structure offers several advantages:

1. Clarity: Provides a clear and organized framework for capturing interactions.
2. Synchronization: Ensures that all modalities are synchronized within each turn.
3. Annotation Efficiency: Facilitates efficient and consistent annotation of data.
4. Model Training: Enhances the quality of training data for AI models, leading to better performance.

Components of Turn-Based Data Structure

Text

Text data includes the dialogue or conversation content, tagged with the speaker and timestamp. This is the primary modality for capturing the verbal component of interactions.

Example

"text": {
    "timestamp": "00:00:01",
    "content": "Hello, how are you?"
}

Audio

Audio data comprises the corresponding audio recordings of the conversation, ensuring that the spoken words match the text data.

Example

"audio": {
    "timestamp": "00:00:01",
    "file": "audio_segment_1.wav"
}

Image

Image data includes any visual content associated with the turn, such as key frames or snapshots from a video. This helps in capturing visual cues like facial expressions and gestures.

Example