Events Metadata Structure#

This document describes the dict-based events metadata system in NeuroConv, used by the discrete-events interfaces (TTL lines, strobes, epocs, markers). It is intended as a reference for developers contributing new events interfaces or modifying existing ones, and to document the design decisions behind the format.

A discrete event is a timestamp with an optional payload (a code, a category, a measurement), produced by an acquisition system such as a TDT store, a NIDQ line, or a marker channel. A single source file may contain more than one event type (a licking behavior, a frame start, a photodiode turning on), so an interface can expose multiple event types; and an experiment may record events with several acquisition systems at once, so a conversion can run multiple events interfaces together. The purpose of the events metadata dict is to let the user specify how the data and metadata is written under such various configurations.

For user-facing instructions on annotating events, see How to Annotate Discrete Events Metadata.

Throughout this document, event_type_source_id means whatever the source format (or its reader) uses to identify a type of event (licking, a camera frame onset, a reward). It may be a human-readable label, an internal numeric code, or a store or line name (e.g. a TDT store PtAB, a NIDQ line XD0); its form varies by format.

Design Principles#

The events metadata system follows the same core principles as the ophys and ecephys systems (see Ophys Metadata Structure), specialized to discrete events:

  1. Dictionary-Based Organization. Everything lives under metadata["Events"], dict-keyed at every level: each interface is namespaced by its metadata_key, each event type within it by its event_metadata_key (defaulting to the source’s event_type_source_id) under an event_types block. Each event type carries a required event_name and event_description (its name and description), and optionally a table_metadata_key naming a shared output table it joins. Keying every level lets several interfaces run in one conversion without clashing, and is consistent with the rest of the NeuroConv metadata.

    metadata["Events"] = {
    "behavioral_session": {                       # an interface, keyed by its metadata_key
        "event_types": {
            "licking": {                          # an event type, keyed by its event_metadata_key
                "event_name": "licking",           # names its own table (CamelCased -> "Licking")
                "event_description": "Lick detections.",
                "columns": {                      # value columns, keyed by field_source_id
                    "port": {"column_name": "lick", "description": "Lick detections."},
                },
            },
            "frame_start": {                      # a second event type in the same interface
                "event_name": "frame_start",
                "event_description": "Camera frame-start pulses.",
                "columns": {},                    # a bare marker: timestamps only
            },
        },
    },
}

    The keys are detailed in The metadata_key Parameter, The event_metadata_key, and The table_metadata_key.

  2. A solo event type names its own table; EventTables is an optional override for merges. By default each event type becomes its own table, named and described from its event_name (CamelCased) and event_description, with no EventTables entry needed. A global EventTables block (at the top level of metadata["Events"], not under any interface) appears only to name a table that several event types share, and when present it has the last word on that table’s name and description. See The table_metadata_key.

  3. Categorical values and the MeaningsTable. A categorical column’s value vocabulary , the labels shown for each raw value and the meanings that become a MeaningsTable , is set per column via column_categories. See The event_metadata_key.

Metadata Structure Overview#

The complete events metadata structure, with two interfaces (a TDT tank and a SpikeGLX NIDQ stream) and a mix of solo and shared-table layouts. The solo types (PtAB, XD0) name their own tables from event_name, so they need no EventTables entry; only the shared behavior table (pooled across both interfaces) is declared in EventTables. Store names and values are real: PtAB port codes and the constant-value PC0_ marker are from TDT demo tanks.

metadata["Events"] = {

    "EventTables": {                              # GLOBAL: only tables SHARED by several event types
        "behavior": {
            "table_name": "BehavioralEvents",     # the NWB object name (CamelCase)
            "description": "Rewards and trial starts sharing one table across interfaces.",
        },
    },

    "tdt_session": {                              # interface 1 (a TDT tank), keyed by its metadata_key
        "event_types": {
            "PtAB": {                             # one entry per event_type_source_id (a TDT epoc store code)
                "event_name": "port_entries",      # solo: names its own table -> "PortEntries"
                "event_description": "Nose-poke port entries, coded by port.",
                "columns": {
                    "strobe": {                   # one value column, keyed by field_source_id
                        "column_name": "port_entry",
                        "description": "Nose-poke port entry, coded by port.",
                        "column_categories": {
                            # keys are the raw strobe values the hardware latched (TDT port codes)
                            "labels":   {64959: "left", 65023: "center", 65535: "right"},
                            "meanings": {64959: "left port entry", 65023: "center port entry", 65535: "right port entry"},
                        },
                    },
                },
            },
            "PC0_": {                             # a second store: a bare marker (constant strobe value 1.0)
                "event_name": "reward",            # its label in the shared table's event_type column
                "event_description": "Reward delivery.",
                "table_metadata_key": "behavior",       # pooled into the shared table
                "columns": {},                    # timestamps only
            },
        },
    },

    "nidq_session": {                            # interface 2 (a SpikeGLX NIDQ stream)
        "event_types": {
            "XD1": {                              # a digital line read as a bare marker
                "event_name": "trial_start",
                "event_description": "Trial-start pulse.",
                "table_metadata_key": "behavior",       # pooled into the same shared table
                "columns": {},
            },
            "XD0": {
                "event_name": "camera_frame",      # solo: names its own table -> "CameraFrame"
                "event_description": "Camera exposure pulses.",
                "columns": {},
            },
        },
    },
}

The metadata_key Parameter#

Events interfaces accept a metadata_key string parameter that selects the interface’s event_types block. It is keyword-only and defaults to None.

class SomeEventsInterface(BaseDataInterface):
    def __init__(self, *, metadata_key: Optional[str] = None, **source_data):
        self.metadata_key = metadata_key
        ...

When None, the interface derives a unique, source-derived snake_case key from the source (the tank or block name), so even two instances of the same interface in one converter get distinct keys with zero configuration. This is more robust than a hardcoded default (a fixed "SpikeGLXNIDQ" would collide the moment two NIDQ interfaces share a converter). An explicit value lets the caller pick a stable, readable name, or deliberately reuse a key.

Its role is disambiguation across interfaces: every interface’s columns live under its own metadata_key, so two interfaces can expose the same event_type_source_id (two tanks both with a store PtAB, two boards both with a line XD0) and never collide, because the full address includes the namespace: metadata["Events"][metadata_key]["event_types"][...].

An interface’s event types are nested inside an event_types block (rather than sitting directly under the metadata_key), which reserves the per-interface block so another kind of per-interface data could be added alongside the event types in the future without reshaping the dict.

The event_metadata_key#

Inside an interface’s event_types block, each event type is keyed by an event_metadata_key, which defaults to its event_type_source_id. It is what you use to reach and edit one event type’s metadata, and its entry holds the type’s name and description, its (optional) table routing, and its value columns:

metadata["Events"]["behavioral_session"]["event_types"]["licking"] = {
    "event_name": "licking",                # the type's name (see the dual role below)
    "event_description": "Lick detections, left or right port.",
    "columns": {                            # value columns, keyed by field_source_id
        "port": {
            "column_name": "lick",          # the column header in the output table
            "description": "Lick detections, left or right port.",
            "column_categories": {          # present only for a categorical column
                "labels":   {1: "left", 2: "right"},
                "meanings": {1: "left lick", 2: "right lick"},
            },
        },
    },
}

An entry holds:

  • event_name and event_description , the type’s name and description (both required, both defaulting to the event_type_source_id). They have a dual role by layout: when the type has its own table (the default), event_name becomes that table’s NWB object name (CamelCased) and event_description its table description; when the type is pooled into a shared table (a merge), event_name becomes the type’s label in that table’s event_type discriminator column and event_description the meaning of that label in the column’s MeaningsTable.

  • table_metadata_key (optional) , which output table the event type is written into; defaults to the event_type_source_id so the type gets its own table. Set it only to merge (see The table_metadata_key).

  • columns , the value columns of the event type, keyed by field_source_id. An absent or empty columns map is a timestamps-only event; a populated one carries the event’s payload.

Each column entry holds:

  • column_name , the column header in the output table (default: the source’s field label if it carries one, else the field_source_id).

  • description , a free-text description of the column, written as its VectorData description in the output table (default: a generic description naming the source).

  • column_categories , the column’s value vocabulary (see below); present only for a categorical column.

The column’s value vocabulary is column_categories:

  • Presence means categorical. column_categories = {labels, meanings} declares the column’s values. The keys of both maps are the raw values the source emits (a TDT strobe value, a decoded code, a line state, e.g. the port codes 64959/65023/65535 above). labels map each raw value to a display label, and meanings map each raw value to a description. Both are optional.

  • Absence means not categorical. Omit column_categories and a numeric column is continuous (raw values written as a plain numeric column) and a string column is free-text.

The table_metadata_key#

table_metadata_key identifies an output table. Unlike the other keys (which nest metadata under them), this one is a reference: an event type names, in its table_metadata_key field, the table it is written into. It is optional and defaults to the event_type_source_id, so by default each event type routes to its own table.

A solo event type (the only one routing to its table) needs no EventTables entry at all: the writer names and describes its table from the type’s event_name (CamelCased) and event_description. An EventTables entry is only needed to name a table that several types share. Shared tables are declared in the global EventTables block, at the top level of metadata["Events"] (separate from any interface’s block):

metadata["Events"]["EventTables"] = {
    "behavior": {"table_name": "BehavioralEvents", "description": "Rewards and trial starts."},
}

EventTables is the only reserved key under metadata["Events"] (every other top-level key is an interface metadata_key). Each entry’s table_name is the NWB object name (CamelCase) of the output EventsTable, and its key is the table_metadata_key that event types point at. When an EventTables entry is present it has the last word on the table’s name and description, overriding what a solo type would otherwise derive.

By default each event type gets its own table with no edits. To put several event types in one table instead, share by linking: point each one’s table_metadata_key at one shared key, and declare that key in EventTables to name the pooled table; see How to Write Multiple Event Types to a Single EventsTable for a worked example. The same edit shares a table across interfaces, since the table block is global. The writer pools the shared types’ events into one time-sorted table and adds an event_type discriminator column holding each row’s event_name (with a MeaningsTable mapping those names to their event_description), so a bare marker (which adds no value column) keeps its identity; each event type contributes only the value columns it carries, filled on its own rows and empty on the others.

Handling events with multiple values per event#

In most acquisition systems an event is either a pure timestamp (a bare marker, e.g. a TTL pulse or a reward) or a timestamp with a single value (a code or label, e.g. a port code). Both write straightforwardly. Every event table has a built-in timestamp column, so a pure timestamp needs no value column at all, it is just an EventsTable with its timestamp column and nothing else (empty columns). A single value adds one value column alongside timestamp (one columns entry).

Some events, though, carry more than one value per occurrence (a structured payload, e.g. a Spike2 TextMark tagged with both a numeric marker code and a text string). Here each field is its own columns entry, keyed by its field_source_id (the field name when the source provides one, otherwise a numeric index), all under the one event type:

# one event type, two fields -> two columns on one table
"TextMark": {
    "table_metadata_key": "textmark",
    "columns": {
        "marker": {"column_name": "marker"},
        "text":   {"column_name": "text"},
    },
}

Because the fields come from the same event, they share its timestamps and the columns sit on the same rows of one table. Each field is named and described independently (its own column_name / description / column_categories).

When Objects Are Created#

The NWB table objects are created at add_to_nwbfile time from the event_types entries and any EventTables overrides. Unlike the ophys/ecephys pipelines, get_metadata() does not seed an EventTables entry per event type: a solo table is named and described directly from its type’s event_name/event_description, so EventTables stays empty until the user declares a shared table. The rules:

  1. A solo type creates its own table. An event type whose table_metadata_key no other type shares is written to a table named to_camel_case(event_name) with event_description as its description, no EventTables entry required. A zero-config run therefore produces one table per event type with no EventTables block at all.

  2. A shared table is declared and created once. When several event types point at one table_metadata_key, that key needs an EventTables entry to name the pooled table (its table_name/description have the last word). The table is created by whichever interface writes first, and later event types , including those from other interfaces , append to it.

  3. An EventTables override wins. A table_metadata_key that does have an EventTables entry takes that entry’s table_name/description, whether the table is solo or shared.

Table keys are independent of any interface’s metadata_key: a table_metadata_key such as "behavior" need not match any interface’s key. No interface owns a table; it is created at write time by whichever event type first references it, and a second interface routing into the same table appends to it (its new columns backfilled on the existing rows, the existing columns filled on its new rows), after which the table is re-sorted so it stays chronological.