NWB Helpers#

Collection of Pydantic models and helper functions for configuring dataset IO parameters for different backends.

class BackendConfiguration(/, **data: 'Any') → 'None'[source]#

Bases: BaseModel

A model for matching collections of DatasetConfigurations to a specific backend.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Apply a codec pipeline to all datasets in this backend configuration.

This method modifies the backend configuration in-place, applying the specified codecs to ALL datasets, regardless of what each one currently holds. Naming the pipeline replaces the pipeline, so a filter a dataset already had is only kept if it is named here too.

Parameters:

compressors (list of str) – The ordered codecs to apply to every dataset, for example ["gzip"] or ["shuffle", "Blosc"]. A filter such as "shuffle" composes with a compression method rather than replacing one, so both live in this list.
compressor_options (list of dict, optional) – The optional parameters for each entry of compressors, positionally matched. Use None for an entry that takes no parameters.

Raises:

ValueError – If a named compression method is not available for this backend type.

Examples

>>> backend_config = get_default_backend_configuration(nwbfile, backend="hdf5")
>>> backend_config.apply_global_compression(["Blosc"], [{"cname": "zstd", "clevel": 5}])
>>> backend_config.apply_global_compression(["shuffle", "gzip"])

Deprecated since version 0.10.2: The compression_method and compression_options arguments, and passing a single compression method and a single options dictionary positionally, are deprecated and will be removed in v0.12.0. Pass lists to compressors and compressor_options instead.

build_remapped_backend(locations_to_remap: dict[str, DatasetIOConfiguration]) → Self[source]#

Build a remapped backend configuration by updating mismatched object IDs.

This function takes a dictionary of new DatasetIOConfiguration objects (as returned by find_locations_requiring_remapping) and updates a copy of the current configuration with these new configurations.

Parameters:: locations_to_remap (dict) – A dictionary mapping locations in the NWBFile to their corresponding new DatasetIOConfiguration objects with updated IDs.
Returns:: A new instance of the backend configuration class with updated object IDs for the specified locations.
Return type:: Self

find_locations_requiring_remapping(nwbfile: NWBFile) → dict[str, DatasetIOConfiguration][source]#

Find locations of objects with mismatched IDs in the file.

This function identifies neurodata objects in the nwbfile that have matching locations with the current configuration but different object IDs. It returns a dictionary of remapped DatasetIOConfiguration objects for these mismatched locations.

Parameters:: nwbfile (pynwb.NWBFile) – The NWBFile object to check for mismatched object IDs.
Returns:: A dictionary where: * Keys: Locations in the NWB of objects with mismatched IDs. * Values: New DatasetIOConfiguration objects corresponding to the updated object IDs.
Return type:: dict[str, DatasetIOConfiguration]

Notes

This function only checks for objects with the same location but different IDs.
It does not identify objects missing from the current configuration.
The returned DatasetIOConfiguration objects are copies of the original configurations

with updated object_id fields.

classmethod from_nwbfile(nwbfile: NWBFile) → Self[source]#: Create a backend configuration from an NWBFile with default chunking and compression settings.

Deprecated since version 0.8.4: The from_nwbfile method is deprecated and will be removed on or after June 2026. Use from_nwbfile_with_defaults or from_nwbfile_with_existing instead.

classmethod from_nwbfile_with_defaults(nwbfile: NWBFile) → Self[source]#

Create a backend configuration from an NWBFile with default chunking and compression settings.

Parameters:: nwbfile (pynwb.NWBFile) – The NWBFile object to extract the backend configuration from.
Returns:: The backend configuration with default chunking and compression settings for each neurodata object in the NWBFile.
Return type:: Self

classmethod from_nwbfile_with_existing(nwbfile: NWBFile) → Self[source]#

Create a backend configuration from an NWBFile using existing dataset settings.

This method extracts existing chunking and compression settings from an NWBFile that has already been written to disk.

Parameters:: nwbfile (pynwb.NWBFile) – The NWBFile object to extract the backend configuration from.
Returns:: The backend configuration with existing chunking and compression settings for each neurodata object in the NWBFile.
Return type:: Self

model_config: ClassVar[ConfigDict] = {'validate_assignment': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod model_json_schema(**kwargs) → dict[str, Any][source]#

Generates a JSON schema for a model class.

Args:

by_alias: Whether to use attribute aliases or not. ref_template: The reference template. union_format: The format to use when combining schemas from unions together. Can be one of:

‘any_of’: Use the [anyOf](https://json-schema.org/understanding-json-schema/reference/combining#anyOf)

keyword to combine schemas (the default). - ‘primitive_type_array’: Use the [type](https://json-schema.org/understanding-json-schema/reference/type) keyword as an array of strings, containing each type of the combination. If any of the schemas is not a primitive type (string, boolean, null, integer or number) or contains constraints/metadata, falls back to any_of.

schema_generator: To override the logic used to generate the JSON schema, as a subclass of: GenerateJsonSchema with your desired modifications

mode: The mode in which to generate the schema.

Returns:

The JSON schema for the given model class.

classmethod schema(**kwargs) → dict[str, Any][source]#

classmethod schema_json(**kwargs) → dict[str, Any][source]#

backend: ClassVar[Literal['hdf5', 'zarr']]#

pretty_backend_name: ClassVar[Literal['HDF5', 'Zarr']]#

data_io_class: ClassVar[type[DataIO]]#

dataset_configurations: dict[str, DatasetIOConfiguration]#

class HDF5BackendConfiguration(/, **data: 'Any') → 'None'[source]#

Bases: BackendConfiguration

A model for matching collections of DatasetConfigurations specific to the HDF5 backend.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

backend: ClassVar[Literal['hdf5']] = 'hdf5'#

data_io_class#: alias of H5DataIO

model_config: ClassVar[ConfigDict] = {'validate_assignment': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

pretty_backend_name: ClassVar[Literal['HDF5']] = 'HDF5'#

dataset_configurations: dict[str, HDF5DatasetIOConfiguration]#

class ZarrBackendConfiguration(/, **data: 'Any') → 'None'[source]#

Bases: BackendConfiguration

A model for matching collections of DatasetConfigurations specific to the Zarr backend.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

backend: ClassVar[Literal['zarr']] = 'zarr'#

data_io_class#: alias of ZarrDataIO

model_config: ClassVar[ConfigDict] = {'validate_assignment': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

pretty_backend_name: ClassVar[Literal['Zarr']] = 'Zarr'#

dataset_configurations: dict[str, ZarrDatasetIOConfiguration]#

number_of_jobs: int#

class DatasetIOConfiguration(/, **data: 'Any') → 'None'[source]#

Bases: BaseModel, ABC

A data model for configuring options about an object that will become a HDF5 or Zarr Dataset in the file.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

property compression_method#: The compression method applied to this dataset.

Deprecated since version 0.10.2: compression_method is deprecated and will be removed in v0.12.0. Use compressors instead.

property compression_options: dict[str, Any] | None#: The parameters of the compression method applied to this dataset.

Deprecated since version 0.10.2: compression_options is deprecated and will be removed in v0.12.0. Use compressor_options instead.

property disk_space_usage_per_chunk_in_bytes: int | None#: The uncompressed size of a single chunk, or None if the dataset is not chunked.

classmethod from_neurodata_object_with_defaults(neurodata_object: Container, dataset_name: Literal['data', 'timestamps'], builder: BaseBuilder | None = None) → Self[source]#

Construct an instance of a DatasetIOConfiguration with default settings for a dataset in a neurodata object in an NWBFile.

Parameters:

neurodata_object (hdmf.Container) – The neurodata object containing the field that will become a dataset when written to disk.
dataset_name (“data” or “timestamps”) – The name of the field that will become a dataset when written to disk. Some neurodata objects can have multiple such fields, such as pynwb.TimeSeries which can have both data and timestamps, each of which can be configured separately.
builder (hdmf.build.builders.BaseBuilder, optional) – The builder object that would be used to construct the NWBFile object. If None, the dataset is assumed to NOT have a compound dtype.

abstractmethod classmethod from_neurodata_object_with_existing(neurodata_object: Container, dataset_name: Literal['data', 'timestamps']) → Self[source]#

Construct an instance of a DatasetIOConfiguration from existing dataset settings.

This method extracts compression and chunking configurations from an already-written dataset. The neurodata object must have been read from an existing NWB file.

Parameters:

neurodata_object (hdmf.Container) – The neurodata object containing the field that has been read from disk.
dataset_name (“data” or “timestamps”) – The name of the field that corresponds to the dataset on disk.

Returns:

A DatasetIOConfiguration instance with settings matching the existing dataset.

Return type:

Self

property full_size_in_bytes: int#: The size the entire source array occupies in memory.

abstractmethod get_data_io_kwargs() → dict[str, Any][source]#

Fetch the properly structured dictionary of input arguments.

Should be passed directly as dynamic keyword arguments (**kwargs) into a H5DataIO or ZarrDataIO.

static get_dataset(neurodata_object: Container, dataset_name: Literal['data', 'timestamps']) → Dataset | Array[source]#

static get_kwargs_from_neurodata_object(neurodata_object: Container, dataset_name: Literal['data', 'timestamps']) → dict[source]#

property maximum_ram_usage_per_iteration_in_bytes: int#: The RAM a single iteration of the write takes, which is one buffer of the source array.

model_config: ClassVar[ConfigDict] = {'validate_assignment': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod model_json_schema(**kwargs) → dict[str, Any][source]#

Generates a JSON schema for a model class.

Args:

by_alias: Whether to use attribute aliases or not. ref_template: The reference template. union_format: The format to use when combining schemas from unions together. Can be one of:

‘any_of’: Use the [anyOf](https://json-schema.org/understanding-json-schema/reference/combining#anyOf)

keyword to combine schemas (the default). - ‘primitive_type_array’: Use the [type](https://json-schema.org/understanding-json-schema/reference/type) keyword as an array of strings, containing each type of the combination. If any of the schemas is not a primitive type (string, boolean, null, integer or number) or contains constraints/metadata, falls back to any_of.

schema_generator: To override the logic used to generate the JSON schema, as a subclass of: GenerateJsonSchema with your desired modifications

mode: The mode in which to generate the schema.

Returns:

The JSON schema for the given model class.

model_post_init(context: Any, /) → None#

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Args:: self: The BaseModel instance. context: The context.

classmethod schema(**kwargs) → dict[str, Any][source]#

classmethod schema_json(**kwargs) → dict[str, Any][source]#

classmethod translate_deprecated_compression_fields(values: dict[str, Any]) → dict[str, Any][source]#: Accept the deprecated compression_method and compression_options spellings for one release cycle.

classmethod validate_all_shapes(values: dict[str, Any]) → dict[str, Any][source]#

object_id: str#

location_in_file: str#

dataset_name: Literal['data', 'timestamps']#

dtype: Annotated[dtype, InstanceOf()]#

full_shape: tuple[int, ...]#

chunk_shape: tuple[Annotated[int, Gt(gt=0)], ...] | None#

buffer_shape: tuple[int, ...] | None#

compressors: list[str | Annotated[FilterRefBase, InstanceOf()] | Annotated[Codec, InstanceOf()]] | None#

compressor_options: list[dict[str, Any] | None] | None#

class HDF5DatasetIOConfiguration(/, **data: 'Any') → 'None'#

Bases: DatasetIOConfiguration

A data model for configuring options about an object that will become a HDF5 Dataset in the file.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

classmethod from_neurodata_object_with_existing(neurodata_object: Container, dataset_name: Literal['data', 'timestamps']) → Self#

Construct an HDF5DatasetIOConfiguration from existing dataset settings.

Parameters:

neurodata_object (hdmf.Container) – The neurodata object containing the field that has been read from disk.
dataset_name (“data” or “timestamps”) – The name of the field that corresponds to the dataset on disk.

Returns:

An HDF5DatasetIOConfiguration instance with settings matching the existing dataset.

Return type:

Self

get_data_io_kwargs() → dict[str, Any]#

Fetch the properly structured dictionary of input arguments.

Should be passed directly as dynamic keyword arguments (**kwargs) into a H5DataIO or ZarrDataIO.

model_config: ClassVar[ConfigDict] = {'validate_assignment': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context: Any, /) → None#

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Args:: self: The BaseModel instance. context: The context.

validate_compressor_order() → Self#: HDF5 does not let the caller choose where each filter sits in the pipeline, so validate against its order.

compressors: list[Literal['shuffle', 'fletcher32', 'lzf', 'gzip', 'szip'] | Annotated[FilterRefBase, InstanceOf()]] | None#

compressor_options: list[dict[str, Any] | None] | None#

class ZarrDatasetIOConfiguration(/, **data: 'Any') → 'None'#

Bases: DatasetIOConfiguration

A data model for configuring options about an object that will become a Zarr Dataset in the file.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

property filter_methods#: The codecs applied to this dataset’s values.

Deprecated since version 0.10.2: filter_methods is deprecated and will be removed in v0.12.0. Use filters instead.

classmethod from_neurodata_object_with_existing(neurodata_object: Container, dataset_name: Literal['data', 'timestamps']) → Self#

Construct a ZarrDatasetIOConfiguration from existing dataset settings.

Parameters:

neurodata_object (hdmf.Container) – The neurodata object containing the field that has been read from disk.
dataset_name (“data” or “timestamps”) – The name of the field that corresponds to the dataset on disk.

Returns:

A ZarrDatasetIOConfiguration instance with settings matching the existing dataset.

Return type:

Self

get_data_io_kwargs() → dict[str, Any]#

Fetch the properly structured dictionary of input arguments.

Should be passed directly as dynamic keyword arguments (**kwargs) into a H5DataIO or ZarrDataIO.

model_config: ClassVar[ConfigDict] = {'validate_assignment': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context: Any, /) → None#

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Args:: self: The BaseModel instance. context: The context.

classmethod translate_deprecated_filter_methods(values: dict[str, Any]) → dict[str, Any]#: Accept the deprecated filter_methods spelling for one release cycle.

validate_compressors_and_options_length_match() → Self#

classmethod validate_filters_and_options_length_match(values: dict[str, Any])#

warn_on_shuffle_in_filters() → Self#: Accept the deprecated spelling of shuffle as a filter method for one release cycle.

compressors: list[Literal['shuffle', 'fletcher32', 'categorize', 'lz4', 'lzma', 'zstd', 'blosc', 'zlib', 'gzip', 'jenkins_lookup3', 'delta', 'bz2', 'packbits'] | Annotated[Codec, InstanceOf()]] | None#

compressor_options: list[dict[str, Any] | None] | None#

filters: list[Literal['shuffle', 'fletcher32', 'categorize', 'lz4', 'lzma', 'zstd', 'blosc', 'zlib', 'gzip', 'jenkins_lookup3', 'delta', 'bz2', 'packbits'] | Annotated[Codec, InstanceOf()]] | None#

filter_options: list[dict[str, Any]] | None#

get_default_backend_configuration(nwbfile: NWBFile, backend: Literal['hdf5', 'zarr']) → HDF5BackendConfiguration | ZarrBackendConfiguration[source]#: Fill a default backend configuration to serve as a starting point for further customization.

get_default_dataset_io_configurations(nwbfile: NWBFile, backend: None | Literal['hdf5', 'zarr'] = None) → Generator[DatasetIOConfiguration, None, None][source]#

Generate DatasetIOConfiguration objects for wrapping NWB file objects with a specific backend.

This method automatically detects all objects in an NWB file that can be wrapped in a hdmf.DataIO. If the NWB file is in append mode, it supports auto-detection of the backend. Otherwise, it requires a backend specification.

Parameters:

nwbfile (pynwb.NWBFile) – An in-memory NWBFile object, either generated from the base class or read from an existing file of any backend.
backend (“hdf5” or “zarr”) – Which backend format type you would like to use in configuring each dataset’s compression methods and options.

Yields:

DatasetIOConfiguration – A summary of each detected object that can be wrapped in a hdmf.DataIO.

get_existing_backend_configuration(nwbfile: NWBFile) → HDF5BackendConfiguration | ZarrBackendConfiguration[source]#

Fill an existing backend configuration to serve as a starting point for further customization.

Parameters:: nwbfile (NWBFile) – The NWBFile object to extract the backend configuration from. The nwbfile must have been read from an io object to work properly.
Returns:: The backend configuration extracted from the nwbfile.
Return type:: HDF5BackendConfiguration | ZarrBackendConfiguration

get_existing_dataset_io_configurations(nwbfile: NWBFile) → Generator[DatasetIOConfiguration, None, None][source]#

Generate DatasetIOConfiguration objects for each neurodata object in an nwbfile.

Parameters:

nwbfile (pynwb.NWBFile) – An NWBFile object that has been read from an existing file with an existing backend configuration.
backend (“hdf5” or “zarr”) – Which backend format type you would like to use in configuring each dataset’s compression methods and options.

Yields:

DatasetIOConfiguration – A configuration object for each dataset in the NWB file.

configure_backend(nwbfile: NWBFile, backend_configuration: HDF5BackendConfiguration | ZarrBackendConfiguration) → None[source]#

Configure all datasets specified in the backend_configuration with their appropriate DataIO and options.

Parameters:

nwbfile (pynwb.NWBFile) – The in-memory pynwb.NWBFile object to configure.
backend_configuration (HDF5BackendConfiguration or ZarrBackendConfiguration) – The configuration model to use when configuring the datasets for this backend.

add_device_from_metadata(nwbfile: NWBFile, modality: str = 'Ecephys', metadata: dict | None = None)[source]#

Add device information from metadata to NWBFile object.

Will always ensure nwbfile has at least one device, but multiple devices within the metadata list will also be created.

Parameters:

nwbfile (NWBFile) – NWBFile to which the new device information is to be added
modality (str) – Type of data recorded by device. Options: - Ecephys (default) - Icephys - Ophys - Behavior
metadata (dict) – Metadata info for constructing the NWBFile (optional). Should be of the format:
```
metadata[modality]['Device'] = [
    {
        'name': my_name,
        'description': my_description
    },
    ...
]
```
Missing keys in an element of metadata['Ecephys']['Device'] will be auto-populated with defaults.

add_subject_to_nwbfile(nwbfile: NWBFile, metadata: dict | None = None) → None[source]#

Add the subject described by metadata["Subject"] to an NWBFile.

Parameters:

nwbfile (NWBFile) – The NWBFile to add the subject to.
metadata (dict, optional) – Metadata dictionary. The subject is read from metadata["Subject"], whose keys are the arguments of Subject:
```
metadata["Subject"] = dict(subject_id="M1", species="Mus musculus", sex="M")
```
A date_of_birth stated as an ISO 8601 string is converted. Metadata that names no subject is not an error: a source that did not record one leaves the file without one.

Raises:

ValueError – If the NWBFile already holds a subject. An NWBFile describes one subject and pynwb refuses to replace it, so a second one is a conflict for the caller to resolve rather than something to overwrite silently.

configure_and_write_nwbfile(nwbfile: NWBFile, nwbfile_path: Annotated[Path, PathType(path_type=file)] | None = None, backend: Literal['hdf5', 'zarr'] | None = None, backend_configuration: BackendConfiguration | None = None) → None[source]#

Write an NWB file using a specific backend or backend configuration.

A backend or a backend_configuration must be provided. To use the default backend configuration for the specified backend, provide only backend. To use a custom backend configuration, provide backend_configuration. If both are provided, backend must match backend_configuration.backend.

Parameters:

nwbfile (NWBFile)
nwbfile_path (FilePath | None, optional)
backend ({“hdf5”, “zarr”}, optional) – The type of backend used to create the file. This option uses the default backend_configuration for the specified backend. If no backend is specified, the backend_configuration is used.
backend_configuration (BackendConfiguration, optional) – Specifies the backend type and the chunking and compression parameters of each dataset. If no backend_configuration is specified, the default configuration for the specified backend is used.

get_default_nwbfile_metadata() → DeepDict[source]#

Return structure with defaulted metadata values required for a NWBFile.

These standard defaults are:

metadata["NWBFile"]["session_description"] = "no description"
metadata["NWBFile"]["identifier"] = str(uuid.uuid4())

Proper conversions should override these fields prior to calling NWBConverter.run_conversion()

Returns:: A dictionary containing default metadata values for an NWBFile, including session description, identifier, and NeuroConv version information.
Return type:: DeepDict

get_module(nwbfile: NWBFile, name: str, description: str = None)[source]#

Check if processing module exists. If not, create it. Then return module.

Parameters:

nwbfile (NWBFile) – The NWB file to check or add the module to.
name (str) – The name of the processing module.
description (str, optional) – Description of the module. Only used if creating a new module.

Returns:

The existing or newly created processing module.

Return type:

ProcessingModule

make_nwbfile_from_metadata(metadata: dict) → NWBFile[source]#

Make NWBFile from available metadata.

Parameters:: metadata (dict) – Dictionary containing metadata for creating the NWBFile. Must contain an ‘NWBFile’ key with required fields.
Returns:: A newly created NWBFile object initialized with the provided metadata.
Return type:: NWBFile

make_or_load_nwbfile(nwbfile_path: Annotated[Path, PathType(path_type=file)] | None = None, nwbfile: NWBFile | None = None, metadata: dict | None = None, overwrite: bool = False, backend: Literal['hdf5', 'zarr'] = 'hdf5', verbose: bool = False)[source]#

Context for automatically handling decision of write vs. append for writing an NWBFile.

Parameters:

nwbfile_path (FilePath) – Path for where to write or load (if overwrite=False) the NWBFile. If specified, the context will always write to this location.
nwbfile (NWBFile, optional) – An in-memory NWBFile object to write to the location.
metadata (dict, optional) – Metadata dictionary with information used to create the NWBFile when one does not exist or overwrite=True.
overwrite (bool, default: False) – Whether to overwrite the NWBFile if one exists at the nwbfile_path. The default is False (append mode).
backend (“hdf5” or “zarr”, default: “hdf5”) – The type of backend used to create the file.
verbose (bool, default: True) – If ‘nwbfile_path’ is specified, informs user after a successful write operation.

repack_nwbfile(*, nwbfile_path: Path, export_nwbfile_path: Path, export_backend: Literal['hdf5', 'zarr', None] = None)[source]#

Repack an NWBFile with a new backend configuration.

Parameters:

nwbfile_path (Path) – Path to the NWB file to be repacked.
export_nwbfile_path (Path) – Path to export the repacked NWB file.
export_backend ({“hdf5”, “zarr”, None}, default: None) – The type of backend used to write the repacked file. If None, the same backend as the input file is used.

NWB Helpers#

This Page