httomo.runner.dataset_store_interfaces.DataSetSource

httomo.runner.dataset_store_interfaces.DataSetSource#

class httomo.runner.dataset_store_interfaces.DataSetSource(*args, **kwargs)[source]#

MPI-aware source for full datasets, where each process handles a chunk, and the data can be read in blocks, sliced in the given slicing dimension

Methods

`__init__`(args, *kwargs)
`finalize`()	Method intended to be called after reading all blocks is done, to give implementations a chance to close files, free memory, etc.
`read_block`(start, length)	Reads a block from the dataset, starting at start of length length, in the current slicing dimension.

Attributes

`aux_data`	Auxiliary data
`chunk_shape`	Returns the shape of a chunk, i.e. the data processed in the current MPI process (whether it fits in memory or not).
`dtype`
`global_index`	Returns the start index of the chunk within the global data array
`global_shape`	Global data shape across all processes that we eventually have to read.
`raw_shape`	Returns raw data shape
`slicing_dim`	Slicing dimension - 0, 1, or 2

property chunk_shape: Tuple[int, int, int]#: Returns the shape of a chunk, i.e. the data processed in the current MPI process (whether it fits in memory or not)

finalize()[source]#: Method intended to be called after reading all blocks is done, to give implementations a chance to close files, free memory, etc.

property global_index: Tuple[int, int, int]#: Returns the start index of the chunk within the global data array

property global_shape: Tuple[int, int, int]#: Global data shape across all processes that we eventually have to read.

read_block(start: int, length: int) → DataSetBlock[source]#: Reads a block from the dataset, starting at start of length length, in the current slicing dimension. Note that start is chunk-based, i.e. mean different things in different processes.