How HTTomo runs a pipeline

HTTomo reads the YAML pipeline, validates its methods and parameters, and constructs an executable pipeline. Method wrappers adapt backend functions to HTTomo’s common data-processing interface.

Fig. 2 HTTomo execution from a YAML pipeline to output. The pipeline is divided into sections; within each section, data is distributed into chunks and processed block by block.

Creating sections

HTTomo groups compatible methods into sections. For each section, it calculates a safe block size from the available memory and the requirements of its methods.

Executing a section

Within each section, the dataset is divided among the available MPI processes. Each process receives one chunk and normally operates independently on its assigned data.

Each chunk is then divided into blocks using the block size calculated for the section. A block passes through every method in the section before the next block is processed.

Moving between sections

At a section boundary, HTTomo may synchronise the MPI processes, save intermediate data or redistribute the dataset. If the data pattern changes between projections and sinograms, HTTomo performs a re-slice.

Completing the pipeline

Processing continues section by section until every block has passed through the pipeline. HTTomo then writes the requested results and monitoring information.