Configure a pipeline

An HTTomo pipeline is an ordered sequence of loading and processing operations defined in YAML. See Pipelines for an introduction to pipelines and YAML templates for the structure of method templates.

Choose an editor

Use a text editor with YAML syntax highlighting and indentation support. Suitable editors include:

• VS Code for the Web Online YAML editing

• Visual Studio Code

• PyCharm

• Sublime Text

• Notepad++ for Windows

• Vim or Neovim for terminal-based editing

Build the pipeline

1. Start with an HTTomo loader.

2. Copy the required processing methods from Available methods into the same YAML file.

3. Arrange the methods in execution order. HTTomo runs them sequentially from top to bottom.

4. Edit each method’s parameters for the input data and processing task. Consult the relevant library documentation for parameter details.

Validate the pipeline

Note

Pipeline validation is integrated into the launcher at Diamond Light Source. See Running HTTomo at Diamond.

Check the pipeline regularly while editing it:

$ python -m httomo check pipeline.yaml

Supplying the input file also validates the dataset paths used by the loader:

$ python -m httomo check pipeline.yaml input.nxs

Always validate the completed pipeline before running it. See Command-line interface for the complete check command syntax.

What validation checks

The checker reports an error when:

• the YAML syntax or indentation is invalid;

• the first pipeline entry is not an HTTomo loader;

• a method or module path is unknown;

• a parameter name or value has the wrong type;

• a required parameter is missing;

• a side-output reference is invalid; or

• a referenced HDF5 dataset does not exist when an input file is supplied.

When a loader parameter is set to auto, supplying the input file also checks that HTTomo can find an NXtomo entry. See Automatic NXtomo discovery.

Common validation errors

Message or symptom	What to check
YAML cannot be parsed	Use spaces rather than tabs and align fields at the same nesting level.
Method is not valid	Copy its method and module_path from Available methods.
Parameter is unknown or has the wrong type	Compare the parameters mapping with the method template.
Dataset path is not valid	Check the loader paths in the input HDF5 file, or use NXtomo automatic discovery where supported.