Pipeline features

This section guides you from the basics of configuring a pipeline to advanced features such as parameter sweeps, saving results and managing memory. It also explains side outputs and padding.

• Configure a pipeline
  • Choose an editor
  • Build the pipeline
  • Validate the pipeline
• Optimise pipeline performance
  • Group methods by data pattern
  • Method metadata
  • Group GPU methods
  • Minimise writing to disk
• Memory and performance
  • Estimate CPU memory before a run
  • Choose a process count
  • Use a runtime memory ceiling
  • Reduce resource use
• Side outputs
  • Define and reference a side output
• Centre of Rotation
  • CoR in HTTomo
  • Auto-centering
  • Manual Centering
• Parameter Sweeping
  • How would this be useful when processing data?
  • What does the output look like?
  • How are parameter sweeps defined in the pipeline YAML file?
  • How big should the input data be?
  • What structure does the output data of a parameter sweep have?
• Save intermediate datasets
• Padding
  • How this can be useful?
  • How to use