Full YAML pipelines

Full YAML pipelines#

This is a collection of ready to be used full pipelines or process lists for HTTomo. See more on What is a process list? and how to Configure efficient pipelines.

HTTomo mainly targets GPU computations, therefore the use of GPU Pipeline templates is preferable. However, when the GPU device is not available or a GPU method is not implemented, the use of CPU Pipeline templates is possible.

Note

The combination of both GPU and CPU methods is possible. If one expects to achieve the faster performance, please use the GPU methods provided, where possible.

GPU Pipeline templates#

The GPU-pipelines consist of methods from httomolibgpu (GPU) and httomolib (CPU) backend Supported libraries. Those libraries are supported directly by the HTTomo development team.

GPU pipeline with auto-centering and the FBP3d_tomobar reconstruction method.

# This pipeline is supported by HTTomo ver. 2.5
# --- Standard tomography loader for NeXus files. ---
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: auto
    image_key_path: auto
    rotation_angles: auto
    preview:
      detector_x:   # horizontal data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
      detector_y:   # vertical data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
    darks: null
    flats: null
# --- Center of Rotation auto-finding. Required for reconstruction bellow. ---
- method: find_center_vo
  module_path: httomolibgpu.recon.rotation
  parameters:
    ind: null  # A vertical slice (sinogram) index to calculate CoR, `mid` can be used for middle
    average_radius: 0 # Average several sinograms to improve SNR, one can try 3-5 range
    cor_initialisation_value: null  # Use if an approximate CoR is known
    smin: -50
    smax: 50
    srad: 6.0
    step: 0.5
    ratio: 0.5
    drop: 20
  id: centering
  side_outputs:
    cor: centre_of_rotation  # An estimated CoR value provided as a side output
# --- Removing unresponsive pixels in the data, aka zingers. Use if sharp streaks are present in the reconstruction. To be applied before normalisation. ---
- method: remove_outlier
  module_path: httomolibgpu.misc.corr
  parameters:
    kernel_size: 3  # The size of the 3D neighbourhood surrounding the voxel. Odd integer.
    dif: 1000 # A difference between the outlier value and the median value of neighbouring pixels.
# --- Normalisation of projection data using collected flats/darks images. --- 
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    flats_multiplier: 1.0
    darks_multiplier: 1.0
    cutoff: 10.0
    minus_log: true  # If Paganin method is used bellow, set it to false.
    nonnegativity: false
    remove_nans: false
# --- Method to remove stripe artefacts in the data that lead to ring artefacts in the reconstruction. --- 
- method: remove_all_stripe
  module_path: httomolibgpu.prep.stripe
  parameters:
    snr: 3.0
    la_size: 61
    sm_size: 21
    dim: 1
# --- Reconstruction method. ---
- method: FBP3d_tomobar
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}  # Reference to center of rotation side output above OR a float number.
    detector_pad: 0 # Horizontal detector padding to minimise circle/arc-type artifacts in the reconstruction
    filter_freq_cutoff: 0.35
    recon_size: null
    recon_mask_radius: 0.95 # Zero pixels outside the mask-circle radius.
    neglog: false # Perform negative log here if it was previously switched off.
# --- Calculate global statistics on the reconstructed volume, required for data rescaling. ---
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
# --- Rescaling the data using min/max obtained from `calculate_stats`. ---
- method: rescale_to_int
  module_path: httomolib.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}}
# --- Saving data into images. ---
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif  # `tif` or `jpeg` can be used.
    asynchronous: true

GPU pipeline as above and Total Variation denoising on the result of the FBP reconstruction.

# This pipeline is supported by HTTomo ver. 2.5
# --- Standard tomography loader for NeXus files. ---
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: auto
    image_key_path: auto
    rotation_angles: auto
    preview:
      detector_x:   # horizontal data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
      detector_y:   # vertical data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
    darks: null
    flats: null
# --- Center of Rotation auto-finding. Required for reconstruction bellow. ---
- method: find_center_vo
  module_path: httomolibgpu.recon.rotation
  parameters:
    ind: null  # A vertical slice (sinogram) index to calculate CoR, `mid` can be used for middle
    average_radius: 0 # Average several sinograms to improve SNR, one can try 3-5 range
    cor_initialisation_value: null  # Use if an approximate CoR is known
    smin: -50
    smax: 50
    srad: 6.0
    step: 0.5
    ratio: 0.5
    drop: 20
  id: centering
  side_outputs:
    cor: centre_of_rotation  # An estimated CoR value provided as a side output
# --- Removing unresponsive pixels in the data, aka zingers. Use if sharp streaks are present in the reconstruction. To be applied before normalisation. ---
- method: remove_outlier
  module_path: httomolibgpu.misc.corr
  parameters:
    kernel_size: 3  # The size of the 3D neighbourhood surrounding the voxel. Odd integer.
    dif: 1000 # A difference between the outlier value and the median value of neighbouring pixels.
# --- Normalisation of projection data using collected flats/darks images. --- 
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    flats_multiplier: 1.0
    darks_multiplier: 1.0
    cutoff: 10.0
    minus_log: true  # If Paganin method is used bellow, set it to false.
    nonnegativity: false
    remove_nans: false
# --- Method to remove stripe artefacts in the data that lead to ring artefacts in the reconstruction. --- 
- method: remove_all_stripe
  module_path: httomolibgpu.prep.stripe
  parameters:
    snr: 3.0
    la_size: 61
    sm_size: 21
    dim: 1
# --- Reconstruction method. ---
- method: FBP3d_tomobar
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}  # Reference to center of rotation side output above OR a float number.
    detector_pad: 0 # Horizontal detector padding to minimise circle/arc-type artifacts in the reconstruction
    filter_freq_cutoff: 0.35
    recon_size: null
    recon_mask_radius: 0.95 # Zero pixels outside the mask-circle radius.
    neglog: false # Perform negative log here if it was previously switched off.
# --- Using denoising method to reduce noise. ---
- method: total_variation_PD
  module_path: httomolibgpu.misc.denoise
  parameters:
    regularisation_parameter: 1.0e-05
    iterations: 1000
    isotropic: true
    nonnegativity: false
    lipschitz_const: 8.0
# --- Calculate global statistics on the reconstructed volume, required for data rescaling. ---
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
# --- Rescaling the data using min/max obtained from `calculate_stats`. ---
- method: rescale_to_int
  module_path: httomolib.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}}
# --- Saving data into images. ---
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif  # `tif` or `jpeg` can be used.
    asynchronous: true

CPU Pipeline templates#

The CPU-pipelines mostly use TomoPy methods. They are executed solely on the CPU and therefore expected to be slower than the GPU pipelines.

CPU pipeline using auto-centering and the gridrec reconstruction method on the CPU (TomoPy).

# This pipeline is supported by HTTomo ver. 2.5
# --- Standard tomography loader for NeXus files. ---
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: auto
    image_key_path: auto
    rotation_angles: auto
    preview:
      detector_x:   # horizontal data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
      detector_y:   # vertical data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
    darks: null
    flats: null
# --- Removing unresponsive pixels in the data, aka zingers. Use if sharp streaks are present in the reconstruction. To be applied before normalisation. ---
- method: remove_outlier
  module_path: tomopy.misc.corr
  parameters:
    dif: 0.1  # A difference between the outlier value and the median value of neighbouring pixels.
    size: 3
    axis: auto
# --- Normalisation of projection data using collected flats/darks images. --- 
- method: normalize
  module_path: tomopy.prep.normalize
  parameters:
    cutoff: null
    averaging: mean
# --- Normalisation of projection data using collected flats/darks images. --- 
- method: minus_log
  module_path: tomopy.prep.normalize
  parameters: {}
# --- Center of Rotation auto-finding. Required for reconstruction bellow. ---
- method: find_center_vo
  module_path: tomopy.recon.rotation
  parameters:
    ind: null  # A vertical slice (sinogram) index to calculate CoR, `mid` can be used for middle
    smin: -50
    smax: 50
    srad: 6
    step: 0.25
    ratio: 0.5
    drop: 20
  id: centering
  side_outputs:
    cor: centre_of_rotation  # An estimated CoR value provided as a side output
# --- Reconstruction method. ---
- method: recon
  module_path: tomopy.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}  # Reference to center of rotation side output above OR a float number.
    sinogram_order: false
    algorithm: gridrec  # Select the required algorithm, e.g. `gridrec`
    init_recon: null
# --- Calculate global statistics on the reconstructed volume, required for data rescaling. ---
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
# --- Rescaling the data using min/max obtained from `calculate_stats`. ---
- method: rescale_to_int
  module_path: httomolib.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}}
# --- Saving data into images. ---
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif  # `tif` or `jpeg` can be used.
    asynchronous: true

DLS-specific templates#

These pipelines are specific to Diamond Light Source processing strategies and can vary between different tomographic beamlines.

Pipeline for 360-degrees data with automatic CoR/overlap finding and stitching to 180-degrees data. Paganin filter is applied to the data.

# This pipeline is supported by HTTomo ver. 2.5
# --- Standard tomography loader for NeXus files. ---
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: auto
    image_key_path: auto
    rotation_angles: auto
    preview:
      detector_x:   # horizontal data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
      detector_y:   # vertical data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
    darks: null
    flats: null
# --- Center of Rotation auto-finding. Required for reconstruction bellow. ---
- method: find_center_360
  module_path: httomolibgpu.recon.rotation
  parameters:
    ind: null  # A vertical slice (sinogram) index to calculate CoR, `mid` can be used for middle
    win_width: 10
    side: null # 'None' corresponds to fully automated determination, '0' to the left side, '1' to the right side.
    denoise: true
    norm: false
    use_overlap: false
  id: centering
  side_outputs:
    cor: centre_of_rotation  # An estimated CoR value provided as a side output
    overlap: overlap # An overlap to use for converting 360 degrees scan to 180 degrees scan.
    side: side
    overlap_position: overlap_position
# --- Normalisation of projection data using collected flats/darks images. --- 
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    flats_multiplier: 1.0
    darks_multiplier: 1.0
    cutoff: 10.0
    minus_log: true  # If Paganin method is used bellow, set it to false.
    nonnegativity: false
    remove_nans: false
# --- Using the overlap provided, converting 360 degrees scan to 180 degrees scan. --- 
- method: sino_360_to_180
  module_path: httomolibgpu.misc.morph
  parameters:
    overlap: ${{centering.side_outputs.overlap}}
    side: ${{centering.side_outputs.side}}
# --- Method to remove stripe artefacts in the data that lead to ring artefacts in the reconstruction. --- 
- method: remove_stripe_based_sorting
  module_path: httomolibgpu.prep.stripe
  parameters:
    size: 11
    dim: 1
# --- Apply a phase contrast filter to improve image contrast. --- 
- method: paganin_filter_tomopy
  module_path: httomolibgpu.prep.phase
  parameters:
    pixel_size: 0.0001
    dist: 50.0
    energy: 53.0
    alpha: 0.001  # Controls the balance between the strength of the filter and the amount of noise reduction. Higher leads to less noise and more blur.
# --- Reconstruction method. ---
- method: FBP3d_tomobar
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}  # Reference to center of rotation side output above OR a float number.
    detector_pad: 0 # Horizontal detector padding to minimise circle/arc-type artifacts in the reconstruction
    filter_freq_cutoff: 0.35
    recon_size: null
    recon_mask_radius: 0.95 # Zero pixels outside the mask-circle radius.
    neglog: false # Perform negative log here if it was previously switched off.
# --- Calculate global statistics on the reconstructed volume, required for data rescaling. ---
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
# --- Rescaling the data using min/max obtained from `calculate_stats`. ---
- method: rescale_to_int
  module_path: httomolib.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}}
# --- Saving data into images. ---
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif  # `tif` or `jpeg` can be used.
    asynchronous: true

Using a distortion correction module as a part of the pipeline with the 360-degrees data.

# This pipeline is supported by HTTomo ver. 2.5
# --- Standard tomography loader for NeXus files. ---
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: auto
    image_key_path: auto
    rotation_angles: auto
    preview:
      detector_x:   # horizontal data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
      detector_y:   # vertical data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
    darks: null
    flats: null
# --- Center of Rotation auto-finding. Required for reconstruction bellow. ---
- method: find_center_360
  module_path: httomolibgpu.recon.rotation
  parameters:
    ind: null  # A vertical slice (sinogram) index to calculate CoR, `mid` can be used for middle
    win_width: 10
    side: null # 'None' corresponds to fully automated determination, '0' to the left side, '1' to the right side.
    denoise: true
    norm: false
    use_overlap: false
  id: centering
  side_outputs:
    cor: centre_of_rotation  # An estimated CoR value provided as a side output
    overlap: overlap # An overlap to use for converting 360 degrees scan to 180 degrees scan.
    side: side
    overlap_position: overlap_position
# --- Normalisation of projection data using collected flats/darks images. --- 
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    flats_multiplier: 1.0
    darks_multiplier: 1.0
    cutoff: 10.0
    minus_log: true  # If Paganin method is used bellow, set it to false.
    nonnegativity: false
    remove_nans: false
# --- Applying optical distortion correction to projections. --- 
- method: distortion_correction_proj_discorpy
  module_path: httomolibgpu.prep.alignment
  parameters:
    metadata_path: REQUIRED  # Provide an absolute path to the text file with distortion coefficients.
    order: 3
    mode: constant
# --- Using the overlap provided, converting 360 degrees scan to 180 degrees scan. --- 
- method: sino_360_to_180
  module_path: httomolibgpu.misc.morph
  parameters:
    overlap: ${{centering.side_outputs.overlap}}
    side: ${{centering.side_outputs.side}}
# --- Method to remove stripe artefacts in the data that lead to ring artefacts in the reconstruction. --- 
- method: remove_stripe_based_sorting
  module_path: httomolibgpu.prep.stripe
  parameters:
    size: 11
    dim: 1
# --- Reconstruction method. ---
- method: FBP3d_tomobar
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}  # Reference to center of rotation side output above OR a float number.
    detector_pad: 0 # Horizontal detector padding to minimise circle/arc-type artifacts in the reconstruction
    filter_freq_cutoff: 0.35
    recon_size: null
    recon_mask_radius: 0.95 # Zero pixels outside the mask-circle radius.
    neglog: false # Perform negative log here if it was previously switched off.
# --- Calculate global statistics on the reconstructed volume, required for data rescaling. ---
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
# --- Rescaling the data using min/max obtained from `calculate_stats`. ---
- method: rescale_to_int
  module_path: httomolib.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}}
# --- Saving data into images. ---
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif  # `tif` or `jpeg` can be used.
    asynchronous: true

Parameter Sweeps templates#

Here we demonstrate how to perform a sweep across multiple values of a single parameter (see Parameter Sweeping for more details).

Note

There is no need to add image saving plugin for sweep runs as it will be added automatically. It is also preferable to keep the preview small as the time of computation can be substantial.

Parameter sweep using the !Sweep tag over several particular values (not a range) of the alpha parameter for the Paganin filter.

# This pipeline is supported by HTTomo ver. 2.5
# --- Standard tomography loader for NeXus files. ---
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: auto
    image_key_path: auto
    rotation_angles: auto
    preview:
      detector_x:   # horizontal data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
      detector_y:   # vertical data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
    darks: null
    flats: null
# --- Center of Rotation auto-finding. Required for reconstruction bellow. ---
- method: find_center_vo
  module_path: httomolibgpu.recon.rotation
  parameters:
    ind: null  # A vertical slice (sinogram) index to calculate CoR, `mid` can be used for middle
    average_radius: 0 # Average several sinograms to improve SNR, one can try 3-5 range
    cor_initialisation_value: null  # Use if an approximate CoR is known
    smin: -50
    smax: 50
    srad: 6.0
    step: 0.5
    ratio: 0.5
    drop: 20
  id: centering
  side_outputs:
    cor: centre_of_rotation  # An estimated CoR value provided as a side output
# --- Normalisation of projection data using collected flats/darks images. --- 
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    flats_multiplier: 1.0
    darks_multiplier: 1.0
    cutoff: 10.0
    minus_log: true  # If Paganin method is used bellow, set it to false.
    nonnegativity: false
    remove_nans: false
# --- Apply a phase contrast filter to improve image contrast. --- 
- method: paganin_filter_tomopy
  module_path: httomolibgpu.prep.phase
  parameters:
    pixel_size: 0.0001
    dist: 50.0
    energy: 53.0
    alpha: !Sweep  # Controls the balance between the strength of the filter and the amount of noise reduction. Higher leads to less noise and more blur.
    - 0.01
    - 0.001
    - 0.0001
# --- Reconstruction method. ---
- method: FBP3d_tomobar
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}  # Reference to center of rotation side output above OR a float number.
    detector_pad: 0 # Horizontal detector padding to minimise circle/arc-type artifacts in the reconstruction
    filter_freq_cutoff: 0.35
    recon_size: null
    recon_mask_radius: 0.95 # Zero pixels outside the mask-circle radius.
    neglog: false # Perform negative log here if it was previously switched off.