Full YAML pipelines

Full YAML pipelines#

This is a collection of ready to be used full pipelines or process lists for HTTomo. See more on What is a process list? and how to Configure efficient pipelines.

HTTomo mainly targets GPU computations, therefore the use of GPU Pipeline templates is preferable. However, when the GPU device is not available or a GPU method is not implemented, the use of CPU Pipeline templates is possible.

Note

The combination of both GPU and CPU methods is possible. If one expects to achieve the faster performance, please use the GPU methods provided, where possible.

GPU Pipeline templates#

The GPU-pipelines consist of methods from httomolibgpu (GPU) and httomolib (CPU) backend Supported libraries. Those libraries are supported directly by the HTTomo development team.

GPU pipeline with auto-centering and the FBP reconstruction method.
# --- Standard tomography loader for NeXus files. ---
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: /entry1/tomo_entry/data/data
    image_key_path: /entry1/tomo_entry/instrument/detector/image_key
    rotation_angles:
      data_path: /entry1/tomo_entry/data/rotation_angle
    preview:
      detector_x:   # horizontal data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
      detector_y:   # vertical data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
# --- Center of Rotation auto-finding. Required for reconstruction bellow. ---
- method: find_center_vo
  module_path: httomolibgpu.recon.rotation
  parameters:
    ind: null  # A vertical slice (sinogram) index to calculate CoR, `mid` can be used for middle
    average_radius: 0 # Average several sinograms to improve SNR, one can try 3-5 range
    cor_initialisation_value: null  # Use if an approximate CoR is known
    smin: -50
    smax: 50
    srad: 6.0
    step: 0.5
    ratio: 0.5
    drop: 20
  id: centering
  side_outputs:
    cor: centre_of_rotation  # A side output of the method, here a CoR scalar value
# --- Removing dead pixels in the data, aka zingers. Use if sharp streaks are present in reconstruction. ---
- method: remove_outlier
  module_path: httomolibgpu.misc.corr
  parameters:
    kernel_size: 3
    dif: 0.1  # A difference between the outlier value and the median value of neighboring pixels.
# --- Normalisation of projection data using collected flats/darks images. --- 
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    cutoff: 10.0
    minus_log: true  # If Paganin method is used bellow, set it to false.
    nonnegativity: false
    remove_nans: false
# --- Method to remove stripe artefacts in the data that lead to ring artefacts in the reconstruction. --- 
- method: remove_all_stripe
  module_path: httomolibgpu.prep.stripe
  parameters:
    snr: 3.0
    la_size: 61
    sm_size: 21
    dim: 1
# --- Reconstruction method. ---
- method: FBP
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}  # Reference to center of rotation side output OR an integer.
    filter_freq_cutoff: 0.35
    recon_size: null
    recon_mask_radius: 0.95 # Zero pixels outside the mask-circle radius.
    neglog: false # Perform negative log here if it was previously switched off.
# --- Calculate global statistics on the reconstructed volume, required for data rescaling. ---
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
# --- Rescaling the data using min/max obtained from `calculate_stats`. ---
- method: rescale_to_int
  module_path: httomolibgpu.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}}
# --- Saving data into images. ---
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif  # `tif` or `jpeg` can be used.
    asynchronous: true
GPU pipeline as above and Total Variation denoising on the result of the FBP reconstruction.
# --- Standard tomography loader for NeXus files. ---
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: /entry1/tomo_entry/data/data
    image_key_path: /entry1/tomo_entry/instrument/detector/image_key
    rotation_angles:
      data_path: /entry1/tomo_entry/data/rotation_angle
    preview:
      detector_x:   # horizontal data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
      detector_y:   # vertical data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
# --- Center of Rotation auto-finding. Required for reconstruction bellow. ---
- method: find_center_vo
  module_path: httomolibgpu.recon.rotation
  parameters:
    ind: null  # A vertical slice (sinogram) index to calculate CoR, `mid` can be used for middle
    average_radius: 0 # Average several sinograms to improve SNR, one can try 3-5 range
    cor_initialisation_value: null  # Use if an approximate CoR is known
    smin: -50
    smax: 50
    srad: 6.0
    step: 0.5
    ratio: 0.5
    drop: 20
  id: centering
  side_outputs:
    cor: centre_of_rotation  # A side output of the method, here a CoR scalar value
# --- Removing dead pixels in the data, aka zingers. Use if sharp streaks are present in reconstruction. ---
- method: remove_outlier
  module_path: httomolibgpu.misc.corr
  parameters:
    kernel_size: 3
    dif: 0.1  # A difference between the outlier value and the median value of neighboring pixels.
# --- Normalisation of projection data using collected flats/darks images. --- 
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    cutoff: 10.0
    minus_log: true  # If Paganin method is used bellow, set it to false.
    nonnegativity: false
    remove_nans: false
# --- Method to remove stripe artefacts in the data that lead to ring artefacts in the reconstruction. --- 
- method: remove_all_stripe
  module_path: httomolibgpu.prep.stripe
  parameters:
    snr: 3.0
    la_size: 61
    sm_size: 21
    dim: 1
# --- Reconstruction method. ---
- method: FBP
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}  # Reference to center of rotation side output OR an integer.
    filter_freq_cutoff: 0.35
    recon_size: null
    recon_mask_radius: 0.95 # Zero pixels outside the mask-circle radius.
    neglog: false # Perform negative log here if it was previously switched off.
# --- Using denoising method to reduce noise. ---
- method: total_variation_PD
  module_path: httomolibgpu.misc.denoise
  parameters:
    regularisation_parameter: 1.0e-05
    iterations: 1000
    isotropic: true
    nonnegativity: false
    lipschitz_const: 8.0
# --- Calculate global statistics on the reconstructed volume, required for data rescaling. ---
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
# --- Rescaling the data using min/max obtained from `calculate_stats`. ---
- method: rescale_to_int
  module_path: httomolibgpu.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}}
# --- Saving data into images. ---
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif  # `tif` or `jpeg` can be used.
    asynchronous: true

CPU Pipeline templates#

The CPU-pipelines mostly use TomoPy methods. They are executed solely on the CPU and therefore expected to be slower than the GPU pipelines.

CPU pipeline using auto-centering and the gridrec reconstruction method on the CPU (TomoPy).
# --- Standard tomography loader for NeXus files. ---
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: /entry1/tomo_entry/data/data
    image_key_path: /entry1/tomo_entry/instrument/detector/image_key
    rotation_angles:
      data_path: /entry1/tomo_entry/data/rotation_angle
    preview:
      detector_x:   # horizontal data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
      detector_y:   # vertical data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
# --- Removing dead pixels in the data, aka zingers. Use if sharp streaks are present in reconstruction. ---
- method: remove_outlier
  module_path: tomopy.misc.corr
  parameters:
    dif: 0.1  # A difference between the outlier value and the median value of neighboring pixels.
    size: 3
    axis: auto
# --- Normalisation of projection data using collected flats/darks images. --- 
- method: normalize
  module_path: tomopy.prep.normalize
  parameters:
    cutoff: null
    averaging: mean
# --- Normalisation of projection data using collected flats/darks images. --- 
- method: minus_log
  module_path: tomopy.prep.normalize
  parameters: {}
# --- Center of Rotation auto-finding. Required for reconstruction bellow. ---
- method: find_center_vo
  module_path: tomopy.recon.rotation
  parameters:
    ind: null  # A vertical slice (sinogram) index to calculate CoR, `mid` can be used for middle
    smin: -50
    smax: 50
    srad: 6
    step: 0.25
    ratio: 0.5
    drop: 20
  id: centering
  side_outputs:
    cor: centre_of_rotation  # A side output of the method, here a CoR scalar value
# --- Reconstruction method. ---
- method: recon
  module_path: tomopy.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}  # Reference to center of rotation side output OR an integer.
    sinogram_order: false
    algorithm: gridrec  # Select the required algorithm, e.g. `gridrec`
    init_recon: null
# --- Calculate global statistics on the reconstructed volume, required for data rescaling. ---
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
# --- Rescaling the data using min/max obtained from `calculate_stats`. ---
- method: rescale_to_int
  module_path: httomolibgpu.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}}
# --- Saving data into images. ---
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif  # `tif` or `jpeg` can be used.
    asynchronous: true

DLS-specific templates#

Those pipelines are specific to Diamond Light Source processing strategies and can vary between different tomographic beamlines.

An example of DIAD-k11 beamline pipeline with auto-centering and FBP reconstruction on the GPU.
# --- Standard tomography loader for NeXus files. ---
- method: standard_tomo_diad
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: /entry/imaging/data
    image_key_path: /entry/instrument/imaging/image_key
    rotation_angles:
      data_path: /entry/imaging_sum/gts_cs_theta
    preview:
      detector_x:   # horizontal data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
      detector_y:   # vertical data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
# --- Center of Rotation auto-finding. Required for reconstruction bellow. ---
- method: find_center_vo
  module_path: httomolibgpu.recon.rotation
  parameters:
    ind: null  # A vertical slice (sinogram) index to calculate CoR, `mid` can be used for middle
    average_radius: 0 # Average several sinograms to improve SNR, one can try 3-5 range
    cor_initialisation_value: null  # Use if an approximate CoR is known
    smin: -50
    smax: 50
    srad: 6.0
    step: 0.5
    ratio: 0.5
    drop: 20
  id: centering
  side_outputs:
    cor: centre_of_rotation  # A side output of the method, here a CoR scalar value
# --- Removing dead pixels in the data, aka zingers. Use if sharp streaks are present in reconstruction. ---
- method: remove_outlier
  module_path: httomolibgpu.misc.corr
  parameters:
    kernel_size: 3
    dif: 0.1  # A difference between the outlier value and the median value of neighboring pixels.
# --- Normalisation of projection data using collected flats/darks images. --- 
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    cutoff: 10.0
    minus_log: true  # If Paganin method is used bellow, set it to false.
    nonnegativity: false
    remove_nans: false
# --- Method to remove stripe artefacts in the data that lead to ring artefacts in the reconstruction. --- 
- method: remove_stripe_based_sorting
  module_path: httomolibgpu.prep.stripe
  parameters:
    size: 11
    dim: 1
# --- Reconstruction method. ---
- method: FBP
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}  # Reference to center of rotation side output OR an integer.
    filter_freq_cutoff: 0.35
    recon_size: null
    recon_mask_radius: 0.95 # Zero pixels outside the mask-circle radius.
    neglog: false # Perform negative log here if it was previously switched off.
# --- Calculate global statistics on the reconstructed volume, required for data rescaling. ---
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
# --- Rescaling the data using min/max obtained from `calculate_stats`. ---
- method: rescale_to_int
  module_path: httomolibgpu.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}}
# --- Saving data into images. ---
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif  # `tif` or `jpeg` can be used.
    asynchronous: true
Pipeline for 360-degrees data with automatic CoR/overlap finding and stitching to 180-degrees data. Paganin filter is applied to the data.
# --- Standard tomography loader for NeXus files. ---
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: /entry1/tomo_entry/data/data
    image_key_path: /entry1/tomo_entry/instrument/detector/image_key
    rotation_angles:
      data_path: /entry1/tomo_entry/data/rotation_angle
    preview:
      detector_x:   # horizontal data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
      detector_y:   # vertical data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
# --- Normalisation of projection data using collected flats/darks images. --- 
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    cutoff: 10.0
    minus_log: true  # If Paganin method is used bellow, set it to false.
    nonnegativity: false
    remove_nans: false
# --- Center of Rotation auto-finding. Required for reconstruction bellow. ---
- method: find_center_360
  module_path: httomolibgpu.recon.rotation
  parameters:
    ind: null  # A vertical slice (sinogram) index to calculate CoR, `mid` can be used for middle
    win_width: 10
    side: null # 'None' corresponds to fully automated determination, '0' to the left side, '1' to the right side.
    denoise: true
    norm: false
    use_overlap: false
  id: centering
  side_outputs:
    cor: centre_of_rotation  # A side output of the method, here a CoR scalar value
    overlap: overlap # An overlap to use for converting 360 degrees scan to 180 degrees scan.
    side: side
    overlap_position: overlap_position
# --- Using the overlap provided, converting 360 degrees scan to 180 degrees scan. --- 
- method: sino_360_to_180
  module_path: httomolibgpu.misc.morph
  parameters:
    overlap: ${{centering.side_outputs.overlap}}
    rotation: left  # 'left' if rotation center is close to the left of the field-of-view, 'right' otherwise.
# --- Apply a phase contrast filter to improve image contrast. --- 
- method: paganin_filter_tomopy
  module_path: httomolibgpu.prep.phase
  parameters:
    pixel_size: 0.0001
    dist: 50.0
    energy: 53.0
    alpha: 0.001  # Controls the balance between the strength of the filter and the amount of noise reduction. Higher leads to less noise and more blur.
# --- Method to remove stripe artefacts in the data that lead to ring artefacts in the reconstruction. --- 
- method: remove_stripe_based_sorting
  module_path: httomolibgpu.prep.stripe
  parameters:
    size: 11
    dim: 1
# --- Reconstruction method. ---
- method: FBP
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}  # Reference to center of rotation side output OR an integer.
    filter_freq_cutoff: 0.35
    recon_size: null
    recon_mask_radius: 0.95 # Zero pixels outside the mask-circle radius.
    neglog: false # Perform negative log here if it was previously switched off.
# --- Calculate global statistics on the reconstructed volume, required for data rescaling. ---
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
# --- Rescaling the data using min/max obtained from `calculate_stats`. ---
- method: rescale_to_int
  module_path: httomolibgpu.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}}
# --- Saving data into images. ---
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif  # `tif` or `jpeg` can be used.
    asynchronous: true
Using distortion correction module as a part of pipeline with 360-degrees data.
# --- Standard tomography loader for NeXus files. ---
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: /entry1/tomo_entry/data/data
    image_key_path: /entry1/tomo_entry/instrument/detector/image_key
    rotation_angles:
      data_path: /entry1/tomo_entry/data/rotation_angle
    preview:
      detector_x:   # horizontal data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
      detector_y:   # vertical data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
# --- Normalisation of projection data using collected flats/darks images. --- 
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    cutoff: 10.0
    minus_log: true  # If Paganin method is used bellow, set it to false.
    nonnegativity: false
    remove_nans: false
# --- Applying optical distortion correction to projections. --- 
- method: distortion_correction_proj_discorpy
  module_path: httomolibgpu.prep.alignment
  parameters:
    metadata_path: REQUIRED  # Provide an absolute path to the text file with distortion coefficients.
    order: 3
    mode: constant
# --- Center of Rotation auto-finding. Required for reconstruction bellow. ---
- method: find_center_360
  module_path: httomolibgpu.recon.rotation
  parameters:
    ind: null  # A vertical slice (sinogram) index to calculate CoR, `mid` can be used for middle
    win_width: 10
    side: null # 'None' corresponds to fully automated determination, '0' to the left side, '1' to the right side.
    denoise: true
    norm: false
    use_overlap: false
  id: centering
  side_outputs:
    cor: centre_of_rotation  # A side output of the method, here a CoR scalar value
    overlap: overlap # An overlap to use for converting 360 degrees scan to 180 degrees scan.
    side: side
    overlap_position: overlap_position
# --- Using the overlap provided, converting 360 degrees scan to 180 degrees scan. --- 
- method: sino_360_to_180
  module_path: httomolibgpu.misc.morph
  parameters:
    overlap: ${{centering.side_outputs.overlap}}
    rotation: left  # 'left' if rotation center is close to the left of the field-of-view, 'right' otherwise.
# --- Method to remove stripe artefacts in the data that lead to ring artefacts in the reconstruction. --- 
- method: remove_stripe_based_sorting
  module_path: httomolibgpu.prep.stripe
  parameters:
    size: 11
    dim: 1
# --- Reconstruction method. ---
- method: FBP
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}  # Reference to center of rotation side output OR an integer.
    filter_freq_cutoff: 0.35
    recon_size: null
    recon_mask_radius: 0.95 # Zero pixels outside the mask-circle radius.
    neglog: false # Perform negative log here if it was previously switched off.
# --- Calculate global statistics on the reconstructed volume, required for data rescaling. ---
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
# --- Rescaling the data using min/max obtained from `calculate_stats`. ---
- method: rescale_to_int
  module_path: httomolibgpu.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}}
# --- Saving data into images. ---
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif  # `tif` or `jpeg` can be used.
    asynchronous: true

Parameter Sweeps templates#

Here we demonstrate how to perform a sweep across multiple values of a single parameter (see Parameter Sweeping for more details).

Parameter sweep over several CoR values (center param) in reconstruction method, and saving the result as tiffs. Note that there is need to add image saving plugin in this case. It is also preferable to keep the preview small and when it is not defined the central slice will be used.
# --- Standard tomography loader for NeXus files. ---
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: /entry1/tomo_entry/data/data
    image_key_path: /entry1/tomo_entry/instrument/detector/image_key
    rotation_angles:
      data_path: /entry1/tomo_entry/data/rotation_angle
    preview:
      detector_x:   # horizontal data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
      detector_y:   # vertical data previewing/cropping.
        # when null, the full data dimension is used, i.e., no previewing
        start: null
        stop: null
# --- Normalisation of projection data using collected flats/darks images. --- 
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    cutoff: 10.0
    minus_log: true  # If Paganin method is used bellow, set it to false.
    nonnegativity: false
    remove_nans: false
# --- Reconstruction method. ---
- method: FBP
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: !SweepRange  # Reference to center of rotation side output OR an integer.
      start: 1100
      stop: 1300
      step: 25
    filter_freq_cutoff: 0.35
    recon_size: null
    recon_mask_radius: 0.95 # Zero pixels outside the mask-circle radius.
    neglog: false # Perform negative log here if it was previously switched off.
Contents