Full YAML pipelines

Full YAML pipelines#

This is a collection of ready to be used pipeline templates aka process lists for HTTomo. See more on What is a process list? and how to Configure efficient pipelines.

CPU Pipeline templates#

CPU-pipelines mostly use TomoPy methods that are executed on the CPU and expected to be slower.

GPU Pipeline templates#

It is recommended to use GPU-based pipelines and methods from the httomolib and httomolibgpu libraries. Those libraries are supported directly by HTTomo development team.

DLS-specific templates#

Those pipelines will use the methods from the httomolib and httomolibgpu libraries.

An example of a typical DIAD (k11) beamline piepeline.

# Standard tomography loader for NeXus files
#-----------------------------------------------#
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: /entry/imaging/data
    image_key_path: /entry/instrument/imaging/image_key
    rotation_angles:
        data_path: /entry/imaging_sum/gts_theta_value
    preview:
      detector_x: # horizontal data previewing/cropping
        start: 
        stop:     
      detector_y: # vertical data previewing/cropping
        start: 8
        stop: 15
# Center of Rotation method for automatic centering. Required for reconstruction.
#-----------------------------------------------#
- method: find_center_vo
  module_path: httomolibgpu.recon.rotation
  parameters: # see online documentation for parameters 
    ind: mid  # specify the vertical index (slice) for calculation. mid - middle
    smin: -50
    smax: 50
    srad: 6
    step: 0.25
    ratio: 0.5
    drop: 20
  id: centering  # method's id for future referencing (see reconstruction)
  side_outputs:  # method's side outputs include scalars and/or some auxiliary data output 
    cor: centre_of_rotation
# Remove dezingers (outliers) in the data
#-----------------------------------------------#   
- method: remove_outlier
  module_path: httomolibgpu.misc.corr
  parameters:
    dif: 0.1 # this might require optimisation
    kernel_size: 3
# Normalisation of projection data with collected flats/darks.
#-----------------------------------------------#    
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    cutoff: 10.0
    minus_log: true # set to false if Paganin method is used
    nonnegativity: false
    remove_nans: false
# Remove stripes in the data that can lead to ring artefacts in reconstruction
#-----------------------------------------------#       
- method: remove_stripe_based_sorting
  module_path: httomolibgpu.prep.stripe
  parameters:
    size: 11
    dim: 1
# Reconstruction method
#-----------------------------------------------#  
- method: FBP
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}} # the reference to the found CoR. Manually found integer can be also used. 
    filter_freq_cutoff: 0.35
    recon_size: null
    recon_mask_radius: 0.95
  save_result: true # set to false if hdf5 is not needed
# Calculate global statistics on the reconstructed volume (min/max needed specifically)
#-----------------------------------------------#
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
# Rescaling the data into 8-bit unsigned integer for saving into tiffs
#-----------------------------------------------#        
- method: rescale_to_int
  module_path: httomolibgpu.misc.rescale
  parameters:
    perc_range_min: 5.0
    perc_range_max: 95.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}} # referring to min/max values of statistics
# Saving the rescaled data into tiffs
#-----------------------------------------------#       
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif

Pipeline for 360-degrees data with automatic CoR finding and stitching to 180-degrees data.

# Standard tomography loader for NeXus files
#-----------------------------------------------#
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: entry1/tomo_entry/data/data
    image_key_path: entry1/tomo_entry/instrument/detector/image_key
    rotation_angles:
      data_path: /entry1/tomo_entry/data/rotation_angle
# Automatic search for the centre of rotation and overlap to perform 360 to 180 degrees data stitching
#-----------------------------------------------#      
- method: find_center_360
  module_path: httomolibgpu.recon.rotation
  parameters:
    ind: mid       # specify the vertical index (slice) for calculation. mid - middle
    win_width: 10
    side: null
    denoise: true
    norm: false
    use_overlap: false
  id: centering # method's id for future referencing (see reconstruction)
  side_outputs: # method's side outputs include scalars and/or some auxiliary data output 
    cor: centre_of_rotation
    overlap: overlap
    side: side
    overlap_position: overlap_position
# Remove dezingers (outliers) in the data
#-----------------------------------------------#   
- method: remove_outlier
  module_path: httomolibgpu.misc.corr
  parameters:
    dif: 0.1     # this might require optimisation
    kernel_size: 3
# Normalisation of projection data with collected flats/darks.
#-----------------------------------------------#    
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    cutoff: 10.0
    minus_log: true # set to false if Paganin method is used
    nonnegativity: false
    remove_nans: false
# Remove stripes in the data that can lead to ring artefacts in reconstruction
#-----------------------------------------------#       
- method: remove_stripe_based_sorting
  module_path: httomolibgpu.prep.stripe
  parameters:
    size: 11
    dim: 1
# Convert 360 degrees data into 180 degrees data
#-----------------------------------------------#       
- method: sino_360_to_180
  module_path: httomolibgpu.misc.morph
  parameters:
    overlap: ${{centering.side_outputs.overlap}} # the reference to the found CoR. 
    rotation: right
- method: FBP
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}
    filter_freq_cutoff: 0.35
    recon_size: null
    recon_mask_radius: 0.95
  save_result: true # save the hdf5 array
# Downsample the reconstructed data to smaller data
#-----------------------------------------------#
- method: data_resampler
  module_path: httomolibgpu.misc.morph
  parameters:
    newshape: [500, 500]
    axis: auto
    interpolation: linear
# Calculate global statistics on the reconstructed volume (min/max needed specifically)
#-----------------------------------------------#    
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
# Rescaling the data into 8-bit unsigned integer for saving into tiffs
#-----------------------------------------------#  
- method: rescale_to_int
  module_path: httomolibgpu.misc.rescale
  parameters:
    perc_range_min: 5.0
    perc_range_max: 95.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}} # referring to min/max values of statistics
# Saving the rescaled data into tiffs
#-----------------------------------------------#  
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif

Parameter Sweeps templates#

These templates demonstrate how to perform a sweep across multiple values of a single parameter (see Parameter Sweeping for more details).