Full YAML pipelines

Full YAML pipelines#

This is a collection of ready to be used pipeline templates aka process lists. See more on What is a process list? and how to Configure efficient pipelines.

CPU Pipeline templates#

Basic TomoPy’s (CPU-only) pipeline for the classical 180-degrees scan
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: entry1/tomo_entry/data/data
    image_key_path: entry1/tomo_entry/instrument/detector/image_key
    rotation_angles:
      data_path: /entry1/tomo_entry/data/rotation_angle
- method: normalize
  module_path: tomopy.prep.normalize
  parameters:
    cutoff: null
    averaging: mean
- method: minus_log
  module_path: tomopy.prep.normalize
  parameters: {}
- method: find_center_vo
  module_path: tomopy.recon.rotation
  parameters:
    ind: null
    smin: -50
    smax: 50
    srad: 6
    step: 0.25
    ratio: 0.5
    drop: 20
  id: centering
  side_outputs:
    cor : centre_of_rotation
- method: recon
  module_path: tomopy.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}
    sinogram_order: false
    algorithm: 'gridrec'
    init_recon: null
- method: rescale_to_int
  module_path: httomolibgpu.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif
TomoPy’s pipeline where Previewing is demonstrated
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: entry1/tomo_entry/data/data
    image_key_path: entry1/tomo_entry/instrument/detector/image_key
    rotation_angles:
        data_path: /entry1/tomo_entry/data/rotation_angle
    preview:
      detector_y:
        start: 30
        stop: 60
- method: find_center_vo
  module_path: tomopy.recon.rotation
  parameters:
    ind: mid
    smin: -50
    smax: 50
    srad: 6
    step: 0.25
    ratio: 0.5
    drop: 20
  id: centering
  side_outputs:
    cor: centre_of_rotation
- method: remove_outlier
  module_path: tomopy.misc.corr
  parameters:
    dif: 0.1
    size: 3
    axis: auto
- method: normalize
  module_path: tomopy.prep.normalize
  parameters:
    cutoff: null
    averaging: mean
- method: minus_log
  module_path: tomopy.prep.normalize
  parameters: {}
- method: remove_stripe_fw
  module_path: tomopy.prep.stripe
  parameters:
    level: null
    wname: db5
    sigma: 2
    pad: true
- method: recon
  module_path: tomopy.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}
    sinogram_order: false
    algorithm: gridrec
    init_recon: null
    #additional parameters: AVAILABLE
  save_result: true
- method: median_filter
  module_path: tomopy.misc.corr
  parameters:
    size: 3
    axis: auto
- method: rescale_to_int
  module_path: httomolibgpu.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif
This pipeline shows how “calculate_stats” module extracts global statistics in order to be passed to “save_to_images” function which uses it to rescale data for saving images
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: entry1/tomo_entry/data/data
    image_key_path: entry1/tomo_entry/instrument/detector/image_key
    rotation_angles:
        data_path: /entry1/tomo_entry/data/rotation_angle
- method: find_center_vo
  module_path: tomopy.recon.rotation
  parameters:
    ind: mid
    smin: -50
    smax: 50
    srad: 6
    step: 0.25
    ratio: 0.5
    drop: 20
  id: centering
  side_outputs:
    cor: centre_of_rotation
- method: remove_outlier
  module_path: tomopy.misc.corr
  parameters:
    dif: 0.1
    size: 3
    axis: auto
- method: normalize
  module_path: tomopy.prep.normalize
  parameters:
    cutoff: null
    averaging: mean
- method: minus_log
  module_path: tomopy.prep.normalize
  parameters: {}
- method: recon
  module_path: tomopy.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}
    sinogram_order: false
    algorithm: gridrec
    init_recon: null
    #additional parameters: AVAILABLE
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
- method: rescale_to_int
  module_path: httomolibgpu.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}}
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif
    jpeg_quality: 95

GPU Pipeline templates#

Basic GPU pipeline which uses functions from the httomolibgpu library.
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: entry1/tomo_entry/data/data
    image_key_path: entry1/tomo_entry/instrument/detector/image_key
    rotation_angles:
      data_path: /entry1/tomo_entry/data/rotation_angle
- method: find_center_vo
  module_path: httomolibgpu.recon.rotation
  parameters:
    ind: mid
    smin: -50
    smax: 50
    srad: 6.0
    step: 0.25
    ratio: 0.5
    drop: 20
  id: centering
  side_outputs:
    cor: centre_of_rotation
- method: remove_outlier
  module_path: httomolibgpu.misc.corr
  parameters:
    dif: 0.1
    kernel_size: 3
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    cutoff: 10.0
    minus_log: true
    nonnegativity: false
    remove_nans: false
- method: remove_stripe_based_sorting
  module_path: httomolibgpu.prep.stripe
  parameters:
    size: 11
    dim: 1
- method: FBP
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}
    filter_freq_cutoff: 0.6
    recon_size: null
    recon_mask_radius: null
  save_result: true
- method: rescale_to_int
  module_path: httomolibgpu.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif

DLS Specific templates#

GPU-based pipeline using httomolibgpu methods for DIAD (k11) data. Global statistics and referencing is used.
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: /entry/imaging/data
    image_key_path: /entry/instrument/imaging/image_key
    rotation_angles:
        data_path: /entry/imaging_sum/gts_theta_value
    preview:
      detector_y:
        start: 8
        stop: 15
- method: find_center_vo
  module_path: httomolibgpu.recon.rotation
  parameters:
    ind: mid
    smin: -50
    smax: 50
    srad: 6
    step: 0.25
    ratio: 0.5
    drop: 20
  id: centering
  side_outputs:
    cor: centre_of_rotation
- method: remove_outlier
  module_path: httomolibgpu.misc.corr
  parameters:
    dif: 0.1
    kernel_size: 3
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    cutoff: 10.0
    minus_log: true
    nonnegativity: false
    remove_nans: false
- method: remove_stripe_based_sorting
  module_path: httomolibgpu.prep.stripe
  parameters:
    size: 11
    dim: 1
- method: FBP
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}
    filter_freq_cutoff: 0.6
    recon_size: null
    recon_mask_radius: null
  save_result: true
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
- method: rescale_to_int
  module_path: httomolibgpu.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}}
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif
    jpeg_quality: 95
GPU-driven pipeline for the 360-degrees data which estimates the CoR value and the overlap. The 180-degrees sinogram is obtained by stitching using the overlap value. The pipeline shows the extensive use of side_outputs and refrencing.
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: entry1/tomo_entry/data/data
    image_key_path: entry1/tomo_entry/instrument/detector/image_key
    rotation_angles:
      data_path: /entry1/tomo_entry/data/rotation_angle
- method: find_center_360
  module_path: httomolibgpu.recon.rotation
  parameters:
    ind: mid
    win_width: 10
    side: null
    denoise: true
    norm: false
    use_overlap: false
  id: centering
  side_outputs:
    cor: centre_of_rotation
    overlap: overlap
    side: side
    overlap_position: overlap_position
- method: remove_outlier
  module_path: httomolibgpu.misc.corr
  parameters:
    dif: 0.1
    kernel_size: 3
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    cutoff: 10.0
    minus_log: true
    nonnegativity: false
    remove_nans: false
- method: remove_stripe_based_sorting
  module_path: httomolibgpu.prep.stripe
  parameters:
    size: 11
    dim: 1
- method: sino_360_to_180
  module_path: httomolibgpu.misc.morph
  parameters:
    overlap: ${{centering.side_outputs.overlap}}
    rotation: right
- method: FBP
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}
    filter_freq_cutoff: 0.6
    recon_size: null
    recon_mask_radius: null
  save_result: true
- method: data_resampler
  module_path: httomolibgpu.misc.morph
  parameters:
    newshape: [500, 500]
    axis: auto
    interpolation: linear
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
- method: rescale_to_int
  module_path: httomolibgpu.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}}
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif
    jpeg_quality: 95
More advanced GPU pipeline for the 360-degrees data. Here we preview the section and then reconstruct it iteratively, the result then downsampled before saving smaller images.
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: entry1/tomo_entry/data/data
    image_key_path: entry1/tomo_entry/instrument/detector/image_key
    rotation_angles:
      data_path: /entry1/tomo_entry/data/rotation_angle
    preview:
      detector_y:
        start: 1000
        stop: 1005
- method: find_center_360
  module_path: httomolibgpu.recon.rotation
  parameters:
    ind: mid
    win_width: 10
    side: null
    denoise: true
    norm: false
    use_overlap: false
  id: centering
  side_outputs:
    cor: centre_of_rotation
    overlap: overlap
    side: side
    overlap_position: overlap_position
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    cutoff: 10.0
    minus_log: true
    nonnegativity: false
    remove_nans: false
- method: sino_360_to_180
  module_path: httomolibgpu.misc.morph
  parameters:
    overlap: ${{centering.side_outputs.overlap}}
    rotation: right
- method: remove_all_stripe
  module_path: httomolibgpu.prep.stripe
  parameters:
    snr: 3.0
    la_size: 61
    sm_size: 21
    dim: 1
- method: CGLS
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: ${{centering.side_outputs.centre_of_rotation}}
    recon_size: null
    iterations: 3
    nonnegativity: true
  save_result: true
- method: data_resampler
  module_path: httomolibgpu.misc.morph
  parameters:
    newshape: [256, 256]
    axis: auto
    interpolation: nearest
- method: calculate_stats
  module_path: httomo.methods
  parameters: {}
  id: statistics
  side_outputs:
    glob_stats: glob_stats
- method: rescale_to_int
  module_path: httomolibgpu.misc.rescale
  parameters:
    perc_range_min: 0.0
    perc_range_max: 100.0
    bits: 8
    glob_stats: ${{statistics.side_outputs.glob_stats}}
- method: save_to_images
  module_path: httomolib.misc.images
  parameters:
    subfolder_name: images
    axis: auto
    file_format: tif
    jpeg_quality: 95

Parameter Sweeps templates#

These templates demonstrate how to perform a sweep across multiple values of a single parameter (see Parameter Sweeping for more details).

Parameter sweep over 6 CoR values (center param) in recon method, and saving the result as tiffs. Note that there is need to add image saving plugin in this case. It is also preferable to keep preview small.
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: entry1/tomo_entry/data/data
    image_key_path: entry1/tomo_entry/instrument/detector/image_key
    rotation_angles:
      data_path: /entry1/tomo_entry/data/rotation_angle
    preview:
      detector_y:
        start: 60
        stop: 67
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    cutoff: 10.0
    minus_log: false
    nonnegativity: false
    remove_nans: true
- method: paganin_filter_tomopy
  module_path: httomolibgpu.prep.phase
  parameters:
    pixel_size: 0.0001
    dist: 50.0
    energy: 53.0
    alpha: 0.001
- method: FBP
  module_path: httomolibgpu.recon.algorithm
  save_result: False
  parameters:
    center: !SweepRange
      start: 60
      stop: 120
      step: 10
    filter_freq_cutoff: 1.1
    recon_size: null
    recon_mask_radius: null
Parameter sweep over 50 (alpha param) values of Paganin filter method, and saving the result as tiffs for both Paganin filter and the reconstruction module.
- method: standard_tomo
  module_path: httomo.data.hdf.loaders
  parameters:
    data_path: entry1/tomo_entry/data/data
    image_key_path: entry1/tomo_entry/instrument/detector/image_key
    rotation_angles:
      data_path: /entry1/tomo_entry/data/rotation_angle
    preview:
      detector_y:
        start: 60
        stop: 67
- method: normalize
  module_path: httomolibgpu.prep.normalize
  parameters:
    cutoff: 10.0
    minus_log: false
    nonnegativity: false
    remove_nans: true
- method: paganin_filter_tomopy
  module_path: httomolibgpu.prep.phase
  parameters:
    pixel_size: 0.0004
    dist: 50.0
    energy: 53.0
    alpha: !SweepRange
      start: 0.001
      stop: 0.5
      step: 0.01 
- method: FBP
  module_path: httomolibgpu.recon.algorithm
  parameters:
    center: 80
    filter_freq_cutoff: 1.1
    recon_size: null
    recon_mask_radius: null