Troubleshooting

Troubleshooting#

Start by reading user.log in the run directory. If it does not explain the failure, inspect debug.log and rerun pipeline validation with the input file:

$ python -m httomo check pipeline.yaml input.nxs

Pipeline validation fails#

Check YAML indentation, method names, parameter names and required values. When an input file is supplied, HTTomo also verifies explicit HDF5 dataset paths. See Validate the pipeline and Pipeline file reference.

HTTomo cannot find NXtomo data#

Inspect the input hierarchy and confirm that the NXtomo NX_class and definition attributes are present. If automatic discovery is not suitable, set data_path, image_key_path and rotation_angles explicitly. See Automatic NXtomo discovery and Creating an NXtomo file.

MPI or parallel HDF5 fails at startup#

Confirm that HTTomo, mpi4py and h5py use compatible MPI libraries and that h5py reports parallel support:

$ python -c "import h5py; print(h5py.get_config().mpi)"

The command must print True. Inconsistent MPI installations commonly cause import errors, immediate termination or hangs during file access.

CUDA or CuPy cannot see a GPU#

Check the NVIDIA driver and CuPy runtime before running HTTomo:

$ nvidia-smi
$ python -c "import cupy; print(cupy.cuda.runtime.getDeviceCount())"

The installed CuPy CUDA package must be compatible with the system driver. Use a CPU pipeline when no CUDA-capable GPU is available.

GPU memory is exhausted#

An out-of-memory error can be caused by either the pipeline block size or other processes already using the device.

  1. Run nvidia-smi and stop unrelated jobs using the selected GPU.

  2. In a multi-process run, assign one MPI rank to each GPU. Do not allow several ranks to select the same device unless the pipeline and hardware were sized for that arrangement.

  3. Reduce the input preview, particularly detector_y while testing.

  4. Set a smaller per-process ceiling, for example --max-memory 12G. For GPU sections this caps the budget used to calculate the block size. It may also cause section data to use slower disk-backed storage.

  5. If one method still fails while others fit, its GPU memory estimator may be inaccurate. Record the method, parameters, input shape and debug.log when reporting the problem.

See Memory and performance for user guidance and Memory estimation and block sizing when diagnosing an estimator.

Pipeline and method templates do not match#

Errors about an unknown method or parameter often mean that a pipeline was generated for a different HTTomo or httomo-backends version. Use Versioned downloads for a released HTTomo version and check the Version compatibility notes before updating one component independently.

The output directory cannot be created or written#

HTTomo creates a run directory below OUT_DIR. Confirm that the parent directory exists where required by the scheduler, is writable by every MPI rank and has enough free space for requested intermediate files. With a multi-node run, use storage visible from every node. The directory supplied to --reslice-dir must already exist and be writable.

Re-slicing is unexpectedly slow#

A change between projection and sinogram processing can require temporary disk-backed data. Put --reslice-dir on fast storage accessible to every participating process. See Re-slicing and the command-line reference.

Finding more information#

The Run output, logs and monitoring page explains common progress messages and generated files. When reporting a reproducible problem, include the HTTomo version, pipeline, relevant log excerpt, execution command and hardware configuration.