EPU Output Directory Structure#

Directory layout on the EPU machine will differ from the layout of that same directory synced to file storage. Layout relevant to epu data intake component is as it appears on the EPU machines, where the watcher runs.

In a user visit there can be more than one EpuSession.dm and more than one “project dir” as there will be one for each grid. We usually treat each EpuSession.dm independently, but they all connect to the same atlas.

grid-data-dir-structure-example
├── EpuSession.dm
├── Images-Disc1
│  ├── GridSquare_8999138
│  │  ├── Data
│  │  │  ├── FoilHole_9015883_Data_9017347_6_20250108_154915.jpg
│  │  │  ├── FoilHole_9015883_Data_9017347_6_20250108_154915.xml
│  │  │  ├── FoilHole_9015883_Data_9017354_6_20250108_154918.jpg
│  │  │  └── FoilHole_9015883_Data_9017354_6_20250108_154918.xml
│  │  ├── FoilHoles
│  │  │  ├── FoilHole_9015889_20250108_154715.jpg
│  │  │  ├── FoilHole_9015889_20250108_154715.xml
│  │  │  ├── FoilHole_9015889_20250108_154725.jpg
│  │  │  └── FoilHole_9015889_20250108_154725.xml
│  │  ├── GridSquare_20250108_151151.jpg
│  │  └── GridSquare_20250108_151151.xml
│  └── GridSquare_8999186
│      ├── Data
│      │  ├── FoilHole_9028219_Data_9017347_50_20250109_062621.jpg
│      │  ├── FoilHole_9028219_Data_9017347_50_20250109_062621.xml
│      │  ├── FoilHole_9028219_Data_9017354_50_20250109_062624.jpg
│      │  └── FoilHole_9028219_Data_9017354_50_20250109_062624.xml
│      ├── FoilHoles
│      │  ├── FoilHole_9028276_20250109_061712.jpg
│      │  ├── FoilHole_9028276_20250109_061712.xml
│      │  ├── FoilHole_9028276_20250109_061722.jpg
│      │  ├── FoilHole_9028276_20250109_061722.xml
│      │  └── FoilHole_9028325_20250109_063110.jpg
│      ├── GridSquare_20250108_152955.jpg
│      └── GridSquare_20250108_152955.xml
└── Metadata
    ├── GridSquare_8999138.dm
    └── GridSquare_8999186.dm

EPU Directory Structure Details#

  • An EPU directory consists of multiple subdirectories as well as .xml, .dm and .jpg files

    • Any .dm files are actually XML files

    • This directory is written to incrementally by EPU software, it does not materialise in a complete state right away

  • An EPU directory will contain a file named EpuSession.dm at root level. Useful information contained in this file:

    • Session name, id and start time;

    • fs path of the EPU directory;

    • A reference to Atlas.dm file (which resides outside the EPU directory);

    • clustering mode and radius

  • An Atlas.dm file contains general information about the acquisition session (which possibly duplicates info already found in EpuSession.dm and, crucially, information about atlas tiles

    • Atlas tiles contain positioning information, which is needed to map physical positions on the grid (measured in number of turns of the actuator) to their pixel coordinates in image files

  • An EPU directory will contain a subdirectory named Metadata/ at root level

    • Metadata directory is flat and contains a large list of files all using a naming convention GridSquare_<gridsquare_id>.dm, for example: GridSquare_8999138.dm, GridSquare_8999186.dm

    • The Metadata/ directory contains all grid squares not just the ones that are captured

    • Some Metadata/GridSquare_<gridsquare_id>.dm files will have a significantly larger file size than others. Those are the ones that were of interest and so further scans took place during the acquisition session

    • The rest of GridSquare files under Metadata/ are typically around 2.8Kb in size

  • An EPU directory contains at least one subdirectory named Images-Disc1/ at root level, and it’s possible to have multiple subdirectories with a naming convention Images-Disc<int>, though in most cases there will only be one.

    • An Images-Disc<int>/ directory will contain a number of subdirectories following a naming convention GridSquare_<gridsquare_id>/, corresponding to GridSquare files in Metadata/, but only for GridSquares of interest where further scanning took place

  • A GridSquare directory matching a glob Images-Disc<int>/GridSquare_<gridsquare_id>/ will contain:

    • a GridSquare manifest file, such as GridSquare_20250108_152955.xml

    • optionally subdirectories FoilHoles/ and Data/, containing FoilHole and Micrograph information, respectively