Run an end-to-end simulation in development#

To simulate all system components running together in a development environment, follow these steps in order. Each component should be run in a separate terminal tab/window and left running in the background.

Prerequisites#

  1. Activate virtual environment in every terminal before proceeding:

    source .venv/bin/activate

  2. Verify test recordings exist:

    ls -la ../smartem-decisions-test-recordings/

    Available recordings:

    • bi37600-29_fsrecord.tar.gz

    • bi37708-42_fsrecord.tar.gz (recommended for testing)

  3. Prepare output directory for playback:

    mkdir -p ../epu-test-dir

Step 1: Launch backend infrastructure services#

Deploy PostgreSQL, RabbitMQ, and containerised API/worker services:

./tools/dev-k8s.sh up

Expected output: All pods should show as “Running” with services accessible on:

  • SmartEM API: http://localhost:30080/docs

  • RabbitMQ Management: http://localhost:30673

  • Adminer Database UI: http://localhost:30808

Verification:

curl -s http://localhost:30080/status

Step 2: Launch local consumer (optional)#

If you want to run a local consumer instead of the containerised worker:

RABBITMQ_URL=amqp://guest:guest@localhost:30672/ python -m smartem_backend.consumer  # pragma: allowlist secret

Note: The containerised deployment already includes a worker, so this step is optional.

Step 3: Launch SmartEM agent in watch mode#

Configure the agent to connect to the containerised API:

python -m smartem_agent watch --api-url http://localhost:30080 ../epu-test-dir

Expected output:

INFO - Initialized SmartEM API client with base URL: http://localhost:30080
INFO - API is reachable at http://localhost:30080 - Status: ...

Step 4: Start session recording playback#

Begin the microscopy data simulation:

python ./tools/fsrecorder/fsrecorder.py replay ../smartem-decisions-test-recordings/bi37708-42_fsrecord.tar.gz ../epu-test-dir/ | tee epu-test-dir.log

Expected behaviour:

  • Progress updates showing file creation events

  • Files being created in ../epu-test-dir/atlas/...

  • SmartEM agent detecting file changes and sending data to API

  • Database records being created (visible via API endpoints)

Step 5: Run external message simulator (complete feedback loop)#

Start the external message simulator to send processing results that trigger agent instructions:

python tools/external_message_simulator.py workflow-simulation --delay 3.0

Alternative simulation options:

# Send specific message types
python tools/external_message_simulator.py gridsquare-prediction --count 3
python tools/external_message_simulator.py foilhole-prediction --count 2

# Batch simulation with different quality scenarios
python tools/external_message_simulator.py batch-simulation --gridsquare-count 5 --scenario mixed

Expected behaviour:

  • External simulator publishes messages to RabbitMQ

  • Backend consumer processes messages and transforms them

  • Agent receives instructions via SSE (Server-Sent Events)

  • Agent logs show received instructions and decision-making

  • Complete feedback loop: files → processing → ML results → agent instructions

Monitoring the simulation#

  1. Check API status and data:

    curl -s http://localhost:30080/status | jq
curl -s http://localhost:30080/acquisitions | jq

  2. Monitor file system activity:

    watch -n 2 "find ../epu-test-dir -type f | wc -l"

  3. View logs:

    • Recording playback: tail -f epu-test-dir.log

    • Kubernetes pods: kubectl logs -f deployment/smartem-http-api -n smartem-decisions

Troubleshooting#

Agent connection issues#

  • Ensure agent uses --api-url http://localhost:30080 (not the default port 8000)

  • Check API accessibility: curl http://localhost:30080/status

Backend services not starting#

  • Check k3s permissions: ensure you can run kubectl get pods

  • Verify Docker registry access in .dev.env configuration

  • Check pod status: kubectl get pods -n smartem-decisions

Consumer connection issues#

  • Use correct RabbitMQ URL: RABBITMQ_URL=amqp://guest:guest@localhost:30672/

  • Verify RabbitMQ is accessible: curl http://localhost:30673 (management UI)

Recording playback issues#

  • Ensure recording file exists and is readable

  • Check target directory permissions: ls -la ../epu-test-dir

  • Monitor playback progress in log file

Success criteria#

A successful end-to-end simulation should show:

  1. ✅ All backend services running (check with kubectl get pods -n smartem-decisions)

  2. ✅ SmartEM agent connected to API (check agent logs)

  3. ✅ Recording playback creating files (check ../epu-test-dir contents)

  4. ✅ API receiving data (check curl http://localhost:30080/acquisitions)

  5. ✅ Files being processed and stored in database

  6. ✅ External simulator publishing messages to RabbitMQ

  7. ✅ Consumer processing external messages and transforming them

  8. ✅ Agent receiving instructions via SSE (check agent logs for instruction reception)

  9. ✅ Complete feedback loop: file detection → processing → ML results → agent instructions

Cleanup#

To stop all services:

# Stop backend services
./tools/dev-k8s.sh down

# Kill background processes
pkill -f smartem_agent
pkill -f smartem_backend.consumer
pkill -f fsrecorder
pkill -f external_message_simulator

# Clean up test data (optional)
rm -rf ../epu-test-dir
rm -f epu-test-dir.log