Looking for a clean way to orchestrate ROS 2 bringup with systemd so the robot recovers from a hard power cut without manual intervention. On a Jetson Orin NX (Ubuntu 22.04, Humble), I’m considering a staged target (network → microcontroller → nodes) with watchdogs and 30s timeouts — curious what patterns you’re using for health checks and retry/backoff to keep boots deterministic.
But > → microcontroller → nodes) with watchdogs and 30s timeouts — curious what patterns you’re using Agree — on our Orin NX (Humble) we made each stage a Type=notify unit with WatchdogSec=20s and a tiny shim that sd_notify(READY=1) only after health checks pass. For the MCU, bind to dev-ttyACM0 with Requires=dev-ttyACM0.device and After=dev-ttyACM0.device plus ExecStartPre=udevadm settle; for nodes, set Restart=always with RestartSec=5s, but make network only Wants=network-online.target so boot never hangs.
On our Orin NX (22.04/Humble) the biggest stabilizer was using systemd’s WatchdogSec with sd_notify: a tiny bringup wrapper pings WATCHDOG=1 until required nodes pass a quick ros2 service check, then sends READY=1; if checks stall, it exits so systemd retries. We also gate bringup After=systemd-time-wait-sync.service to avoid DDS weirdness from clock jumps, and cap each stage with RuntimeMaxSec=20s instead of one blanket “30s timeouts.” Small caveat: prefer Wants=network-online.target over Requires so a flaky link doesn’t block the whole boot. @hannah_spring and the docs help here: https://www.freedesktop.org/software/systemd/man/latest/syste.
But quick tip: we use a ‘dirty-boot’ stamp to decide how aggressive bringup should be — ExecStopPost writes /var/lib/robot/clean_shutdown, and on next boot units with ConditionPathExists=!/var/lib/robot/clean_shutdown run the full reset before starting ROS 2 (then clear the stamp); see systemd.unit. Caveat: make the reset path idempotent and clear the stamp only after the graph is healthy.
One trick that’s kept our boots deterministic: a udev rule for the MCU’s VID/PID with ENV{SYSTEMD_WANTS}=mcu-ready.target, then bringup units have After=mcu-ready.target so nothing runs until the device really enumerates. Paired with @teddyc’s sd_notify flow, we add StartLimitIntervalSec/StartLimitBurst and a small RandomizedDelaySec to avoid restart storms after a hard cut. Caveat: if the MCU might be absent, add a PathExists timeout and fall back to a degraded target so the rest comes up — like brewing coffee even if the grinder jams.
And this drove me nuts until we pinned discovery: we run a Fast DDS Discovery Server as a systemd service that starts before bringup and point nodes at it, so discovery doesn’t hinge on the network stage and boots stay deterministic after hard cuts. Small caveat: beware network-online.target hanging; we use systemd-networkd-wait-online --interface=eth0 --timeout=5 so you still hit your “30s timeouts” and move on. refs: 5.3.4. Discovery Server Settings - 3.4.0 and systemd-networkd-wait-online.service.
Biggest stabilizer for our Humble bringup on Orin NX: make the launch wrapper a tiny watchdog client. Run it as Type=notify with WatchdogSec=10 and have it sd_notify READY once the core nodes are up, then keep kicking; if any required node drops (we poll the ROS graph), exit so systemd restarts cleanly. Pair that with Restart=on-failure and a modest StartLimitBurst/StartLimitIntervalSec to avoid thrash; your 30s stages are fine, but I’d prefer ConditionPathExists=/dev/ttyACM0 over long timeouts; docs: systemd.service.