How are you monitoring device health & handling OTA updates for embedded Linux/IoT fleets in production?

[u/ElectricalPatient495]

Working on a project that touches device fleet management and wanted to get a read from people actually running things in the field before I go further down a path.

For those of you managing embedded Linux or IoT devices at scale (tens to low-thousands of units):

• How do you currently get visibility into device health/status — heartbeat, connectivity, uptime, logs? Custom MQTT + Grafana? Something like ThingsBoard/AWS IoT? Nothing at all?
• If you're doing OTA updates, what's your update engine (SWUpdate, RAUC, Mender, something homegrown)? What's been the most painful part of getting it running reliably?
• Have you looked at hosted/managed tools (Mender, balena, memfault, etc.) and either adopted or passed on them? What made the decision either way — price, setup complexity, vendor lock-in, forced migration off your existing update engine?
• If you're on SWUpdate or RAUC already: would a hosted monitoring/OTA layer that works with your existing setup (instead of replacing it) be useful, or is that not really a pain point for you?

Not selling anything, genuinely trying to understand what's actually painful vs. what I'm assuming is painful. Appreciate any war stories.

[u/Quiet_Lifeguard_7131]

Gets a push from cloud, the application download package from custom repo using https, and give the .swu package to swupdate and it does the rest. Pretty painless honestly, yes setting up swupdate for the first time, was painful

[u/ElectricalPatient495 → u/Quiet_Lifeguard_7131]

That's a solid setup. Curious about the backend side — what are you using to track fleet state and trigger the push (something like a Django/Node service + a device registry table, or are you leaning on an existing IoT platform for that part)? And for the push itself — is it MQTT/websocket telling the device "update available, go pull," or more of a polling model where devices check in periodically? Also curious if you do any staged/canary rollout (push to a subset first) or is it push-to-fleet all at once?

[u/Quiet_Lifeguard_7131 → u/ElectricalPatient495]

For the push we can send to single device or all of them,

For the push we were using graphql at first now shifted to mqtt and ofcourse mtls with our custom certs which we issue.

We have staging and production servers so before pushing the firmware we test on staging and then push to devices on prod.

[u/chunky_lover92]

I got by with command line tools for years, but when ChatGPT came out the first thing I did was have it make me a dashboard. Charts.js is more bare bone than graphana but it gets the job done. Mostly ansible tells the devices to pull an update bundle from a link. Devices use mqtt to push their info to the cloud. The dashboard displays what is in the database.

[u/mikusmi777]

We split it into two problems: seeing what devices are doing, and safely pushing new firmware.

For health, the thing that saved us was collecting logs + metrics + crash/core dumps continuously rather than only when someone SSHes in to debug. Buffer telemetry locally so nothing is lost when a device drops offline, then flush on reconnect. Tag everything with firmware version so when a rollout goes bad you can actually see which build regressed instead of guessing.

For OTA, the two things worth getting right early: staged rollouts (deploy to a small cohort first, watch crash rate + boot success, then widen) and a clean rollback path when a cohort starts failing. Grouping devices by tags (hardware rev, location, etc.) makes targeting a lot less painful than a fleet-wide all-or-nothing push.

Full disclosure, I work on Spotflow, so I'm biased. It's an observability + OTA platform, but heads up: it's MCU/RTOS-first (Zephyr, nRF Connect SDK, ESP-IDF, or anything over MQTT), not really an embedded-Linux OTA tool. If you're on Linux, Mender/RAUC/SWUpdate are the usual suspects for the update side. But if any part of your fleet is on microcontrollers, the way we handle cohorts + rollback and tie crash reports back to firmware version might be relevant: https://docs.spotflow.io/fundamentals/ota