Logtide 1.7.0 is out today, and it is the first release that looks below the application layer. Until now Logtide knew everything about your logs, traces and metrics, and nothing about the box they run on. A disk could reallocate sectors for weeks and the first log line about it would be an I/O error from Postgres.

The idea came out of a chat with a friend on November 7. I had been thinking about something I called "Logtide Maintenance": you install it on a server and it tells you the hardware is going bad before it actually breaks. He had the same idea at the same moment, and then said the useful thing: before building it, do the research on what can actually break. So that is where this started, with a list: disks, SSD wear, ECC memory, CPU machine checks and throttling, fans and power supplies, temperatures, network ports.

The second question was whether anyone already does this, and the honest answer is yes, partly. Netdata monitors SMART and has anomaly detection. Scrutiny is a very good disk-only dashboard. Dell iDRAC and HPE InfoSight do predictive failure analysis, but only on their own servers. Zabbix, Checkmk and Prometheus with smartctl_exporter have SMART and IPMI templates that work on fixed thresholds. What none of them does is put the hardware counters next to the logs. And a lot of hardware failures show up in the logs first: ECC and machine check errors in dmesg, I/O timeouts, SATA and NVMe link resets, PCIe AER errors. Logtide already has the logs, so that is the angle.

If you're new here: Logtide is an open-source log management and SIEM platform built for European SMBs. Privacy-first, self-hostable, GDPR-compliant. No Elastic cluster to babysit, just Docker Compose and the storage engine of your choice.

What's New

Hardware Health

There is a new section under Monitor, called Hardware (g h if you like shortcuts). It watches the physical health of bare-metal and homelab hosts:

Component What it reads
Disks SMART for HDD and SATA SSD, the NVMe health log: reallocated and pending sectors, uncorrectable and media errors, wear, spare, temperature
Memory ECC corrected and uncorrected errors per DIMM, from EDAC
CPU temperatures against the limits the sensor reports, thermal throttling counters
Cooling fans and board temperatures
Power PSU state and redundancy, voltages (IPMI)
Network CRC errors and link flaps on physical ports

Each host gets a status per component, the findings that explain it, and 7 day trends. Virtual machines are detected and shown as "No sensors", because a VM has nothing real to report and I would rather say that than show a wall of empty charts.

Monitor > Hardware with three hosts: one critical, one OK and a virtual machine with no sensors

Open a host and you get the findings first, then each component in detail. Here the SATA disk reallocated 24 sectors in a week while SMART still says "Passed", and one DIMM is logging corrected ECC errors:

The host detail page: findings, collectors and the disk table

Trends, not thresholds

This is the part I spent the most time on. SMART has its own verdict, "PASSED" or "FAILED", and it is close to useless as an early warning. A drive can report PASSED while it reallocates sectors every day, because the vendor thresholds are set so the verdict flips very late.

So the rules look at how the counters move, not only at where they are:

  • Any reallocated sector is a warning. Ten or more new ones in 7 days is critical, even if SMART still says PASSED.
  • Pending and uncorrectable sectors are a warning as soon as they are not zero.
  • UDMA CRC errors that grew in the last 24 hours are a warning. In practice that is usually a bad SATA cable, not a bad disk, and the finding says so.
  • ECC corrected errors that grew in the last 24 hours are a warning. A hundred or more in a day is critical: that DIMM is degrading fast. Any uncorrected error is critical.
  • SSD endurance is a warning at 80% of rated wear and critical at 95%.
  • A device that stops reporting is a finding too. A disk that dropped off the bus or a pulled PSU should not just vanish from the page.

Counters are compared over 24 hour and 7 day windows, and the comparison survives counter resets (a reboot resets the EDAC counts, for example).

One thing I want to be upfront about, and the docs say it too: this is early warning, not prediction. Google's 2007 study on a large disk population and Backblaze's numbers since then agree that a large share of disks, around a third, fail without any SMART warning at all. Hardware Health will catch the ones that do warn, and it will catch them earlier than SMART's own verdict. It will not catch a disk that dies silently. Keep your backups.

Debouncing was the other half of the work. A temperature sensor sitting at its limit will cross it and come back every minute, and a monitoring tool that pages you every minute gets muted within a day. A worse status has to hold for at least 3 minutes over two evaluations before it counts. Going back to OK takes 10 minutes of quiet. Each degradation is one episode with one incident, not one incident per evaluation.

Incidents, notifications and a detection pack

A degraded host opens an incident with the new source hardware, and it goes through notification routing like everything else: there is a new Hardware health event type for channels, email fallback, PagerDuty and webhooks (hardware.health.degraded and hardware.health.recovered). When the host recovers, the PagerDuty incident is resolved too, even if you muted the host in the meantime.

Not everything hardware related is a counter. Some of it only shows up in the kernel log: machine check exceptions, APEI errors, disk link resets, NVMe timeouts, PCIe AER, thermal and power events from the BMC. The new Hardware Health detection pack has 21 Sigma rules for those. It works on any kernel log you already ship to Logtide, with or without the agent.

The agent lives in its own repo

The data comes from a small agent: one Python file, standard library only, Python 3.8 or newer, about 1,500 lines. It reads smartctl, EDAC, hwmon, thermal throttle counters, ipmitool sensors and the BMC event log, and sends everything as OTLP metrics to the endpoint Logtide already had. Hardware lines from /dev/kmsg and BMC events go in as normal logs under the service hardware. Every external command has a time budget and SMART runs in its own thread, because a disk that is actually dying can hang smartctl for minutes, and the agent should keep reporting the rest of the machine when that happens.

It started inside the main repo and moved out before the release: logtide-dev/logtide-hw-agent, MIT licensed, with its own version numbers starting at 1.0.0. The core is AGPL. An agent that people install on every machine they own should be easy to audit, fork and package, and it should be able to ship a fix for a weird RAID controller without waiting for a Logtide release.

Run it with Docker:

docker run -d --name logtide-hw-agent --privileged \ -v /sys:/host/sys:ro -v /proc:/host/proc:ro \ -e HOST_ROOT=/host \ -e LOGTIDE_URL=https://api.logtide.dev \ -e LOGTIDE_API_KEY=lp_your_project_key \ logtide/hw-agent:1

Or, if Logtide runs on the same box, with the new compose profile: set HW_AGENT_API_KEY and run docker compose --profile hardware up -d. There is also a systemd unit for hosts where you would rather not run Docker. The full setup, including how to read each finding, is in the Hardware Health docs, and there is a step by step tutorial for a home server.

Everything Else Worth Knowing

Clickable rows in Traces. A click anywhere on an Operations row filters on that operation, and a click anywhere on a trace row opens the trace. Ctrl or middle click opens it in a new tab.

A shorter Operations list. The card shows the 8 slowest operations with a "Show all" toggle, so the trace list is no longer pushed off the screen.

The login page while signed in. Opening /login with an active session used to show the sign-in form, which looked exactly like being logged out. It now sends you to the dashboard.

New capability hardware.max_hosts, unlimited by default.

Upgrading 

docker compose pull docker compose up -d

Migration 079 runs on startup. It adds the hardware_hosts table and the new incident source, and does not touch existing data. The agent is optional: nothing changes until you start one.

What's Next

Hardware Health 1.0 deliberately leaves a few things out, and they are the obvious next candidates:

  • UPS monitoring through NUT, and RAID controller health (battery, virtual disk state) through the vendor CLIs. Both were on the original list of things that break.
  • Redfish, for servers where the BMC speaks it and IPMI is the worse option.
  • Per-host thresholds, for the box that runs hot by design.
  • Windows and macOS agents. The agent is Linux only today.

On the rest of Logtide, the open items are a domain expiry check for uptime monitors (#280) and a horizontal scaling strategy for the whole stack (#221).

Full changelog: CHANGELOG.md.

If you run Logtide on real hardware, I would love to know what the agent finds on your machines, and especially what it gets wrong. Controllers and BMCs are a zoo, and the agent has only seen the ones I could get my hands on. Open an issue, start a discussion, or drop a ⭐.

🌊