An email is not an alarm

On Friday, 13 November, fifteen guests at the Hotel Ling Bao, part of the Phantasialand theme park near Cologne, were affected after the pool's automatic dosing system put too much chlorine into the water. Phantasialand told WDR that because of the Microsoft outage no emails arrived, so nobody noticed that the sauna area's safety system had raised the alarm by email long before.

Most headlines blame Microsoft. I think the lesson is elsewhere. Exchange Online going down for a day is bad, but sooner or later it happens to every email provider. If a safety alarm's only path to a human is an inbox, losing that path must not go unnoticed. From the reports, that is what happened here: the silence looked exactly like everything being fine.

What we know

According to WDR, 13 guests first complained of respiratory irritation, and the number of people affected later rose to 15. Four were taken to hospital, and one with a moderate injury was kept in for observation. Tests on site found a defect in the dosing system. The pool was closed, the water replaced, and it has since reopened. Tom's Hardware, citing Notebookcheck, says 40 emergency workers responded.

The Exchange Online incident (EX1550543) started around 16:20 UTC on 12 November, and Microsoft's service health dashboard shows it resolved at 16:10 UTC on 13 November. Microsoft traced it to a problem left over from an earlier network capacity upgrade.

What we don't know: the exact time of the incident, who made the dosing system, and whether anything other than email was wired up. The details come from the hotel and the fire brigade, nobody has published a technical account, and one commenter on Hacker News questioned whether the dates line up at all. So I'm arguing from what the reports say, and I'd treat the specifics with some caution.

Two things failed here: the dosing and the warning. Protection against a dangerous overdose has to work on site, and designing it is a job for pool engineers, not for me. The warning, though, is something every developer builds in some form, so that is what this article is about.

The short version

  1. Treat silence as a fault. A monitor that hasn't checked in on time is an alarm, not good news.
  2. Never send a safety-relevant alert through a single channel. Email can be one of them.
  3. Until someone acknowledges an alert, it should move on to the next person. Acknowledging doesn't fix anything, but it means someone owns the problem.
  4. Where people can get hurt, you also need local protection that doesn't depend on email or the internet.
  5. Test the whole chain on a schedule, from the sensor to the phone in someone's pocket.

Why email makes a poor alarm

Email was never designed to arrive on time. SMTP is store-and-forward: a server that can't hand a message over puts it in a queue and keeps retrying for hours, sometimes days, and the sender hears nothing for a long while. A message that does arrive can end up in spam, bounce off a full mailbox, sit in a shared inbox nobody reads at the weekend, or light up a phone on silent. And if the sending device's password expires or someone changes the SMTP settings, it simply stops sending. Nothing visibly breaks.

That last case is the real trap. If the only sign of trouble is a message, then no message means no trouble. A broken alarm and a healthy pool look the same from the outside.

Silence has to raise an alarm too

What I use in my own systems is heartbeat monitoring, sometimes called a dead man's switch. The system doesn't only speak up when something is wrong; it checks in regularly to say it is alive and the readings are normal. An external service (Healthchecks.io, Better Stack, Cronitor and others) expects that check-in, and if it doesn't arrive within the agreed window, the service raises the alarm itself. The question stops being "did anyone send a warning?" and becomes "has anyone told us everything is fine?"

In Laravel this is one line per scheduled task:

php
// routes/console.php
Schedule::command('backup:run')
    ->dailyAt('02:00')
    ->pingOnSuccess('https://hc-ping.com/your-check-uuid');

If the backup fails, or the scheduler itself stops running (a lost cron entry, a full disk, a server that didn't come back after a reboot), the ping never happens and the alert goes out from somewhere else. Two things have to be right for this to work. The command must exit with a non-zero code when it fails, because that is how Laravel knows it didn't succeed. And the external monitor needs the expected schedule and a grace period, so it knows when a missing ping is late. A successful ping only tells you the command finished without an error. Whether you can actually restore from the backup is something only a test restore will tell you.

What about background jobs?

For queues I send a notification on every failed job, from Queue::failing in a service provider or from the job's own failed() method. That only fires when a job runs and fails, though. If the worker itself is down, there is no failed job, no event and no notification. You have to watch that separately: are jobs being processed, and how long are they waiting? Horizon can notify you about long wait times, but the simplest check is a heartbeat again: have the scheduler push a tiny job onto the queue every few minutes that pings the external monitor. If the workers stop, the ping stops.

One more trap: don't send that notification through the same queue it is reporting on. A stuck queue won't deliver its own alarm.

How many channels?

At least two, and they should be independent. Anything that matters gets alerted by email plus a push notification, Slack or Telegram, and an SMS for the truly urgent. Independence is the point. Email and Teams on the same Microsoft 365 tenant depend on the same provider, and Microsoft said last week's outage also affected other Microsoft 365 services. They count as one channel.

The second rule is acknowledgement. Until someone has said "I've got this", nobody has the problem. On-call tools for running applications (PagerDuty, Opsgenie and the like) escalate for this reason: if the first person doesn't respond within, say, ten minutes, the next one gets called. An acknowledged alert isn't a fixed problem, but it is one that somebody is working on.

Where people can get hurt, the alarm belongs on site

Remote notifications are useful for a pool technician who isn't on the premises. But the people at risk were in the pool and sauna area. Where a fault can hurt people, you need protection that works locally and doesn't depend on email or the internet. Whether that should have been a local alarm, an automatic shut-off or both, I can't say without knowing the controller. Fire alarms go off on site for a reason. Notifications over the internet are a second layer on top of that.

What about your own systems?

You probably don't run a pool, but you almost certainly have something that only reports by email: a backup job, a payment webhook, an expiring certificate, a disk filling up, a supplier import. Ask your team one question: if this stopped tomorrow, how would we find out? If the honest answer is "when a customer complains", you know what to fix this week.

Need a hand?

If you'd like someone to go through how your Laravel or PHP application reports its own failures, and add heartbeat checks, alerts for failed and stuck jobs and a second channel where they're missing, that's work I take on as a Laravel developer. Tell me what your system runs on.


Sources

Why Claude should stay replaceable

How I keep an application independent from Claude and other AI providers using an owned interface, real evaluation cases and a fallback that works.

All articles →