Originally published at tothenoc.substack.com

Why we test from the outside first.

A client could not connect to mail. The server was up. The service was up. I tested authentication on the server: it passed. I tested TLS on the server: it passed. I looked in the logs for the client's attempts: nothing. Not a failed login, not a rejected connection. Nothing.

That "nothing" was the answer, and I lost an hour before I read it correctly.

The server runs an intrusion prevention layer. It sits in front of every service and drops packets from addresses it decided are hostile. A dropped packet never reaches the mail server, so the mail server has nothing to log. The client sees "could not connect". The server sees a quiet evening.

What made the address hostile: a few failed logins. The user had a wrong password saved, the mail client kept retrying, and every retry from the phone, the laptop and the tablet came from the same public IP. One ban, the whole house offline. The harder you try to fix the password, the faster you lock yourself out.

Two months later I did it to myself. I was checking certificates on our own mail ports, from my own connection: open TLS, look at the certificate, close. To the server, a handful of sessions that opened and never said anything look like a scanner. Four-hour ban on my home IP, website and mail included.

I did not notice it. What noticed it was a daily check that runs from outside and reads our sitemaps. It came back with zero pages, on every site at once. Zero on everything is not a content problem. It is a network problem, between the checker and the server.

Every test I ran on the server passed both times. Every one of them was true and useless, because none of them went through the path the user takes.

The rule we keep now:

  • test from where the user is, not from the server,
  • when a client cannot connect and the server has no trace of it, check the ban list before anything else,
  • a result of zero on every target is the checker being blocked, until proven otherwise,
  • do not let anyone "keep trying" a password against a server that bans.

Inside checks tell you the service is alive. Only an outside check tells you anyone can reach it.

If you are deciding who watches this for you, these are the questions to ask a provider: what an outsourced NOC actually covers, and what it costs.

-- Alex