Stolen passwords just lost first place. Verizon’s 2026 Data Breach Investigations Report says 31% of breaches now start with software vulnerabilities, “beating stolen passwords as the top way attackers get in.” The same report still lists stolen credentials among the most frequent causes of breaches, and those attacks leave a trail in one boring table: the authentication log.
The pitch for machine learning here is familiar. Signature rules only catch what someone already wrote a rule for, and an unsupervised model learns “normal” and flags the rest. I wanted to see what that looks like on login data with real numbers, so I built the whole loop with Isolation Forest anomaly detection in Python and scikit-learn: a synthetic auth log with labeled attacks, per-user features, training, threshold selection, evaluation against a hand-written rule baseline, per-alert explanations, and a drift scenario that forces a retrain.
Short version of the results: the rules won on precision (0.906 vs 0.426), the forest won on recall (0.894 vs 0.780), and the forest was the only one that caught any of the quietest attack. Every number below comes from a real run of the anomaly detection Python code in this post.
What we’ll build:
A log generator with a fixed seed and four labeled attack types
Per-user, per-hour features that only look backward in time
An Isolation Forest and three ways to set its threshold
Precision, recall and PR-AUC, plus a comparison with SIEM-style rules
“Why did this fire?” explanations for single alerts
A drift week that breaks the threshold, and two ways to fix it
Stack: Python 3.13.18, scikit-learn 1.9.1, pandas 3.0.5, numpy 2.5.3, pytest 9.1.1. The full experiment runs in well under a minute on 2 CPUs.
What the attacks look like in an auth log
Before choosing among anomaly detection algorithms, it helps to write down what each attack leaves behind. These are the four I generate, mapped to MITRE ATT&CK where it applies:
| Attack | What the log shows | Feature that should notice |
|---|---|---|
| Credential stuffing (T1110.004) | A few datacenter IPs try 1-2 passwords on many accounts within an hour | ip_fanout, new_device, n_fail |
| Brute force, then takeover | 10-39 failures on one account, then a success from a new device abroad | n_fail, fail_ratio, new_country |
| Impossible travel | A clean login from another continent shortly after the real user’s login | geo_speed_kmh, new_device |
| Low-and-slow stuffing | One failed try per account from a residential IP in the victim’s own city | almost nothing, by design |
OWASP describes credential stuffing as “the automated injection of stolen username and password pairs” and notes it works because people reuse passwords. MITRE’s detection guidance for T1110.004 starts where you’d expect: rapid failures across different users from one IP. Identity providers ship their own versions of the travel signal: Microsoft Entra ID Protection, for example, has both “atypical travel” and “impossible travel” risk detections. The fourth attack is there to keep the experiment honest. A careful attacker spreads attempts across residential proxies and stays under every per-IP and per-account threshold.
Pipeline from raw login events to hourly per-user windows, ten features, Isolation Forest score, alert-budget threshold and an explained alert, with a retraining loop
Step 1. Generate auth logs with labeled attacks
Real labeled login data is hard to get and impossible to share, so the generator does three things on purpose:
Normal users have habits. Each of 400 users has a home city, 1-3 home IPs, 1-2 devices, a typical login hour and a login rate. About 8% of logins come from a mobile IP and 4% start with a mistyped password.
Normal users also do odd things. Some take business trips, buy new phones or forget their password and fail 4-8 times in a row. These are the false positives any detector has to live with.
The training period is not clean. Days 1-21 contain a quarter-scale copy of the attacks, unlabeled, because your historical logs aren’t clean either.
The attack injector is the interesting part:
def attacks(rng, users, day_from, day_to, scale=1.0): rows = [] n = lambda k: max(1, int(round(k * scale))) # 1) Credential stuffing: a few datacenter IPs, many users, 1-2 tries each, rare success. for _ in range(n(2)): t0 = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=float(rng.uniform(0, 24))) ips = [_ip(rng, "45.95") for _ in range(3)] for u in rng.choice(users, size=n(40), replace=False): ts = t0 + pd.Timedelta(minutes=float(rng.uniform(0, 50))) for _ in range(int(rng.integers(1, 3))): ok = rng.random() < 0.05 rows.append(_event(ts, u["user"], rng.choice(ips), "Frankfurt", "bot-" + str(rng.integers(1e6)), 16509, ok, "credential_stuffing")) if ok: break # 2) Brute force -> takeover: many failures on one account, then success from a new device abroad. for u in rng.choice(users, size=n(6), replace=False): ts = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=float(rng.uniform(0, 24))) city, ip = rng.choice(ATTACK_CITIES), _ip(rng, "91.200") for k in range(int(rng.integers(10, 40))): rows.append(_event(ts + pd.Timedelta(seconds=15 * k), u["user"], ip, city, "atk-dev", 9123, False, "brute_force")) rows.append(_event(ts + pd.Timedelta(minutes=12), u["user"], ip, city, "atk-dev", 9123, True, "brute_force")) # 3) Impossible travel: phished password used from another continent, no failures at all. for u in rng.choice(users, size=n(10), replace=False): ts = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=(u["peak_hour"] + 1.5) % 24) rows.append(_event(ts - pd.Timedelta(hours=1), u["user"], u["ips"][0], u["city"], u["devices"][0], u["asn"], True, "normal")) rows.append(_event(ts, u["user"], _ip(rng, "102.89"), rng.choice(ATTACK_CITIES), "atk-" + u["user"], 36873, True, "impossible_travel")) # 4) Low-and-slow stuffing via residential proxies in the victim's own city: one try per user. for u in rng.choice(users, size=n(25), replace=False): ts = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=(u["peak_hour"] + rng.normal(0, 2)) % 24) rows.append(_event(ts, u["user"], _ip(rng, "73." + str(rng.integers(50, 99))), u["city"], u["devices"][0], u["asn"], False, "low_and_slow")) return rows «Full generator (users, normal traffic, benign oddities)»:
def make_users(rng, n_users): users = [] for u in range(n_users): city = rng.choice(HOME_CITIES, p=[.22, .18, .14, .14, .1, .12, .1]) users.append({ "user": f"u{u:04d}", "city": city, "ips": [_ip(rng, f"73.{rng.integers(0, 50)}") for _ in range(rng.integers(1, 4))], "devices": [f"d{u:04d}{k}" for k in range(rng.integers(1, 3))], "asn": int(rng.choice([7922, 701, 20115, 5650, 2856, 3320])), "peak_hour": float(rng.normal(14, 2.5)), # UTC "rate": float(rng.uniform(0.8, 5.0)), # logins per weekday }) return users def normal_traffic(rng, users, days, day0=0): rows = [] for u in users: for d in range(day0, day0 + days): weekend = (START + pd.Timedelta(days=d)).dayofweek >= 5 for _ in range(rng.poisson(u["rate"] * (0.3 if weekend else 1.0))): hour = (u["peak_hour"] + rng.normal(0, 2.0)) % 24 ts = START + pd.Timedelta(days=d, hours=hour) ip = rng.choice(u["ips"]) if rng.random() > 0.08 else _ip(rng, "172.56") # mobile dev = rng.choice(u["devices"]) if rng.random() < 0.04: # typo, then success rows.append(_event(ts, u["user"], ip, u["city"], dev, u["asn"], False, "normal")) ts += pd.Timedelta(seconds=int(rng.integers(5, 60))) rows.append(_event(ts, u["user"], ip, u["city"], dev, u["asn"], True, "normal")) return rows def benign_oddities(rng, users, day_from, day_to): """Legit behavior that *looks* suspicious: trips, new phones, forgotten passwords.""" rows = [] for u in rng.choice(users, size=max(1, len(users) // 25), replace=False): # business trips d = int(rng.integers(day_from, day_to)) dest = rng.choice([c for c in HOME_CITIES if c != u["city"]] + ["Singapore", "Warsaw"]) for k in range(3): ts = START + pd.Timedelta(days=d + k, hours=float(rng.uniform(7, 20))) rows.append(_event(ts, u["user"], _ip(rng, "185.20"), dest, u["devices"][0], 9009, True, "normal", "trip")) for u in rng.choice(users, size=max(1, len(users) // 30), replace=False): # new phone ts = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=u["peak_hour"] % 24) rows.append(_event(ts, u["user"], u["ips"][0], u["city"], f"{u['user']}-new", u["asn"], True, "normal", "new_phone")) for u in rng.choice(users, size=max(1, len(users) // 40), replace=False): # forgot password ts = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=u["peak_hour"] % 24) for k in range(int(rng.integers(4, 9))): rows.append(_event(ts + pd.Timedelta(seconds=20 * k), u["user"], u["ips"][0], u["city"], u["devices"][0], u["asn"], False, "normal", "forgot_password")) return rows def generate(seed=42, n_users=400, train_days=21, test_days=7): """Days [0, train_days) are 'history' (with a little unlabeled attack noise), the rest is test.""" rng = np.random.default_rng(seed) users = make_users(rng, n_users) total = train_days + test_days rows = normal_traffic(rng, users, total) rows += benign_oddities(rng, users, 0, total) rows += attacks(rng, users, 3, train_days, scale=0.25) # contamination nobody labeled rows += attacks(rng, users, train_days, total) df = pd.DataFrame(rows).sort_values("ts", kind="stable").reset_index(drop=True) df["is_attack"] = df["label"] != "normal" return df, users python 3.13....
