SOC 2.0

What this is

Every large organisation runs software that watches its computers and raises an alarm when something looks like an attack. The category is called a SIEM. The alarms come from detection rules — written descriptions of suspicious behaviour, like someone just ran a command that dumps saved passwords out of memory.

I wanted to learn how to write those rules well. Reading them is not the same as running them, so I built the system that runs them, and then I attacked it.

The problem with a system nobody attacks

For months my setup was a closed loop. I wrote a rule, I wrote a test for it, the test passed. That loop is comfortable and it teaches you almost nothing, because the only events the system ever saw were events I had written myself. Of course they matched — I wrote both sides.

The interesting question is not whether a rule catches the attack you imagined. It is what the rule does when it meets a real computer doing ordinary work.

Constraints

One machine, with 23.3 GB of memory, running the monitoring system in containers plus three virtual machines. All of it does not fit at once, so the virtual machines take turns and the heavier parts start only when needed.

There are also things this setup simply cannot do. Watching raw network traffic needs hardware access my host does not give me — so that component sits in the configuration and has never processed a single packet. I would rather say that plainly than imply coverage I do not have.

What I built

Ten services behind an encrypted front door: a web API, a database, a search engine for storing events, a pipeline that collects and reshapes incoming logs, and a dashboard. On top of that:

• A detection engine running 618 rules — 600 published by the open-source security community, 18 written by me.
• A correlation engine, which is the part that matters most. A single suspicious event is usually nothing. Two related suspicious events in the right order are an attack. This watches for the sequence.
• Reputation lookups, which ask outside services whether an internet address is known to be dangerous. This is the part that went wrong, and it has its own writeup on the homepage.

Then I stopped feeding it myself and pointed real machines at it:

• An Ubuntu machine, reporting everything its users ran.
• A Windows machine, reporting the same, plus a tool that safely performs real attack techniques so I could see whether my rules noticed.
• A Kali machine — not monitored. The attacker.

What broke

Everything interesting happened in the first few hours of real data, and none of it had been caught by a test suite that was passing 100%.

The Linux side could not read its own logs. Ninety real events arrived, and the part of the pipeline that reshapes incoming logs did not understand the format they were in — so it filed them without the one field every Linux rule depends on. Every Linux rule was structurally incapable of firing. The rules were fine. The plumbing in front of them was not, and nothing reported an error.

One route had never worked at all. It tried to write to two database columns that do not exist, so every Windows login and antivirus event was dropped with an error. The tests passed throughout, because they exercised a different path.

Events were dropped after the system said “OK”. The pipeline sent a field in one shape, storage expected another, storage rejected the record, the pipeline threw it away — and the API had already told the sender everything was fine. Silent data loss behind a success message is the worst failure mode there is.

Long commands were rejected instead of shortened. Twenty-four real events were lost for being too long — and they were, of course, exactly the disguised-command events that the rule exists to catch.

My own logging drowned out the attack. Detailed Linux logging produced about 33 events a second, which hit the system’s own rate limit and crowded out the password-guessing attack happening at the same time. The real attack did not register until I turned the noise down. Collecting more is not the same as seeing more.

And the largest one: the reputation scoring produced roughly 82% false alarms within minutes of seeing real traffic. Four faults stacked on each other. That story is on the homepage, in full, with the code.

The outcome

Rules that had only ever fired on my own test data started firing on real behaviour:

• A password-guessing attack from the Kali machine, followed by commands running on the target — flagged as a single critical two-step attack, with the attacker’s address captured. Not an injected test. A real attack across a real network.
• A second two-step chain on Windows: someone looking around the machine, then attempting to pull saved credentials out of memory.
• Disguised PowerShell commands, credential-dumping tools, and ordinary-looking built-in programs used for attack purposes.
• Five community rules fired on real activity for the first time.

About eight of my rule families have now fired on real data. Eleven of my 32 rules have an automated test. Those two numbers are the honest maturity statement for this project, and I would rather publish them than a coverage percentage nobody can check.

What is still open

There is no network separation — everything sits on one flat network, so a compromise anywhere reaches everywhere. Watching raw network traffic needs hardware this host cannot provide. Roughly 24 of my rules have still never seen real data.

All of it is queued behind moving the lab onto proper virtualisation hardware, which is the next real piece of work rather than a line on a roadmap.