Engineering
Ingestion pipeline
Every SIEM, whatever the badge on it, moves data through the same six stages. Understand them once and you can operate, debug and capacity-plan any platform. Click a stage to see what it does and what the event looks like as it leaves.
Ingest rate
14.2k EPS
End-to-end latency
555 ms
collection to searchable
Parse fallthrough
0.9%
events hitting the generic decoder
Daily volume
620 GB
hot tier 71% used
Collection
Agents, syslog receivers, cloud pull
One collector is disconnected; two feeds are buffering locally.
Event as it leaves this stage
%ACME-FW-106023: DENY src=198.51.100.23:44120 dst=10.30.1.5:443 proto=TCP
Raw at collection
What an engineer tunes here
- Agent groups
- Syslog listeners
- Cloud connectors
- Local buffer size
Throughput
14.2k EPS
Added latency
40 ms
Health
Degraded
Where data silently disappearsfailure modes›
- Collector down: obvious once you look, invisible if nobody monitors event counts per source.
- Decoder mismatch after a vendor format change: events arrive but land as unparsed text, so field queries return nothing.
- Timezone drift at normalisation: events exist but sort into the wrong hour and vanish from your time window.
- Queue backpressure: everything is “working”, just forty minutes late — fatal during a live incident.
- Retention rollover: the hunt window you assumed was 90 days quietly became 30.
Capacity arithmetic you should be able to do out loud›
Average event size × events per second × 86,400 gives raw daily volume. Multiply by your retention days and your replication factor to get storage. At 620 GB/day with 14 days hot, that is roughly 8.7 TB hot before replication — which is why filtering low-value events at collection is the cheapest performance work available.
Filtering without losing evidence›
Drop volume at the edge only where you can defend the decision: verbose debug channels, health checks, duplicate feeds. Never drop authentication, process creation, or anything referenced by an existing detection. When in doubt, route it to a cheaper tier rather than deleting it.
Learn this capability
Hands-on labs that build the technique behind this workspace. Skills transfer to any mainstream SIEM.
Intermediate · Theory · 18 min
SIEM architecture overview
You keep the SIEM healthy. Everything analysts do depends on data arriving, parsing correctly and being searchable.
Intermediate · Simulation · 25 min
Diagnosing a silent log source
The domain controller source has sent no events for two hours. Detection coverage for authentication is effectively off.
Intermediate · Guided Lab · 35 min
Parser lab: custom firewall format
A new firewall model sends a proprietary log format. Until you parse it, none of its data is searchable or detectable.