Engineering
Parsers & field normalisation
Raw text is not searchable knowledge. Decoders extract fields; normalisation renames them onto one shared model so a single detection works across firewalls, endpoints and cloud logs. Edit the pattern and watch extraction change live.
Active decoders
5
Draft / incomplete
1
Schema coverage
93%
fields mapped to the common model
Unmapped fields
1
parked in the custom namespace
Decoders
Select one to load it into the workbench.
Decoder workbench
Named capture groups become fields. Edit either box and the extraction re-runs.
Extraction result
matched · 7 fields extracted
- code
- 106023
- action
- DENY
- src_ip
- 198.51.100.23
- src_port
- 44120
- dst_ip
- 10.30.1.5
- dst_port
- 443
- proto
- TCP
Try it: delete the proto= group from the firewall pattern and watch the field disappear. Any detection that filtered on transport protocol would silently stop matching — this is how a “working” pipeline produces false negatives after a format change.
Common schema mapping
One field name per concept, no matter which source produced it.
| Common field | Type | Contributing sources | Mapped | Note |
|---|---|---|---|---|
| event.code | keyword | windows-security, edge-firewall | mapped | Numeric identifier for the source event type. |
| host.name | keyword | all | mapped | Canonical asset name from the inventory join. |
| user.name | keyword | windows-security, linux-auth | mapped | Lower-cased, domain stripped into user.domain. |
| source.ip | ip | all network feeds | mapped | Rejects non-routable placeholders such as 0.0.0.0. |
| destination.port | long | edge-firewall, web-access-combined | mapped | Coerced from string to numeric at normalisation. |
| process.command_line | text | windows-security | mapped | Kept as analysed text for substring hunting. |
| dns.question.name | keyword | dns-resolver | mapped | Lower-cased; registered-domain enrichment added. |
| physical.door.id | keyword | badge-reader | unmapped | No common-schema home yet — parked in the custom namespace. |
| file.hash.sha256 | keyword | integrity monitoring | mapped | Feeds indicator matching at enrichment. |
Why normalisation is the highest-leverage engineering workconcept›
Without a shared model you write the same detection five times, once per source, and maintain five sets of exclusions. With one, a rule on source.ip works across every network feed you will ever onboard — including the one you have not bought yet. This is also what makes analyst skills portable between platforms: the field names differ, the concepts do not.
Normalisation rules that prevent painful bugs›
- Store timestamps in UTC and keep the original offset in a separate field.
- Coerce ports and counts to numbers so range queries and aggregations work.
- Lower-case identity and hostname values, and keep the raw original alongside.
- Keep the unmodified raw event. Every parsing mistake is recoverable if the original survives.
- Version your decoders and backtest the change — a parser edit is a detection change in disguise.
Learn this capability
Hands-on labs that build the technique behind this workspace. Skills transfer to any mainstream SIEM.
Intermediate · Guided Lab · 35 min
Parser lab: custom firewall format
A new firewall model sends a proprietary log format. Until you parse it, none of its data is searchable or detectable.
Intermediate · Guided Lab · 30 min
Normalisation and field mapping
Parsed is not enough. Until the firewall's 'src' maps to source.ip, analysts cannot pivot across sources with one query.
Intermediate · Simulation · 25 min
Diagnosing a silent log source
The domain controller source has sent no events for two hours. Detection coverage for authentication is effectively off.