Threat detection (SIEM)

Collecting security logs from many systems in one place and running detection rules over them, so that suspicious activity becomes an alert an analyst can act on.

2 technologies on this page ↓

Live modeldata or requestresponsebusy / idle instance

What you’re watchingThree log sources feed a collector, which parses each line and sends a copy to storage and to the rule engine. Almost every event (green) matches no rule and is only kept for later searches. The rare suspicious one (red) matches a detection rule and becomes an alert for an analyst.

How it works

Agents and connectors ship security-relevant logs to one place: cloud audit trails such as AWS CloudTrail, Windows event logs, sign-ins from an identity provider, firewall and DNS logs. A parser turns each line into named fields, and a rule engine runs detection rules over the stream. The simplest rule matches a single event: a CloudTrail call made with the AWS root user, or whoami.exe started with /all. A correlation rule looks across a time window, grouped by a field such as user or host: 10 or more failed logons by one user within 10 minutes, followed by a successful one. Rules run either on each event as it arrives or as a scheduled query over a lookback window; Microsoft Sentinel’s scheduled rules, for example, run every 5 minutes to every 14 days. A match becomes an alert with a severity and tags, such as the MITRE ATT&CK technique it points to, and goes to an analyst to triage, while the raw logs stay stored for investigations and hunts. Each SIEM has its own query language (KQL in Sentinel, YARA-L in Google Security Operations, SPL in Splunk), so a rule written for one doesn’t run on another. Shared rule formats fix that. A Sigma rule is written once in YAML against generic field names and a log source such as product: windows, category: process_creation; pySigma backends then convert it into each SIEM’s query, with processing pipelines mapping the field names, or an engine such as RSigma evaluates it directly on the event stream.

Think of it as

A security desk in a large building. Cameras and badge readers on every floor feed one wall of screens. The guard keeps a short list of things to watch for (“the same badge failing at a door ten times, then opening it”, “anyone in the server room at 3 a.m.”) and calls for help only when one of them happens. A shared rulebook lets guards in other buildings, with other camera brands, watch for the same things.

Reach for it when

Attacks show up only in logs: credentials used from an unusual place, root or admin accounts used directly, a system process started from an odd directory
Activity is spread across many systems, so one event looks harmless and only the sequence means something, such as many failed logins and then a success
Regulations such as PCI DSS or HIPAA expect security events to be logged and reviewed centrally
You run more than one SIEM, or may switch: keep rules in Sigma and convert them instead of maintaining a copy per query language

Think twice when

Collecting every log with no rules or owner: a SIEM only finds what its rules look for, and alerts nobody reads protect nobody
Treating a managed detection service as a SIEM: Amazon GuardDuty analyzes CloudTrail, VPC flow and DNS logs with AWS’s threat intelligence and ML models, then discards those logs, so they aren’t kept for your own searches
Assuming a converted rule behaves the same everywhere: field names and log sources differ between data sets, so conversion needs a pipeline that maps them, and some Sigma modifiers, such as re for regular expressions, only work where the backend supports them

A real example

A company forwards the Windows Security event log from its servers to its SIEM and keeps its detections in Git as Sigma YAML, following the example in the Sigma correlation specification. Rule failed_login matches event ID 4625 (529 on old systems); rule successful_login matches 4624 (or 528). An event_count correlation counts failed_login events per User and fires at 10 or more within 10 minutes. A temporal_ordered correlation, at level high, fires when that is followed by a successful login for the same user in the same 10-minute window: a password guessed by brute force. CI converts them with sigma convert into the SIEM’s query language. Ordinary logons match nothing. One account with 14 failures and then a success raises a single high-severity alert, which an analyst picks up, while the individual failed logons stay in storage for the investigation.

Technologies

Specific products, and what each does differently.

RSigmaOpen sourceA Rust toolkit that parses Sigma rules, compiles them into matchers and evaluates them against log events in real time, correlations included, as one self-contained binary.

SigmaOpen sourceAn open YAML format for detection rules over log events, with a community repository of more than 3,000 rules and tools that convert them into each SIEM’s query language.


RSigma	Open source	Tiger Data (formerly Timescale)	MIT	2026-10-12
Sigma	Open source	SigmaHQ	DRL 1.1 (rules); LGPL 2.1 (pySigma, sigma-cli); specification public domain	2026-10-12

What each provider calls it

AWS: Amazon GuardDuty (managed threat detection over CloudTrail management events, VPC flow logs and DNS logs; Extended Threat Detection for multi-stage attack sequences); AWS Security Hub CSPM gathers findings from GuardDuty, Inspector and Macie; Amazon Security Lake centralizes security logs in your own S3 buckets as Parquet in the OCSF schema
Google Cloud: Google Security Operations (Google SecOps, formerly Chronicle): YARA-L 2.0 rules over normalized UDM events, single-event or multi-event with a `match` time window; curated detections
Azure: Microsoft Sentinel: scheduled analytics rules, each a KQL query run every 5 minutes to 14 days over a lookback window, with an alert threshold; rule templates from Microsoft and solution vendors
Open source & others: Sigma rules (converted with pySigma and sigma-cli, or evaluated by RSigma); Wazuh (3,000+ built-in rules and decoders); Falco (YAML rules over Linux syscalls, a CNCF graduated project); source-available: Elastic Security’s detection-rules repo (Elastic License v2)

One idea, many rule languages#

Every SIEM runs detection rules over stored or streaming events, but each writes them in its own language. Sigma sits across them as a shared format.

Platform	Rules are written in	Runs over	Ready-made rules
Sigma	YAML: `logsource`, `detection`, `condition`; separate correlation rules with `group-by` and `timespan`	Any log source, after conversion to a SIEM query or in an engine such as RSigma	The SigmaHQ repository: more than 3,000 rules under the Detection Rule License 1.1
Microsoft Sentinel	KQL, in scheduled analytics rules that run every 5 minutes to 14 days	Data collected into the Sentinel workspace, over a lookback window of 5 minutes to 14 days	Templates written by Microsoft or by the vendor of a solution
Google Security Operations	YARA-L 2.0: `events`, an optional `match` window such as `$user over 5m`, and `condition`	Events normalized into the Unified Data Model (UDM)	Curated detections
Elastic Security	One TOML file per rule, with the query (mostly KQL when the repo opened in 2020) and metadata such as the scheduling interval	Data that follows the Elastic Common Schema (ECS)	elastic/detection-rules, under the Elastic License v2
Wazuh	XML rules and decoders	Logs that its decoders have parsed	More than 3,000 rules and decoders out of the box
Falco	YAML rules files	Linux syscalls, plus other sources such as CloudTrail through plugins	A default rule set

pySigma backends exist for many of these targets, among them Splunk SPL, Elasticsearch (Lucene, EQL), Kusto Query Language for Sentinel and Microsoft Defender XDR, and Google SecOps UDM search and YARA-L 2.0 (the last in beta).

Detection as code#

Rules are small text files, so teams keep them in Git and review changes like code. Elastic said in 2020 that it had moved its rules to a repository with automated tests that validate query syntax, schema use and metadata on every contribution. Sigma’s own site pitches the same workflow: version control over detections and deployment from a CI/CD pipeline straight into the SIEM.

There are two ways to run a shared rule. Conversion turns it into the target’s own query: sigma convert -t splunk -p sysmon rules/ uses a backend for the query language and a processing pipeline that maps Sigma’s generic field names, such as CommandLine, to the names in your data. Direct evaluation skips the SIEM query language: an engine such as RSigma compiles the rules into matchers, applies the same kind of pipeline and checks each event as it arrives, keeping correlation windows in memory.

Tests close the loop. A rule should fire on a known-bad sample and stay quiet on a day of normal traffic. RSigma’s rule backtest replays event files against per-rule expectations and writes a JUnit report for CI, and its linter checks rules against the Sigma specification before they ship.

In code · yaml

# Sigma correlation example from the specification (trimmed): many failed logins, then a success
title: Correlation - Multiple Failed Logins Followed by Successful Login
correlation:
    type: temporal_ordered
    rules:
        - multiple_failed_login
        - successful_login
    group-by:
        - User
    timespan: 10m
level: high
---
title: Multiple failed logons
name: multiple_failed_login
correlation:
    type: event_count
    rules:
        - failed_login
    group-by:
        - User
    timespan: 10m
    condition:
        gte: 10
---
title: Single failed login
name: failed_login
logsource:
    product: windows
    service: security
detection:
    selection:
        EventID:
            - 529
            - 4625
    condition: selection
---
title: Successful login
name: successful_login
logsource:
    product: windows
    service: security
detection:
    selection:
        EventID:
            - 528
            - 4624
    condition: selection

Often confused with

Logs Alerting Incident management Common Vulnerabilities and Exposures (CVE)Structured logging

Where it came from2005–2026

Security log tools were first sold as two kinds of product: security information management (SIM), for long-term storage, analysis and reporting of log data, and security event management (SEM), for real-time monitoring, correlation of events and notifications. In 2005, Gartner analysts Mark Nicolett and Amrit Williams named the combination security information and event management (SIEM). NIST later defined a SIEM tool as an application that gathers security data from information system components and presents it as actionable information through a single interface.

For years the detection logic stayed locked inside each product, in its own query language. Florian Roth and Thomas Patzke felt that pain while handling incidents: good searches could not be shared with teams on other SIEMs. Roth committed the first Sigma rules on December 24, 2016, the two refined the format for two to three months, and Sigma was released in 2017 as a generic YAML signature format for log events, with a converter to SIEM queries. Its README puts it this way: “Sigma is for log files what Snort is for network traffic and YARA is for files.”

Detection as code followed. In June 2020, Elastic opened its detection rules to the public as TOML files with automated tests. Sigma’s specification 2.0 (August 2024) replaced the old aggregation expressions with correlation rules, and the sigmac converter was retired in favor of pySigma. Cloud providers built their own detection services and rule languages, such as YARA-L in Google Security Operations and KQL analytics rules in Microsoft Sentinel, while open engines such as RSigma (2026) evaluate Sigma rules directly on a stream.

Why it’s built this way

Generic log sources and field names, mapped later: A Sigma rule names a log source such as category: process_creation instead of Sysmon, Windows Security auditing and every other product-specific source. The specification calls this abstraction the advantage: the rule isn’t tied to one telemetry source, and converters map it to indexes and fields.
Simple searches and a few common correlations, not a full query language: Sigma’s design philosophy trades expressiveness for portability: basic searches and a few common correlations (such as counts) cover more than 90% of day-to-day detection needs, and keeping to them means a detection can be shared across all sorts of security environments.
Correlations as separate rules that reference other rules: A correlation names the rules it counts or orders, so correlations can be chained, and single matches can stay low or informational while the correlation raises the level.
Rules as plain files: Sigma chose YAML because it is easy to read and write; files in Git get version control, review and deployment through CI/CD, the same path Elastic took for its rules in 2020.

Timeline

2005
Gartner analysts Mark Nicolett and Amrit Williams introduce the term SIEM
2016
Florian Roth commits the first Sigma rules on December 24
2017
Sigma is released; MISP adds native Sigma support in version 2.4.70 (March)
2020
Elastic opens its public detection-rules repository on June 30
2024
Sigma specification 2.0 adds correlation rules; sigmac reaches end of life on August 1
2025
Sigma specification 2.1 adds value sum, average and percentile correlations
2026
RSigma ships its first feature-complete release on February 17

Sources27

Improve this page on GitHub