Support for PCRE regular expressions by jnasselle · Pull Request #6480 · wazuh/wazuh

jnasselle · 2020-11-03T13:49:52Z

Related issue
205

Description

Hello team!

This PR add support for Perl Compatible Regular Expressions(PCRE) and allow to select osregex, osmatch or pcre2 using new type attribute for several options/fields

Decoders:

Option	Supported types	Default type
program_name	osregex,osmatch,pcre2	osmatch
prematch	osregex,pcre2	osregex
regex	osregex,pcre2	osregex

Rules:

Option	Supported types	Default type
regex	osregex,osmatch,pcre2	osregex
field	osregex,osmatch,pcre2	osregex
match	osregex,osmatch,pcre2	osmatch
action	osregex,osmatch,pcre2	string
extra_data	osregex,osmatch,pcre2	osmatch
hostname	osregex,osmatch,pcre2	osmatch
id	osregex,osmatch,pcre2	osmatch
location	osregex,osmatch,pcre2	osmatch
match	osregex,osmatch,pcre2	osmatch
program_name	osregex,osmatch,pcre2	osmatch
protocol	osregex,osmatch,pcre2	osmatch
user	osregex,osmatch,pcre2	osmatch
url	osregex,osmatch,pcre2	osmatch
srcport	osregex,osmatch,pcre2	osmatch
dstport	osregex,osmatch,pcre2	osmatch
status	osregex,osmatch,pcre2	osmatch
system_name	osregex,osmatch,pcre2	osmatch
extra_data	osregex,osmatch,pcre2	osmatch
srcgeoip	osregex,osmatch,pcre2	osmatch
dstgeoip	osregex,osmatch,pcre2	osmatch

`libpcre2` specs

PCRE2 10.34 used
Included and linked in Wazuh Manager and Wazuh Agent
8 bit and UTF-8 support
Static library compilation only
JIT compiler set to auto, in order to test arch support. Code don't use this feature yet
Compilation time increases 10% compared with base branch of this PR.

`ossec-analysisd` `getconfig endpoint` example

Rules

[
  {
    "sigid": 100001,
    "level": 5,
    "regex": {
      "pattern": "^Hello World.$",
      "negate": false,
      "type": "pcre2"
    }
  },
 {
    "sigid": 100002,
    "level": 7,
    "dstport": {
      "pattern": "^Hello from Wazuh.$",
      "negate": true,
      "type": "osregex"
    }
  }
]

Decoders

[{
	"id": 519,
	"name": "test_pcre2_prematch",
	"children": [{
		"id": 519,
		"name": "test_pcre2_prematch",
		"parent": "test_pcre2_prematch",
		"order": [
			"id",
			"otro"
		],
		"use_own_name": "false",
		"accumulate": "no",
		"prematch": {
			"pattern": "MAGICWORD",
			"type": "osregex"
		},
		"regex": {
			"pattern": " testing after_parent id:(\\S+) (\\w+)",
			"type": "pcre2"
		},
		"type": "syslog"
	}],
	"use_own_name": "false",
	"accumulate": "no",
	"prematch": {
		"pattern": "test_pcre(\\d)_prematch",
		"type": "osregex"
	},
	"type": "syslog"
}, {
	"id": 517,
	"name": "test_lcre2_program_name",
	"children": [],
	"use_own_name": "false",
	"accumulate": "no",
	"program_name": {
		"pattern": "^pcre2_house(cat((?i)s|)|)$",
		"type": "pcre2"
	},
	"type": "syslog"
}]

Examples

Decoders

<decoder name="test_pcre2">
  <program_name type="pcre2">^(?i)test_pcre2</program_name>
  <regex type="pcre2">(?i)\S+ (\S+) (\w+) (\S+)</regex>
  <order>url,action,querystring</order>
</decoder>

Rules

<group name="qa,test">
 <rule id="100001" level="3">
    <decoded_as>test_pcre2</decoded_as>
    <match>test_regex</match>
    <regex type="pcre2">(?i)regex_example</regex>
    <description>Testing PCRE2 regex</description>
  </rule>
  <rule id="100002" level="3">
    <decoded_as>test_pcre2</decoded_as>
    <match>test_field</match>
    <field name="querystring" type="pcre2">(?i)format=json</field>
    <description>Testing PCRE2 dynamic field</description>
  </rule>
</group>

NOTE: (?i) in PCRE syntax means to be case insensitive.

Logs/Alerts example

Lower case log

ossec-testrule: Type one log per line.

Dec 19 17:20:08 ubuntu test_pcre2[12345]:test_regex regex_example https://localhost GET format=xml


**Phase 1: Completed pre-decoding.
       full event: 'Dec 19 17:20:08 ubuntu test_pcre2[12345]:test_regex regex_example https://localhost GET format=xml'
       timestamp: 'Dec 19 17:20:08'
       hostname: 'ubuntu'
       program_name: 'test_pcre2'
       log: 'test_regex regex_example https://localhost GET format=xml'

**Phase 2: Completed decoding.
       decoder: 'test_pcre2'
       url: 'https://localhost'
       action: 'GET'
       querystring: 'format=xml'

**Phase 3: Completed filtering (rules).
       Rule id: '100001'
       Level: '3'
       Description: 'Testing PCRE2 regex'
**Alert to be generated.


Dec 19 17:20:08 ubuntu test_pcre2[12345]:test_regex regex_example https://localhost GET format=json


**Phase 1: Completed pre-decoding.
       full event: 'Dec 19 17:20:08 ubuntu test_pcre2[12345]:test_regex regex_example https://localhost GET format=json'
       timestamp: 'Dec 19 17:20:08'
       hostname: 'ubuntu'
       program_name: 'test_pcre2'
       log: 'test_regex regex_example https://localhost GET format=json'

**Phase 2: Completed decoding.
       decoder: 'test_pcre2'
       url: 'https://localhost'
       action: 'GET'
       querystring: 'format=json'

**Phase 3: Completed filtering (rules).
       Rule id: '100001'
       Level: '3'
       Description: 'Testing PCRE2 regex'
**Alert to be generated.

Upper case log

ossec-testrule: Type one log per line.

DEC 19 17:20:08 UBUNTU TEST_PCRE2[12345]:TEST_REGEX REGEX_EXAMPLE HTTPS://LOCALHOST GET FORMAT=XML


**Phase 1: Completed pre-decoding.
       full event: 'DEC 19 17:20:08 UBUNTU TEST_PCRE2[12345]:TEST_REGEX REGEX_EXAMPLE HTTPS://LOCALHOST GET FORMAT=XML'
       timestamp: 'DEC 19 17:20:08'
       hostname: 'UBUNTU'
       program_name: 'TEST_PCRE2'
       log: 'TEST_REGEX REGEX_EXAMPLE HTTPS://LOCALHOST GET FORMAT=XML'

**Phase 2: Completed decoding.
       decoder: 'test_pcre2'
       url: 'HTTPS://LOCALHOST'
       action: 'GET'
       querystring: 'FORMAT=XML'

**Phase 3: Completed filtering (rules).
       Rule id: '100001'
       Level: '3'
       Description: 'Testing PCRE2 regex'
**Alert to be generated.


DEC 19 17:20:08 UBUNTU TEST_PCRE2[12345]:TEST_REGEX REGEX_EXAMPLE HTTPS://LOCALHOST GET FORMAT=JSON


**Phase 1: Completed pre-decoding.
       full event: 'DEC 19 17:20:08 UBUNTU TEST_PCRE2[12345]:TEST_REGEX REGEX_EXAMPLE HTTPS://LOCALHOST GET FORMAT=JSON'
       timestamp: 'DEC 19 17:20:08'
       hostname: 'UBUNTU'
       program_name: 'TEST_PCRE2'
       log: 'TEST_REGEX REGEX_EXAMPLE HTTPS://LOCALHOST GET FORMAT=JSON'

**Phase 2: Completed decoding.
       decoder: 'test_pcre2'
       url: 'HTTPS://LOCALHOST'
       action: 'GET'
       querystring: 'FORMAT=JSON'

**Phase 3: Completed filtering (rules).
       Rule id: '100001'
       Level: '3'
       Description: 'Testing PCRE2 regex'
**Alert to be generated.

…uring libpcre2 compilation

… string in w_expression_match function

[X] extra_data [X] hostname [X] location [X] program_name [x] protocol [X] user [X] url [X] srcport [X] dstport [x] status [X] system_name [x] data [x] srcgeoip [x] dstgeoip

jnasselle and others added 22 commits October 20, 2020 17:10

Import the PCRE2 library into the external dependencies

c76936f

Remove --disable-dependency-tracking flag and fix of CFLAG variable d…

8bfb19d

…uring libpcre2 compilation

analysisd base support pcre2 expression

6fc24f0

Added tests for w_free_expression_t

67ba84f

Supports compiled patterns with group storage

a2b2316

Added tests for w_expression_test

404c153

Add gets regex patterns

59fd58f

merge match and test functions

733dde5

Added PCRE2 structure for unit tests

95a397a

Added tests for w_expression_match

189afb2

Free memory of regex_matching variable and add the cases for OSIP and…

01b8d7b

… string in w_expression_match function

Added tests for w_expression_PCRE2_fill_regex_match

23e2cd0

Fixed compatibility of OSRegex_Execute_ex with previous tests

f2806a4

Fix Makefile for agent and winagent compilation

eaffc9a

Added test for w_expression_PCRE2_fill_regex_match

15a30bf

Added tests for w_expression_get_regex_type

aa862dc

Added test for w_expression_compile

a88db13

Completed coverage of expression.c

47c863d

Analysisd decoder accept PCRE2 expressions

fb537fc

Allow Wazuh accept PCRE2 in rules

c20e38d

Fixed w_free_expression_t_exp_type_pcre2

b4639af

Defined regex type

0890de7

jnasselle marked this pull request as ready for review November 3, 2020 14:14

jnasselle requested review from JcabreraC, Lopuiz and juliancnn November 3, 2020 14:14

juliancnn and others added 3 commits November 4, 2020 17:43

Add support OSRegex&PCRE2 static field (in rules)

484c2eb

[X] extra_data [X] hostname [X] location [X] program_name [x] protocol [X] user [X] url [X] srcport [X] dstport [x] status [X] system_name [x] data [x] srcgeoip [x] dstgeoip

use w_get_attr_val_by_name function in rules

8405b08

Added unit test structure for os_xml

78b1346

Lopuiz removed the request for review from juliancnn November 5, 2020 16:41

Lopuiz requested review from vikman90 and removed request for JcabreraC and Lopuiz November 5, 2020 16:41

Remove some variables in Rules_OP_ReadRules function

d0b44cc

Lopuiz force-pushed the dev-205-regex-pcre2 branch from ea2d012 to d0b44cc Compare November 5, 2020 16:59

jnasselle assigned jnasselle, juliancnn, Lopuiz and JcabreraC Nov 5, 2020

jnasselle added module/analysis Issues related to the Analysis daemon feed module/building Compilation of the core product regex Issue related to regular expressions labels Nov 5, 2020

Merge branch 'master' into dev-205-regex-pcre2

067457c

vikman90 approved these changes Nov 6, 2020

View reviewed changes

vikman90 merged commit 3a3b700 into master Nov 6, 2020

vikman90 deleted the dev-205-regex-pcre2 branch November 6, 2020 18:00

juliancnn mentioned this pull request Jun 29, 2022

Wazuh-dbd does not support custom types in rules #14081

Open

chemamartinez mentioned this pull request Sep 19, 2022

Add new settings ignore and restrict to logcollector #14782

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for PCRE regular expressions#6480

Support for PCRE regular expressions#6480
vikman90 merged 27 commits intomasterfrom
dev-205-regex-pcre2

jnasselle commented Nov 3, 2020 •

edited by JcabreraC

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

jnasselle commented Nov 3, 2020 • edited by JcabreraC Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Decoders:

Rules:

libpcre2 specs

ossec-analysisd getconfig endpoint example

Examples

Decoders

Rules

Logs/Alerts example

Lower case log

Upper case log

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jnasselle commented Nov 3, 2020 •

edited by JcabreraC

Loading

`libpcre2` specs

`ossec-analysisd` `getconfig endpoint` example