Improve XML validation with custom XML entities by vicferpoy · Pull Request #4783 · wazuh/wazuh

vicferpoy · 2020-03-25T12:51:13Z

Hello team, this closes #4767 .

We found that our XML validator was failing with directories such as:

<directories>c:\</directories>

This was due to the function that replaces XML entities before trying to validate the file, replacing every \< for <, resulting in << for the case above.

The solution was to add the character \ as a new XML entity. In addition to that, I refactored the code to include more custom XML entities in the future (if needed) using a simple python dict.

Before the fix

root@0b8daa62b61d:/# curl -u foo:bar -k -X POST -H 'Content-type: application/xml' -d @copy.conf "https://127.0.0.1:55000/manager/files?path=etc/ossec.conf&pretty&overwrite=true"
{
   "error": 1113,
   "message": "XML syntax error"
}

After the fix

root@0b8daa62b61d:/# curl -u foo:bar -k -X POST -H 'Content-type: application/xml' -d @copy.conf "https://127.0.0.1:55000/manager/files?path=etc/ossec.conf&pretty&overwrite=true"
{
   "error": 0,
   "data": "File updated successfully"
}

Regards.

wazuhci · 2020-03-30T09:24:20Z

The stress test has been launched. The charts will be posted when they are ready.
I will use the following configuration:

General

Commit: 32706c2
Packages:

Stress test parameters:

Manager system: centos
Launch CentOS agent: true
Launch Ubuntu agent: true
Launch Windows agent: true
Test duration: 60
Scan frequency: 10
Modules to test:
- logcollector: true
- syscheck: true
- rootcheck: true
- sca: true
- osquery: true
- syscollector: true
- active-response: true
- azure-logs: true
- cis-cat: true
- docker-listener: true
- open-scap: true
- vulnerability-detector: false
FIM special test: None
FIM sleep: 300
FIM max files: 25000
SCA sleep: 30
SCA max registers: 5000
Docker image name: hello-world
Docker sleep: 5
Docker jobs: 10
Logcollector special test: None
Logcollector max files: 50000
Logcollector creation sleep: 300
Logcollector write files: 1000
Logcollector write sleep: 10
Vuln sleep: 60
Authd port: 1515
Syscollector package: imagemagick.app
Syscollector package version: 7.0.8.48

Regards!.

wazuhci · 2020-03-30T09:32:36Z

Analysis_dynamic_valgrind logcollector analysis:

Commit: 32706c2a733134ab5e0437dd3a32cd95f13931db
Parameters: track-origins=yes leak-check=full keep-stacktraces=alloc-and-free track-fds=yes num-callers=20 show-leak-kinds=definite,indirect
Agent config: ossec.conf
Agent logs: ossec.log
Coverage:

Lines executed:31.40% of 1860

Coverage report detail: coverage-report-summary
Valgrind result: Analysis_dynamic_valgrind-v3.12.0-B907.log

wazuhci · 2020-03-30T09:36:06Z

Analysis_dynamic_valgrind syscheckd analysis:

Commit: 32706c2a733134ab5e0437dd3a32cd95f13931db
Parameters: track-origins=yes leak-check=full keep-stacktraces=alloc-and-free track-fds=yes num-callers=20 show-leak-kinds=definite,indirect
Agent config: ossec.conf
Agent logs: ossec.log
Coverage:

Lines executed:26.50% of 2996

Coverage report detail: coverage-report-summary
Valgrind result: Analysis_dynamic_valgrind-v3.12.0-B908.log

wazuhci · 2020-03-30T09:44:16Z

The stress test has been launched. The charts will be posted when they are ready.
I will use the following configuration:

General

Stress test parameters:

Manager system: centos
Launch CentOS agent: true
Launch Ubuntu agent: true
Launch Windows agent: true
Test duration: 60
Scan frequency: 10
Modules to test:
- logcollector: true
- syscheck: true
- rootcheck: true
- sca: true
- osquery: true
- syscollector: true
- active-response: true
- azure-logs: true
- cis-cat: true
- docker-listener: true
- open-scap: true
- vulnerability-detector: false
FIM special test: None
FIM sleep: 300
FIM max files: 25000
SCA sleep: 30
SCA max registers: 5000
Docker image name: hello-world
Docker sleep: 5
Docker jobs: 10
Logcollector special test: None
Logcollector max files: 50000
Logcollector creation sleep: 300
Logcollector write files: 1000
Logcollector write sleep: 10
Vuln sleep: 60
Authd port: 1515
Syscollector package: imagemagick.app
Syscollector package version: 7.0.8.48

Regards!.

wazuhci · 2020-03-30T09:51:07Z

Analysis_dynamic_valgrind logcollector analysis:

Commit: 7c9da98
Parameters: track-origins=yes leak-check=full keep-stacktraces=alloc-and-free track-fds=yes num-callers=20 show-leak-kinds=definite,indirect
Agent config: ossec.conf
Agent logs: ossec.log
Coverage:

Lines executed:31.40% of 1860

Coverage report detail: coverage-report-summary
Valgrind result: Analysis_dynamic_valgrind-v3.12.0-B915.log

wazuhci · 2020-03-30T09:56:34Z

Analysis_dynamic_valgrind syscheckd analysis:

Commit: 7c9da98
Parameters: track-origins=yes leak-check=full keep-stacktraces=alloc-and-free track-fds=yes num-callers=20 show-leak-kinds=definite,indirect
Agent config: ossec.conf
Agent logs: ossec.log
Coverage:

Lines executed:26.50% of 2996

Coverage report detail: coverage-report-summary
Valgrind result: Analysis_dynamic_valgrind-v3.12.0-B916.log

wazuhci · 2020-03-30T10:37:59Z

The stress test has been launched. The charts will be posted when they are ready.
I will use the following configuration:

General

Commit: d6931a0
Packages:
- [manager 404: Not Found 0.commitd6931a0 rpm](https://packages-dev.wazuh.com/warehouse/pullrequests/404: Not Found.null/rpm/var/wazuh-manager-404: Not Found-0.commitd6931a0.x86_64.rpm)
- [agent 404: Not Found 0.commitd6931a0 rpm](https://packages-dev.wazuh.com/warehouse/pullrequests/404: Not Found.null/rpm/var/wazuh-agent-404: Not Found-0.commitd6931a0.x86_64.rpm)
- [manager 404: Not Found 0.commitd6931a0 deb](https://packages-dev.wazuh.com/warehouse/pullrequests/404: Not Found.null/deb/var/wazuh-manager_404: Not Found-0.commitd6931a0_amd64.deb)
- [agent 404: Not Found 0.commitd6931a0 deb](https://packages-dev.wazuh.com/warehouse/pullrequests/404: Not Found.null/deb/var/wazuh-agent_404: Not Found-0.commitd6931a0_amd64.deb)
- [agent 404: Not Found 0.commitd6931a0 msi](https://packages-dev.wazuh.com/warehouse/pullrequests/404: Not Found.null/windows/wazuh-agent-404: Not Found-0.commitd6931a0.msi)

Stress test parameters:

Manager system: centos
Launch CentOS agent: true
Launch Ubuntu agent: true
Launch Windows agent: true
Test duration: 60
Scan frequency: 10
Modules to test:
- logcollector: true
- syscheck: true
- rootcheck: true
- sca: true
- osquery: true
- syscollector: true
- active-response: true
- azure-logs: true
- cis-cat: true
- docker-listener: true
- open-scap: true
- vulnerability-detector: false
FIM special test: None
FIM sleep: 300
FIM max files: 25000
SCA sleep: 30
SCA max registers: 5000
Docker image name: hello-world
Docker sleep: 5
Docker jobs: 10
Logcollector special test: None
Logcollector max files: 50000
Logcollector creation sleep: 300
Logcollector write files: 1000
Logcollector write sleep: 10
Vuln sleep: 60
Authd port: 1515
Syscollector package: imagemagick.app
Syscollector package version: 7.0.8.48

Regards!.

Improve XML validation with custom XML entities

fd84def

vicferpoy requested a review from crd1985 March 25, 2020 12:51

vicferpoy self-assigned this Mar 25, 2020

Add fix to CHANGELOG.md

f6c72f7

Add '--' replacement in comments for XML validation

7c9da98

vicferpoy force-pushed the dev-improve-xml-validator branch from 32706c2 to 7c9da98 Compare March 30, 2020 09:41

crd1985 approved these changes Mar 30, 2020

View reviewed changes

Merge branch '3.12' into dev-improve-xml-validator

d6931a0

crd1985 merged commit b72473f into 3.12 Mar 30, 2020

crd1985 deleted the dev-improve-xml-validator branch March 30, 2020 10:37

vicferpoy mentioned this pull request Apr 23, 2020

XML validation fails when using backslash on agent.conf #4947

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve XML validation with custom XML entities#4783

Improve XML validation with custom XML entities#4783
crd1985 merged 4 commits into3.12from
dev-improve-xml-validator

vicferpoy commented Mar 25, 2020

Uh oh!

wazuhci commented Mar 30, 2020

Uh oh!

wazuhci commented Mar 30, 2020

Uh oh!

wazuhci commented Mar 30, 2020

Uh oh!

wazuhci commented Mar 30, 2020

Uh oh!

wazuhci commented Mar 30, 2020

Uh oh!

wazuhci commented Mar 30, 2020

Uh oh!

wazuhci commented Mar 30, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

vicferpoy commented Mar 25, 2020

Before the fix

After the fix

Uh oh!

wazuhci commented Mar 30, 2020

General

Stress test parameters:

Uh oh!

wazuhci commented Mar 30, 2020

Analysis_dynamic_valgrind logcollector analysis:

Uh oh!

wazuhci commented Mar 30, 2020

Analysis_dynamic_valgrind syscheckd analysis:

Uh oh!

wazuhci commented Mar 30, 2020

General

Stress test parameters:

Uh oh!

wazuhci commented Mar 30, 2020

Analysis_dynamic_valgrind logcollector analysis:

Uh oh!

wazuhci commented Mar 30, 2020

Analysis_dynamic_valgrind syscheckd analysis:

Uh oh!

wazuhci commented Mar 30, 2020

General

Stress test parameters:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants