Skip to content

Add maximum files per second option#6830

Merged
vikman90 merged 11 commits intomasterfrom
5645-add-limit-files-scanned-per-sec
Feb 15, 2021
Merged

Add maximum files per second option#6830
vikman90 merged 11 commits intomasterfrom
5645-add-limit-files-scanned-per-sec

Conversation

@antoniomanuelfr
Copy link

@antoniomanuelfr antoniomanuelfr commented Dec 4, 2020

Related issue
#5645

Description

This PR aims to add a new option for FIM.
With this option, a limit can be set for how many files per second are read.
If this limit is reached, the scan will be stoped for 1 second and it will continue as normal.

Closes #5645

Configuration options

<syscheck>
    <disabled>no</disabled>

    <!-- Directories to check  (perform all possible verifications) -->
    <directories>/test</directories>
    <max_files_per_second>75</max_files_per_second

Logs/Alerts example

2020/12/04 13:06:40 ossec-syscheckd[23270] create_db.c:62 at fim_scan(): INFO: (6008): File integrity monitoring scan started.
2020/12/04 13:06:40 ossec-syscheckd[23270] run_check.c:102 at send_syscheck_msg(): DEBUG: (6321): Sending FIM event: {"type":"scan_start","data":{"timestamp":1607087200}}
2020/12/04 13:06:40 ossec-syscheckd[23270] create_db.c:68 at fim_scan(): DEBUG: (6348): Size of '/var/ossec/queue/diff' folder: 0.00000 KB.
2020/12/04 13:06:40 ossec-syscheckd[23270] run_check.c:149 at check_max_fps(): DEBUG: (6258): Maximum number of files read per second reached, sleeping.

Tests

  • Compilation without warnings in every supported platform
    • Linux
    • Windows
    • MAC OS X
  • Source installation
  • Source upgrade
  • Review logs syntax and correct language
  • Memory tests for Linux
    • Scan-build report
    • Coverity
    • Valgrind (memcheck and descriptor leaks check)
  • Memory tests for Windows
    • Scan-build report
  • Memory tests for macOS
    • Scan-build report
    • Leaks
    • AddressSanitizer

@antoniomanuelfr antoniomanuelfr added the module/fim File Integrity Monitoring label Dec 4, 2020
@antoniomanuelfr antoniomanuelfr self-assigned this Dec 4, 2020
@antoniomanuelfr antoniomanuelfr marked this pull request as ready for review December 4, 2020 15:03
Copy link
Contributor

@Molter73 Molter73 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to add max_fps to the configuration on demand.

@antoniomanuelfr antoniomanuelfr force-pushed the 5645-add-limit-files-scanned-per-sec branch 3 times, most recently from 9c9c90c to 51ab6bc Compare December 11, 2020 07:31
Copy link
Contributor

@Molter73 Molter73 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While testing this PR I realized that the change is also affecting the initial scan which could delay the start of whodata and realtime mode operations. I suggest adding a parameter to check_max_fps in order to skip the check on the initial scan, potentially we could add a configuration option so this can switched on or off by the user.

I also realized that the limit is not being applied to Windows registry entries, we could add it in a future PR.

Copy link
Contributor

@Molter73 Molter73 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@antoniomanuelfr antoniomanuelfr force-pushed the 5645-add-limit-files-scanned-per-sec branch from b650aaa to 023d030 Compare December 15, 2020 10:40
@antoniomanuelfr antoniomanuelfr force-pushed the 5645-add-limit-files-scanned-per-sec branch 2 times, most recently from e42f88a to b3e6c73 Compare December 24, 2020 08:33
@antoniomanuelfr antoniomanuelfr force-pushed the 5645-add-limit-files-scanned-per-sec branch from b3e6c73 to 920acd5 Compare January 18, 2021 11:05
@antoniomanuelfr
Copy link
Author

antoniomanuelfr commented Jan 20, 2021

This is the integration test report in a Ubuntu VM. The test max_fps is failing due to a typo that has been fixed in this commit. After the typo has been corrected, the test passed.
report_max_fps.zip

These are the integration test in jenkins:
agent_centos_html_report_test_integration_B1017_20210119082045.zip
agent_ubuntu_html_report_test_integration_B1017_20210119082045.zip
agent_windows_html_report_test_integration_B1017_20210119082045.zip

@antoniomanuelfr antoniomanuelfr marked this pull request as draft January 25, 2021 11:55
@antoniomanuelfr antoniomanuelfr marked this pull request as ready for review January 25, 2021 12:35
@antoniomanuelfr antoniomanuelfr marked this pull request as draft February 1, 2021 09:26
@antoniomanuelfr antoniomanuelfr marked this pull request as ready for review February 1, 2021 11:50
@antoniomanuelfr antoniomanuelfr force-pushed the 5645-add-limit-files-scanned-per-sec branch from ce16159 to 356347d Compare February 1, 2021 12:19
Copy link
Contributor

@Molter73 Molter73 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@antoniomanuelfr antoniomanuelfr force-pushed the 5645-add-limit-files-scanned-per-sec branch from 713a47a to 2dedae5 Compare February 4, 2021 11:50
@vikman90 vikman90 merged commit 4d1f04f into master Feb 15, 2021
@vikman90 vikman90 deleted the 5645-add-limit-files-scanned-per-sec branch February 15, 2021 10:15
@antoniomanuelfr antoniomanuelfr changed the title Add maximum files per second option for FIM scan Add maximum files per second option Jun 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module/fim File Integrity Monitoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a configurable limit to the amount of files scanned per second

3 participants