Skip to content

[beelzebub] Improve log event handling #14402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

[beelzebub] Improve log event handling #14402

wants to merge 14 commits into from

Conversation

colin-stubbs
Copy link
Contributor

@colin-stubbs colin-stubbs commented Jul 2, 2025 •
edited by efd6
Loading

Proposed commit message

[beelzebub] Improve log event handling

Beelzebub events now include both event.Headers and event.HeadersMap for
HTTP headers, this PR ensures support for both and minimises ingest
processor utilisation when both exist. 

To maintain backwards compatibility with existing index mappings
HeadersMap is now used in preference to avoid gsup/kvp ingest processor
operation, and is renamed to event.Headers. The original event.Headers
string is stored in event.HeadersText though it will be removed unless
events are tagged with preserve_duplicate_custom_fields.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • Testing with latest Beelzebub release.
  • Testing via elastic-package with 8.18.2 and 9.0.3 stacks

How to test this PR locally

Test and review diff.

Related issues

Screenshots

Not applicable.

colin-stubbs and others added 9 commits February 27, 2025 10:15
@colin-stubbs
initial commit
7eb3d31
@colin-stubbs
update
7fab573 
add password redaction, remove S3 until properly tested, improve ECS mapping and field cleanups
@colin-stubbs
update dashboard to use ECS fields
2fc3e0b 
user.name etc now mapped
@colin-stubbs
Fix PR link
b9aa1ca 
Swap default to PR that now exists
@colin-stubbs
update batch based on code review by @kcreddy
2a236ab 
Addresses review feedback for the Beelzebub package. This includes:
- Adds the package to CODEOWNERS
- Updates documentation to include Logstash as an alternative option to fluentd for shipping logs, removes fluentd configuration example
- Appends relevant values to related.ip and related.user
- Bumps version to 0.1.0
- Full package re-test post build via elastic-package test
@colin-stubbs @kcreddy
Apply suggestions from code review
886380f 
Co-authored-by: Krishna Chaitanya Reddy Burri <[email protected]>
@colin-stubbs
update README
153312b
@colin-stubbs
Merge remote-tracking branch 'origin/main' into beelzebub
cdeb280
@colin-stubbs
support newer log format with HeadersMap, add Cookies field def
921e43a
@colin-stubbs colin-stubbs requested a review from a team as a code owner July 2, 2025 13:03
@andrewkroh andrewkroh added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:beelzebub Beelzebub (Community supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Jul 2, 2025
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@andrewkroh andrewkroh added the enhancement New feature or request label Jul 2, 2025
@efd6 efd6 changed the title Enhancement: [beelzebub] Improve log event handling [beelzebub] Improve log event handling Jul 3, 2025
efd6
efd6 reviewed Jul 3, 2025
View reviewed changes
@efd6
Copy link
Contributor

efd6 commented Jul 3, 2025

/test

@efd6
Copy link
Contributor

efd6 commented Jul 3, 2025

/test

@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
96.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@elasticmachine
Copy link

💚 Build Succeeded

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@efd6 efd6 efd6 left review comments

@mariocandela mariocandela mariocandela approved these changes

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:beelzebub Beelzebub (Community supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[beelzebub]: Improve log format handling
5 participants
@colin-stubbs @elasticmachine @efd6 @mariocandela @andrewkroh