Skip to content

[beelzebub] Improve log event handling #14402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: main
Choose a base branch
from

Conversation

colin-stubbs
Copy link
Contributor

@colin-stubbs colin-stubbs commented Jul 2, 2025

Proposed commit message

[beelzebub] Improve log event handling

Beelzebub events now include both event.Headers and event.HeadersMap for
HTTP headers, this PR ensures support for both and minimises ingest
processor utilisation when both exist. 

To maintain backwards compatibility with existing index mappings
HeadersMap is now used in preference to avoid gsup/kvp ingest processor
operation, and is renamed to event.Headers. The original event.Headers
string is stored in event.HeadersText though it will be removed unless
events are tagged with preserve_duplicate_custom_fields.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • Testing with latest Beelzebub release.
  • Testing via elastic-package with 8.18.2 and 9.0.3 stacks

How to test this PR locally

Test and review diff.

Related issues

Screenshots

Not applicable.

colin-stubbs and others added 9 commits February 27, 2025 10:15
add password redaction, remove S3 until properly tested, improve ECS mapping and field cleanups
user.name etc now mapped
Swap default to PR that now exists
Addresses review feedback for the Beelzebub package. This includes:
- Adds the package to CODEOWNERS
- Updates documentation to include Logstash as an alternative option to fluentd for shipping logs, removes fluentd configuration example
- Appends relevant values to related.ip and related.user
- Bumps version to 0.1.0
- Full package re-test post build via elastic-package test
Co-authored-by: Krishna Chaitanya Reddy Burri <[email protected]>
@colin-stubbs colin-stubbs requested a review from a team as a code owner July 2, 2025 13:03
@andrewkroh andrewkroh added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:beelzebub Beelzebub (Community supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Jul 2, 2025
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@andrewkroh andrewkroh added the enhancement New feature or request label Jul 2, 2025
@efd6 efd6 changed the title Enhancement: [beelzebub] Improve log event handling [beelzebub] Improve log event handling Jul 3, 2025
@efd6
Copy link
Contributor

efd6 commented Jul 3, 2025

/test

@efd6
Copy link
Contributor

efd6 commented Jul 3, 2025

/test

Copy link

@elasticmachine
Copy link

💚 Build Succeeded

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:beelzebub Beelzebub (Community supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[beelzebub]: Improve log format handling
5 participants