Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lib: Add new APIs to support creating processor unit for input and output #9957

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

yunzvanessa
Copy link

@yunzvanessa yunzvanessa commented Feb 19, 2025

Summary

  1. Add two new library APIs to support creating processor units for input and output
  • flb_input_processor_unit(flb_ctx_t *ctx, const char *event_type, const char *processor_unit_name, int ffd, ...): create a processor unit with specific event type and processor unit name for an input plugin.
  • flb_output_processor_unit(flb_ctx_t *ctx, const char *event_type, const char *processor_unit_name, int ffd, ...): create a processor unit with specific event type and processor unit name for an output plugin.
  1. This PR also has a fix for processor which updates function signature of flb_processor_unit_create to use const char* for unit_name

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • [N/A] Example configuration file for the change:

  • Debug log output from testing the change:

# Example output log from an application that uses fluentbit as a C libary API. With the new processor unit APIs I added, I 
# enabled otel_envelop and content_modifier processor units to manipulate the log messages from the tail input plugin, 
# before sending the logs to openetelemetry output plugin

[2025/02/18 11:22:16] [debug] [input:tail:tail.0] inode=8603124, ErrorLog.txt, events: IN_ATTRIB
[2025/02/18 11:22:16] [debug] [input:tail:tail.0] inode=8603164, 8000_ErrorLog.txt, events: IN_ATTRIB
[2025/02/18 11:22:16] [debug] [input:tail:tail.0] inode=8603167, 8001_ErrorLog.txt, events: IN_ATTRIB
[2025/02/18 11:22:16] [debug] [input:tail:tail.0] inode=8594915, 8002_ErrorLog.txt, events: IN_ATTRIB
[2025/02/18 11:22:16] [debug] [input:tail:tail.0] inode=8594955, 7997_ErrorLog.txt, events: IN_ATTRIB
[2025/02/18 11:22:17] [debug] [input:tail:tail.0] inode=8603124, ErrorLog.txt, events: IN_MODIFY
[2025/02/18 11:22:17] [debug] [task] created task=0x7f22ec1853f0 id=1 OK
[2025/02/18 11:22:17] [debug] [output:stdout:stdout.1] task_id=1 assigned to thread #0
[2025/02/18 11:22:17] [debug] [out flush] cb_destroy coro_id=2
[2025/02/18 11:22:17] [ warn] [output:opentelemetry:opentelemetry.0] error performing HTTP request, remote host=remote.host.name:4318 connection error
[2025/02/18 11:22:17] [debug] [upstream] KA connection #65 to remote.host.name:4318 is now available
[2025/02/18 11:22:17] [debug] [upstream] KA connection #65 to remote.host.name:4318 has been disconnected by the remote service
[2025/02/18 11:22:17] [debug] [out flush] cb_destroy coro_id=1
[2025/02/18 11:22:17] [debug] [retry] new retry created for task_id=0 attempts=1
[2025/02/18 11:22:17] [ warn] [engine] failed to flush chunk '30926-1739906536.401558167.flb', retry in 8 seconds: task_id=0, input=tail.0 > output=opentelemetry.0 (out_id=0)
[2025/02/18 11:22:17] [debug] [input:tail:tail.0] inode=8603124, ErrorLog.txt, events: IN_MODIFY
[2025/02/18 11:22:17] [debug] [input:tail:tail.0] inode=8603124, ErrorLog.txt, events: IN_MODIFY
[2025/02/18 11:22:17] [debug] [input:tail:tail.0] inode=8603124, ErrorLog.txt, events: IN_MODIFY
  • Attached Valgrind output that shows no leaks or memory corruption was found
# Output of running valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes bin/fluent-bit
==27800== LEAK SUMMARY:
==27800==    definitely lost: 0 bytes in 0 blocks
==27800==    indirectly lost: 0 bytes in 0 blocks
==27800==      possibly lost: 0 bytes in 0 blocks
==27800==    still reachable: 99,872 bytes in 3,382 blocks
==27800==         suppressed: 0 bytes in 0 blocks
==27800==
==27800== For counts of detected and suppressed errors, rerun with: -v
==27800== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [N/A] Run local packaging test showing all targets (including any new ones) build.
  • [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [N/A] Documentation required for this feature

Backporting

  • [N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Vanessa Zhang added 2 commits February 18, 2025 15:47
…use const char* for unit_name

Signed-off-by: Vanessa Zhang <[email protected]>
This patch introduces new APIs to create a single processor unit for input and output instances:

1. flb_input_processor_unit(flb_ctx_t *ctx, const char *event_type, const char *processor_unit_name, int ffd,  …)

Create a single input processor unit for a specific event type and unit name

2. flb_output_processor_unit(flb_ctx_t *ctx, const char *event_type, const char *processor_unit_name, int ffd,  …)

Create a single output processor unit for a specific event type and unit name

Signed-off-by: Vanessa Zhang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant