Skip to content

Conversation

@GreyLilac09
Copy link
Contributor

@GreyLilac09 GreyLilac09 commented Oct 30, 2025

Summary

  • Create metrics for cache size, events, and evictions
  • Always track cache size (for metrics tracking)
  • Previously, the size for the event values for gauges (and other metrics) was always 0, which is inaccurate and messes up memory tracking
  • Get the item after inserting to LRU cache, ensuring it moves to the back of the LRU cache and doesn't get evicted immediately if the cache is full
  • Added some tests for incremental_to_absolute, absolute_to_incremental, and LRU cache capacity policy behavior

Vector configuration

data_dir: ./vector-data-dir
sources:
  s0:
    type: statsd
    address: "0.0.0.0:8127"
    mode: udp
  internal_metrics:
    scrape_interval_secs: 10
    type: internal_metrics
transforms:
  t0:
    type: incremental_to_absolute
    inputs:
      - s0
    cache:
      max_bytes: 5000
      max_events: 100
  filter_cache_metrics:
    type: filter
    inputs:
      - internal_metrics
    condition: '.name == "component_cache_evictions_total" || .name == "component_cache_bytes" || .name == "component_cache_events"'
sinks:
  blackhole:
    type: blackhole
    inputs:
      # - s0
      - t0
  file:
    type: file
    inputs:
      - filter_cache_metrics
    path: /vector-%Y-%m-%d.log
    encoding:
      codec: json

How did you test this PR?

Graph showing new metrics:
Screenshot 2025-10-30 at 1 25 18 PM

Graph showing consistent allocated bytes for incremental_to_absolute transform when max_bytes is set:

Screenshot 2025-10-30 at 1 31 05 PM

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Notes

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
  • Some CI checks run only after we manually approve them.
    • We recommend adding a pre-push hook, please see this template.
    • Alternatively, we recommend running the following locally before pushing to the remote branch:
      • make fmt
      • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
      • make test
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

@GreyLilac09 GreyLilac09 requested review from a team as code owners October 30, 2025 17:27
@github-actions github-actions bot added domain: transforms Anything related to Vector's transform components domain: sinks Anything related to the Vector's sinks domain: external docs Anything related to Vector's external, public documentation domain: core Anything related to core crates i.e. vector-core, core-common, etc labels Oct 30, 2025
@GreyLilac09 GreyLilac09 changed the title fix(incremental_to_absolute transform): fix memory tracking and emit cache metrics fix(incremental_to_absolute transform): fix LRU cache memory tracking and emit cache metrics Oct 30, 2025
@GreyLilac09 GreyLilac09 changed the title fix(incremental_to_absolute transform): fix LRU cache memory tracking and emit cache metrics fix(incremental_to_absolute transform): fix LRU cache memory handling and emit cache metrics Oct 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: core Anything related to core crates i.e. vector-core, core-common, etc domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks domain: transforms Anything related to Vector's transform components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants