Skip to content

[BUG] Writable Warm - FullFileCachedIndexInput Clone map can cause memory explosion #18659

@rayshrey

Description

@rayshrey

Describe the bug

The current implementation of FullFileCachedIndexInput maintain a set of all clones generated from an IndexInput
These clones get cleaned up only when their parent IndexInput gets closed.

When the base parent IndexInput is not closed, it results in all the clones being still open and having their entries in the clone set.

The size of the set grows too large and contributes significantly to memory overhead.

Related component

Storage:Remote

To Reproduce

Run update test procedure for nyc_taxis workload.and monitor heap usage (it will keep on increasing)

Expected behavior

Clones should get cleaned up as soon as they become unreferenced.

Additional Details

Heap dump analysis showing FullFileCachedIndexInput objects as the main contributor to memory.

Image
Image

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Status

    ✅ Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions