Mechanism for plugins to share state between extension points

[liu-cong]

We currently use a CycleState, which is a map for the scheduler plugins to put and get state info. This is mainly used by the prefix cache aware plugin to share states between the Score and PostCycle extension points.

And to deprecate the PostCycle extension point in favor of the PreRequest, we need a mechanism to share states between the scheduler and director layers.

From the discussion in #1062, there seems to be 2 major questions:

1. The use of a generic CycleState map itself.

Do we need a safer mechanism rather than a generic map? Should different scheduling profiles share state and if so, how to make sure they don't corrupt each other?

2. How to let plugins communicate across layers.

Specifically for the prefix cache plugin, how to share state between the Scorer and PreRequest extension points.

cc @nirrozenbaum @kfswain @ahg-g

@kfswain will take the AI to create a proposal.

[liu-cong]

/assign @kfswain

[ahg-g]

Just to clarify, this is a about state that is expected to live only during the processing of a request.

[elevran]

Clarification: @liu-cong The same plugin instance is running in both cases?
If so, perhaps can use an internal map, keyed on request id, to carry state from Scorer to PreRequest (which can clear that once processed to avoid leaking)?

[liu-cong]

If so, perhaps can use an internal map, keyed on request id, to carry state from Scorer to PreRequest (which can clear that once processed to avoid leaking)?

If I understand it correctly, you mean having the plugin itself manage the per-request state internally as a map and handle the cleanup itself? While this can work, there is some complexity making sure the map is cleaned up properly. For example, what if PreRequest extension point is not called due to earlier request processing errors or a bug? Then the plugin needs to maintain some GC process.

[nirrozenbaum]

If so, perhaps can use an internal map, keyed on request id, to carry state from Scorer to PreRequest (which can clear that once processed to avoid leaking)?

If I understand it correctly, you mean having the plugin itself manage the per-request state internally as a map and handle the cleanup itself? While this can work, there is some complexity making sure the map is cleaned up properly. For example, what if PreRequest extension point is not called due to earlier request processing errors or a bug? Then the plugin needs to maintain some GC process.

@liu-cong I think this is a good direction.
so scheduler plugins can share state using CycleState.
and every plugin can share state with itself within multiple extension points that do not necessarily belong to the same layer (e.g., Score + PostResponse).
as you mentioned this might require some cleanup and maintenance. therefore I think we should provide the mechanism (as we provide CycleState) and embed it in plugins.
then a plugin can decide to use it or not, but a plugin writer wouldn't need to worry about these things.