Support Create Partitions #623
Conversation
|
Working on tests still but posting for initial feedback |
|
thank you for the PR @BlueCollarChris |
|
Just seeing the comments now and will get back to you on Monday. I had been exploring a different route locally where the sync is done outside the client in a linked genserver. I wasn't sure if the sync process were to block the client would be disrupted from handling other requests. I'll see about getting those changes up shortly. |
| interval = Interval | ||
| } = State | ||
| ) -> | ||
| sync_partitions(Client, Sup, Config), |
There was a problem hiding this comment.
This ensure the new partitions get a producer but then we need to tell the client to update the metadata so it can use the new partition.
| brod_supervisor3:start_child(SupPid, Spec). | ||
|
|
||
| %% @doc Dynamically start a per partition producer | ||
| -spec start_producer(pid(), pid(), brod:topic(), non_neg_integer(), brod:producer_config()) -> |
There was a problem hiding this comment.
variant for starting just the partition producer with the correct child spec
|
@zmstone got the update in I mentioned over the weekend. Is there a specific formatter you recommend looks like the one for my IDE formatted some stuff showing more changes than were there previously. I did make the change to start the partition producer under the brod_supervisor3 as mentioned in your previous comment. Working on the tests for the brod_partitions_sync module still. |
|
Hi @BlueCollarChris About code format: we use https://github.com/WhatsApp/erlfmt Possible to leave code reformat out in this PR? it makes review easier. |
|
I have been completely swamped but will get to updating the formatting of this very soon, sorry for the delay |
Co-authored-by: zmstone <[email protected]>
|
thanks. there are some conflicts. |
2 participants
Adds in the ability to create partitions for an existing topic.
Adds in the ability for clients to synchronize their topic metadata to pick up new partitions and start a producer.