Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure copy-only and copy-and-stream are resumable just like stream mode #24

Open
shayonj opened this issue Nov 4, 2024 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@shayonj
Copy link
Collaborator

shayonj commented Nov 4, 2024

If a the bulk copy/load experiences a network disruption today or a deployment comes in, the entire process would need to be started over again. Lets make it resumable like streaming replicator.

Further clarification

  • Streaming is fully resumable, say if you have to shutdown the container, or a SIGTERM comes, it will flush, save state, exit.
  • The copy modes are not resumable today, so if you need to interrupt the copy, for now its best to dump/delete the table in the target DB and start the process again. It parallelizes the data syncs, so should be fast, depending on table size.
  • Once the operation moves from copy to stream (which happens automatically when you use copy-and-stream) the operation is resumable-able again, since the initial sync is complete and the new data is flowing from the replication slot.
@shayonj shayonj added the enhancement New feature or request label Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant