Fix(Source-Intercom): Update Pagination #55829
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Solves: https://github.com/airbytehq/oncall/issues/7494 Missing data due to incorrect pagination.
How
There were a few things off with the migration to manifest change.
The following changes were made:
admins,tags,teams,segments,company_attributes,contact_attributes, stream(s) do not have input parameters for pagination, so it was removed. DocRemoved
page_sizeforcompaniesstream since it doesn't use this (API defaults to 100) [Doc]I also updated the
error_handlerfor HTTP 500s. The API docs say we must restart the sync entirely, but the current option,action: RETRY, would restart it from the current page. - Ideally, I think we should have a 'RESTART' action built into the CDK. So, I will make a separate ticket for that here: https://github.com/airbytehq/airbyte-internal-issues/issues/12107conversations,tickets, &contacts- pagination was off.It has been corrected & tested locally. (Except for tickets, we do not have premium to test, but these all use the same type of pagination, so the changes should work for that, too.)
The issue with these is each stream had a pagination object hardcoded in
request_body_json, which I think was conflicting with thepaginatorcomponent.Also, the paginator injector seemed to not be injecting, so it was changed to fix this.
Lastly, the stop condition seemed a bit off, so this was also updated to make sure we didn't miss records.
Review guide
User Impact
Can this PR be safely reverted and rolled back?