feat: Apply post_process automatically to all stream types

[edgarrmondragon]

With this change, developers won't need to re-implement the call to post_process when they override the public Stream.get_records(...) method.

A few examples just from MeltanoLabs: https://github.com/search?q=org%3AMeltanoLabs+%22self.post_process%22&type=code.

i.e.

• https://github.com/MeltanoLabs/tap-dbt/blob/49cbf08f91d297d73f2cf19d73614a9a65a31f70/tap_dbt/streams.py#L107-L111
• https://github.com/MeltanoLabs/tap-postgres/blob/d7e11bcaae0d87717771a5a38744d0ec674dabf5/tap_postgres/client.py#L259-L267
• https://github.com/MeltanoLabs/tap-jira/blob/caa12cdc5ee890a440e61df880f76172c9a85e8a/tap_jira/streams.py#L2402-L2406
• https://github.com/MeltanoLabs/tap-messagebird/blob/37a64a4b178e7c1e511ac4481bd69b2e145c6df6/tap_messagebird/streams.py#L98-L102

Summary by Sourcery

Automatically apply post_process method to all stream types during record synchronization

New Features:

• Automatically invoke post_process method for all stream types during record sync

Enhancements:

• Updated record processing workflow to consistently apply post-processing across different stream types

Tests:

• Added test case to verify post-processing functionality for streams

[sourcery-ai]

Reviewer's Guide

This pull request refactors record post-processing by centralizing the post_process method call within the base Stream._sync_records method. This ensures post_process is uniformly applied to records from all stream types (e.g., SQL, REST) immediately after they are fetched and before other processing steps. Consequently, explicit post_process calls were removed from stream-specific subclasses, and relevant type hints were updated to use types.Record.

File-Level Changes

Change	Details	Files
Centralized post_process logic in Stream._sync_records and updated related type hints.	Invokes self.post_process() for each record within _sync_records after retrieval. Skips further processing for records if post_process returns None. Updated the post_process method signature and Stream.get_records return type to use types.Record.	singer_sdk/streams/core.py
Removed redundant post_process calls from specialized stream classes.	Deleted explicit post_process() invocation from SQLStream.get_records(). Deleted explicit post_process() invocation from RESTStream.get_records().	singer_sdk/streams/sql.py singer_sdk/streams/rest.py
Added unit tests to verify post_process functionality for record filtering and transformation.	Implemented test_post_process_drops_record to verify records can be filtered out. Implemented test_post_process_transforms_record to verify records can be modified. Updated test data in tests/core/conftest.py (e.g. adding updatedAt field) to support new tests.	tests/core/test_streams.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.71%. Comparing base (d376b62) to head (917bb89).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3023      +/-   ##
==========================================
+ Coverage   91.64%   91.71%   +0.06%     
==========================================
  Files          62       62              
  Lines        5315     5311       -4     
  Branches      686      684       -2     
==========================================
  Hits         4871     4871              
+ Misses        312      310       -2     
+ Partials      132      130       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[codspeed-hq]

CodSpeed Performance Report

Merging #3023 will not alter performance

_{Comparing post-process-all-streams (917bb89) with main (d376b62)}

Summary

✅ 8 untouched benchmarks

[edgarrmondragon]

@sourcery-ai review

[edgarrmondragon]

@sourcery-ai review

[edgarrmondragon]

@sourcery-ai review

[edgarrmondragon]

@sourcery-ai review

[sourcery-ai]

Hey @edgarrmondragon - I've reviewed your changes - here's some feedback:

• Consider if adding updatedAt to SimpleTestStream records in conftest.py affects unrelated tests expecting the previous record structure.
• For streams inheriting directly from Stream (rather than SQLStream or RESTStream), post_process is now called by the framework; verify this doesn't conflict with any manual post_process calls in custom get_records implementations.

Here's what I looked at during the review

• 🟢 General issues: all looks good
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

Hey @edgarrmondragon - I've reviewed your changes and they look great!

Here's what I looked at during the review

• 🟢 General issues: all looks good
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}