Skip to content

DOCSP-48557 Update Spark streaming write configuration to include all batch options #261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Jul 8, 2025
Merged

DOCSP-48557 Update Spark streaming write configuration to include all batch options #261

merged 6 commits into from
Jul 8, 2025

Conversation

mballard-mdb
Copy link
Contributor

@mballard-mdb mballard-mdb commented Jul 2, 2025 •
edited
Loading

Pull Request Info

PR Reviewing Guidelines

JIRA - https://jira.mongodb.org/browse/DOCSP-48557

Staging Links

  • batch-mode/batch-write-config
  • streaming-mode/streaming-write-config

    • Self-Review Checklist

    • Is this free of any warnings or errors in the RST?
    • Did you run a spell-check?
    • Did you run a grammar-check?
    • Are all the links working?
    • Are the facets and meta keywords accurate?
    • Are the page titles greater than 20 characters long and SEO relevant?

    @netlify Netlify
    Copy link

    netlify bot commented Jul 2, 2025 •
    edited
    Loading

    ✅ Deploy Preview for docs-spark-connector ready!

    Name Link
    🔨 Latest commit e385628
    🔍 Latest deploy log https://app.netlify.com/projects/docs-spark-connector/deploys/686c2b4dfa2c6c00082f67cd
    😎 Deploy Preview https://deploy-preview-261--docs-spark-connector.netlify.app
    📱 Preview on mobile
    Toggle QR Code...

    QR Code

    Use your smartphone camera to open QR code link.

    To edit notification comments on pull requests, go to your Netlify project configuration.

    @docs-builder-bot
    Copy link

    docs-builder-bot commented Jul 2, 2025 •
    edited
    Loading

    🔄 Deploy Preview for docs-spark-connector processing

    Item Details
    🔨 Latest Commit e49a0a3354219da9ac519ed6d1472d44007cc40a
    😎 Deploy Preview https://deploy-preview-261--docs-spark-connector.netlify.app
    🔍 Build Logs View Logs

    shuangela
    shuangela requested changes Jul 2, 2025
    View reviewed changes
    Copy link
    Contributor

    @shuangela shuangela left a comment

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    great job on this! added some comments related to wording as well as a question about a default setting in one of the parameters.

    source/batch-mode/batch-write-config.txt Outdated
    w Option </reference/write-concern/#w-option>` in the {+mdb-server+}
    manual.
    |
    | **Default:** ``1``
    Copy link
    Contributor

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    Q: How did you get that this default is 1? the server manual states only that "If the write concern is missing the w field, MongoDB sets the w option to the default write concern," and then later on in the table it says "{ w: "majority" } is the default write concern for most MongoDB deployments". The implicit default write concern for most mongo deployments seems to be majority: https://www.mongodb.com/docs/manual/reference/write-concern/#std-label-wc-default-behavior

    This question applies to the other writeConcern.w option as well

    Copy link
    Contributor Author

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    I just moved what was on the batch page to the stream page, so I didn't write any of the info. I assumed most of it was correct but I'll double check everything now.

    Copy link
    Contributor Author

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    I see that it can be 1 or majority, so I'm going to include both, with majority taking precedence as it is most cases like you said

    Copy link
    Contributor

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    I think it's a bit odd if this option has two potential defaults, so it would be good to double check with the technical reviewer in this case

    Copy link
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    Defaults to Acknowledged which is the default set by the server.

    @mballard-mdb mballard-mdb requested a review from shuangela July 2, 2025 18:12
    shuangela
    shuangela approved these changes Jul 2, 2025
    View reviewed changes
    Copy link
    Contributor

    @shuangela shuangela left a comment

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    lgtm with a note to ask tech reviewer for confirmation on something!

    @mballard-mdb mballard-mdb requested a review from rozza July 7, 2025 20:17
    rozza
    rozza approved these changes Jul 8, 2025
    View reviewed changes
    Copy link
    Member

    @rozza rozza left a comment

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    LGTM!

    @mballard-mdb mballard-mdb merged commit 792cdc0 into mongodb:master Jul 8, 2025
    6 checks passed
    mballard-mdb added a commit to mballard-mdb/docs-spark-connector that referenced this pull request Jul 8, 2025
    @mballard-mdb
    DOCSP-48557 Update Spark streaming write configuration to include all…
    bfa2fa2 
    … batch options (mongodb#261)

* DOCSP-48557 spark options update

* DOCSP-48557 links and reorder

* DOCSP-48667 link fix + mdb-server consistency

* DOCSP-48557 wording

* DOCSP-48557 remove ital

* DOCSP-49557 update default w concern

(cherry picked from commit 792cdc0)
    @mballard-mdb mballard-mdb mentioned this pull request Jul 8, 2025
    Merged
    mballard-mdb added a commit to mballard-mdb/docs-spark-connector that referenced this pull request Jul 8, 2025
    @mballard-mdb
    DOCSP-48557 Update Spark streaming write configuration to include all…
    d3f740d 
    … batch options (mongodb#261)

* DOCSP-48557 spark options update

* DOCSP-48557 links and reorder

* DOCSP-48667 link fix + mdb-server consistency

* DOCSP-48557 wording

* DOCSP-48557 remove ital

* DOCSP-49557 update default w concern

(cherry picked from commit 792cdc0)
    @mballard-mdb mballard-mdb mentioned this pull request Jul 8, 2025
    Merged
    mballard-mdb added a commit to mballard-mdb/docs-spark-connector that referenced this pull request Jul 8, 2025
    @mballard-mdb
    DOCSP-48557 Update Spark streaming write configuration to include all…
    e89ba0e 
    … batch options (mongodb#261)

* DOCSP-48557 spark options update

* DOCSP-48557 links and reorder

* DOCSP-48667 link fix + mdb-server consistency

* DOCSP-48557 wording

* DOCSP-48557 remove ital

* DOCSP-49557 update default w concern

(cherry picked from commit 792cdc0)
    @mballard-mdb mballard-mdb mentioned this pull request Jul 8, 2025
    Merged
    mballard-mdb added a commit to mballard-mdb/docs-spark-connector that referenced this pull request Jul 8, 2025
    @mballard-mdb
    DOCSP-48557 Update Spark streaming write configuration to include all…
    b9cd179 
    … batch options (mongodb#261)

* DOCSP-48557 spark options update

* DOCSP-48557 links and reorder

* DOCSP-48667 link fix + mdb-server consistency

* DOCSP-48557 wording

* DOCSP-48557 remove ital

* DOCSP-49557 update default w concern

(cherry picked from commit 792cdc0)
    @mballard-mdb
    Copy link
    Contributor Author

    💚 All backports created successfully

    Status Branch Result
    ✅ v10.5
    ✅ v10.4
    ✅ v10.3
    ✅ v10.2

    Note: Successful backport PRs will be merged automatically after passing CI.

    Questions ?

    Please refer to the Backport tool documentation

    @mballard-mdb mballard-mdb mentioned this pull request Jul 8, 2025
    Merged
    mballard-mdb added a commit that referenced this pull request Jul 8, 2025
    @mballard-mdb
    DOCSP-48557 Update Spark streaming write configuration to include all…
    ec06478 
    … batch options (#261) (#262)

* DOCSP-48557 spark options update

* DOCSP-48557 links and reorder

* DOCSP-48667 link fix + mdb-server consistency

* DOCSP-48557 wording

* DOCSP-48557 remove ital

* DOCSP-49557 update default w concern

(cherry picked from commit 792cdc0)
    @mballard-mdb mballard-mdb deleted the DOCSP-48557-spark-batch-options branch July 8, 2025 13:36
    mballard-mdb added a commit that referenced this pull request Jul 8, 2025
    @mballard-mdb
    DOCSP-48557 Update Spark streaming write configuration to include all…
    e27c12e 
    … batch options (#261) (#264)

* DOCSP-48557 spark options update

* DOCSP-48557 links and reorder

* DOCSP-48667 link fix + mdb-server consistency

* DOCSP-48557 wording

* DOCSP-48557 remove ital

* DOCSP-49557 update default w concern

(cherry picked from commit 792cdc0)
    mballard-mdb added a commit that referenced this pull request Jul 8, 2025
    @mballard-mdb
    DOCSP-48557 Update Spark streaming write configuration to include all…
    f430782 
    … batch options (#261) (#263)

* DOCSP-48557 spark options update

* DOCSP-48557 links and reorder

* DOCSP-48667 link fix + mdb-server consistency

* DOCSP-48557 wording

* DOCSP-48557 remove ital

* DOCSP-49557 update default w concern

(cherry picked from commit 792cdc0)
    mballard-mdb added a commit that referenced this pull request Jul 8, 2025
    @mballard-mdb
    DOCSP-48557 Update Spark streaming write configuration to include all…
    ebdb00e 
    … batch options (#261) (#265)

* DOCSP-48557 spark options update

* DOCSP-48557 links and reorder

* DOCSP-48667 link fix + mdb-server consistency

* DOCSP-48557 wording

* DOCSP-48557 remove ital

* DOCSP-49557 update default w concern

(cherry picked from commit 792cdc0)
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Reviewers

    @rozza rozza rozza approved these changes

    @shuangela shuangela shuangela approved these changes

    Assignees
    No one assigned
    Labels
    None yet
    Projects
    None yet
    Milestone
    No milestone
    Development

    Successfully merging this pull request may close these issues.
    4 participants
    @mballard-mdb @docs-builder-bot @rozza @shuangela