feat: Adding with_schema_unchecked method for RecordBatch
#7402
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
etseidl
reviewed
Apr 10, 2025
arrow-array/src/record_batch.rs
Outdated
| /// | ||
| /// If provided schema is not compatible with this [`RecordBatch`] columns the runtime behavior | ||
| /// is undefined | ||
| pub fn with_schema_force(self, schema: SchemaRef) -> Result<Self, ArrowError> { |
There was a problem hiding this comment.
Maybe with_schema_unchecked would be better?
arrow-array/src/record_batch.rs
Outdated
Comment on lines
362
to
364
| /// Forcibly overrides the schema of this [`RecordBatch`] | ||
| /// without additional schema checks however bringing all the schema compatibility responsibilities | ||
| /// to the caller site. |
There was a problem hiding this comment.
Suggested change
| /// Forcibly overrides the schema of this [`RecordBatch`] | |
| /// without additional schema checks however bringing all the schema compatibility responsibilities | |
| /// to the caller site. | |
| /// Overrides the schema of this [`RecordBatch`] | |
| /// without additional schema checks. Note, however, that this pushes all the schema compatibility responsibilities | |
| /// to the caller site. In particular, the caller guarantees that `schema` is a superset | |
| /// of the current schema as determined by [`Schema::contains`]. |
arrow-array/src/record_batch.rs
Outdated
| @@ -1637,4 +1653,57 @@ mod tests { | |||
| "bar" | |||
| ); | |||
| } | |||
|
|
|||
| #[test] | |||
| fn test_batch_with_force_schema() { | |||
There was a problem hiding this comment.
Could you please also add a check where the forced schema succeeds?
arrow-array/src/record_batch.rs
Outdated
| // Wrong number of columns | ||
| let invalid_schema_more_cols = Schema::new(vec
|
Thanks @etseidl for the feedback. Addressed all the comments |
comphead
changed the title
with_schema_force method for RecordBatchwith_schema_unchecked method for RecordBatch
etseidl
approved these changes
Apr 10, 2025
|
I think this needs to either be reverted or made unsafe, it allows constructing invalid ArrayData with safe code... Edit: PR to make this unsafe - #7405 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #.
Rationale for this change
Sometimes Arrow-rs users don't require schema checks on
RecordBatchrelying on their schema checks and schema validity rules. However now it is not possible to override the schema without performance impact.Examples: apache/datafusion#15162
apache/datafusion#15603
Proposed
with_schema_uncheckedmethod forRecordBatchoverrides the schema without additional schema checks however bringing all the schema compatibility responsibilities to the caller site.What changes are included in this PR?
Are there any user-facing changes?