HBASE-28447 New site configuration option "hfile.block.size" #5820
+147
−18
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduce a new configuration setting - "hfile.block.size" that, if set, will define the default blocksize to use when writing HFiles if a column family schema does not define its own non-default block size. This is a bit complicated but required for compatability. The rules are: - If the schema specifies a non default block size, use it. - Otherwise, if the configuration specifies a non default block size, use it. - Otherwise, use the default block size. The default is defined by HConstants.DEFAULT_BLOCKSIZE.
apurtell
changed the title
virajjasani
approved these changes
Apr 10, 2024
Comment on lines
+42
to
+45
| @Test | ||
| public void testGetBlockSize() throws IOException { | ||
| int eightK = 8 * 1024; | ||
| int oneM = 1024 * 1024; |
There was a problem hiding this comment.
I wonder if this could have been part of any existing test class, but no problem having it's own separate class either
There was a problem hiding this comment.
I think it's best to start a TestStoreUtils class for testing StoreUtils. Some other units might be moved in here or added later. That would make more sense to me.
|
💔 -1 overall
This message was automatically generated. |
|
The spotless check fails but when I run 'mvn spotless:apply' on my local branch there are no changes. |
|
Ah, there was something wrong with CI during the spotless check |
|
Ya that happens sometimes, not sure why |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
No table level configuration? |
| * @param schemaBlockSize The block size as specified in the column family schema. | ||
| * @return The block size to use. | ||
| */ | ||
| public static int getBlockSize(Configuration conf, int schemaBlockSize) { |
There was a problem hiding this comment.
A clarification :
Though you have already called out the "non default block size" - One Question - Right now there can be a case where the site configuration has non default block size say 1MB(to be applied to all the table/CF), and for some CF/table one wants explicitly 64 KB - so setting 64KB explicitly again in the schema won't be picked, as that is the default.
For such cases, one will need to explicitly set the schemaBlocksize(either BLOCKSIZE or configuration override in schema) for all required tables to 1MB.
|
@apurtell is this ready to be merged? It currently has fixVersion 2.6.0, and I'm looking to create the next RC tomorrow |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduce a new configuration setting - "hfile.block.size" that, if set, will define the default blocksize to use when writing HFiles if a column family schema does not define its own non-default block size. This is a bit complicated but required for compatability. The rules are:
The default is defined by HConstants.DEFAULT_BLOCKSIZE.
Given how compound configurations work the precedence order for a non default block size is: BLOCKSIZE in the column family schema > "hfile.block.size" in CF or table level schema > "hfile.block.size" in site configuration > HConstants.DEFAULT_BLOCKSIZE