PHOENIX-4906 Introduce a master coprocessor to handle cases where merging regions of salted table might cause an issue #1552
Conversation
|
I suggest using a more generic name, like PhoenixMasterObserver. This will make our lives easier if we ever need to add more features to it. |
|
@stoty i have changed the names accordingly. Please review when you get some cycles. |
mnpoonia
force-pushed
the
salt_sp_cases
branch
2 times, most recently
from
56edc6c to
fa00f5b
Compare
…ging regions of salted table might cause an issue
| Configuration conf = ctx.getEnvironment().getConfiguration(); | ||
| PreparedStatement stmt; | ||
| ResultSet resultSet; | ||
| try (Connection conn = QueryUtil.getConnectionOnServer(conf)) { |
There was a problem hiding this comment.
I wonder if we should get the salted status via a PTable object from the CQSI cache instead.
Wouldn't that be faster ?
| @BeforeClass | ||
| public static synchronized void doSetup() throws Exception { | ||
| Map<String, String> serverProps = Maps.newHashMapWithExpectedSize(3); | ||
| serverProps.put("hbase.coprocessor.master.classes", PhoenixMasterObserver.class.getName()); |
There was a problem hiding this comment.
There is no way to automatically load a Master coprocessor on Phoenix startup, right ?
There was a problem hiding this comment.
I think better to use config for master/regionserver coproc loading. Even if we do figure out letting phoenix update coproc at runtime, pretty sure we would end up using private/unstable APIs that can break across releases.
I was wondering this same point sometime back and hence decided to document this detail that we need to use this coproc by updating hbase config, having no other choice but to document it #1561 (comment)
|
Also, since this is a broad topic, perhaps we could split this feature into its separate JIRA ? |
|
I am mostly positive about this change but need help in two things
@virajjasani @kadirozde @apurtell @stoty Looking forward for your suggestions |
|
I think using something like org.apache.phoenix.util.PhoenixRuntime.getTable(Connection, String)is the easiest (and perhaps fastest). As for the processor, I could not find a way to load from code, it has to be configured manually. |
|
Few questions @mnpoonia
A couple of questions if Phoenix needs to be involved @virajjasani @apurtell your thoughts? |
|
@mnpoonia Also can we ensure that it is not a Phoenix Bug in ScanRanges? |
|
Salting is not an HBase feature, it's a Phoenix feature (at least as used by Phoenix). Ideally, HBase had a MergePolicy mechanism simliar to the SplitPolicy mechanism where we could add this logic per table, but it doesn't. I admit I that I have never really tracked down why we have this merging restriction, but it seems to be a performance optimization. |
@stoty I referring to what was explained here in the hot-spotting section of the hbase book. I think Phoenix just facilitates this under the cover for the user. What I was asking is shouldn't we be seeing this issue in a purely HBase table with row keys distributed with salt prefixes? I also think we should investigate more why the scanRanges are behaving such. Will try and dig more. |
|
Yes, the problem is caused by this assumption built into the Phoenix implementation. |
@jpisaac I am not sure but my understanding is that hbase doesnot have a concept of salting. Clients can use salting to reduce the hotpostitng but then it is upto clients that how they want to manage all aspects of it. HBase doesn't care about salting in case of split or merge. In phoenix we have this limitation that every row in a single region should have same salt. If we mix(merge) two regions of different salts phoenix scans starting behaving weird. |
| if(saltBuckets > 0 ) { | ||
| System.out.println("Number of buckets="+saltBuckets); | ||
| return !regionsHaveSameSalt(regionsToMerge); | ||
| if(saltBuckets <= 0 ) { |
There was a problem hiding this comment.
Nit: Having negative salt buckets here points to some serious problem with the tabke.
I'd just check for 0 here.
No description provided.