HBASE-28836 Parallize the file archival to improve the split times #6616
Conversation
|
@virajjasani @stoty I am giving it another try. I hope this doesn't fail this time. I am running tests locally and they haven't finished yet. But i am optimistic this time. 🤞🏾 |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
I still see some failures that are not the usual flakies. |
mnpoonia
force-pushed
the
parallel_HBASE-28836
branch
2 times, most recently
from
4bd65e7 to
d3b481d
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
@stoty No test failure this time. Please have a look. Let me know if i am missing something here. |
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
Show resolved
Hide resolved
| Queue<File> failures, String startTime) { | ||
| LOG.trace("Archiving {} files concurrently into directory: {}", files.size(), baseArchiveDir); | ||
|
|
||
| ExecutorService executorService = Executors.newCachedThreadPool(); |
There was a problem hiding this comment.
Most such similar pools are configurable.
Have you configured making the thread pool configurable ?
Would it make sense to use a global pool here, and limit the number of concurrent move operations ?
There was a problem hiding this comment.
Considering we are shutting down the threads after execution is it okay if we give some valid constant rather than a configuration? I am of the opinion that one more configuration would not help us. I also understand that having a max cap on number of threads is an important aspect.
There was a problem hiding this comment.
TBH I don't have a lot of experience with object storage deletion performance.
Are resolveAndArchive calls serial, or is it possible to have multiple invocations running at the same time ?
What do you think @wchevreuil, @BukrosSzabolcs ?
mnpoonia
force-pushed
the
parallel_HBASE-28836
branch
from
2063007 to
63f84cc
Compare
… methods. (apache#6500) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Pankaj Kumar<[email protected]> HBASE-28836 Parallize the file archival to improve the split times
Add a config to limit thread per region
mnpoonia
force-pushed
the
parallel_HBASE-28836
branch
from
63f84cc to
ccb7f27
Compare
| Map<File, Future<Boolean>> futureMap = new HashMap<>(); | ||
| // Submit file archiving tasks | ||
| // default is 16 which comes equal hbase.hstore.blockingStoreFiles default value | ||
| int maxThreads = conf.getInt("hbase.hfilearchiver.per.region.thread.pool.max", 16); |
There was a problem hiding this comment.
@stoty Does this code make sense. Here we are able to put a limit on number of threads per region dir. And it aligns with what i was initially thinking to implement as well.
Had to force push my branch for PR because i had messed it up. Sorry for that.
There was a problem hiding this comment.
That is definitely an improvement.
As I said, I don't have a lot of experience with object storage, neither do I know how these archival chores are started.
On a large system, we have 10s os thousands of regions, so this could still be a lot of threads, which may overload AWS's computers.
If these are started on the RSs as opposed to master, maybe it would make sense use a per-instance pool ?
I don't have enough background info to have a solid opinion, just sharing my thoughts.
There was a problem hiding this comment.
On a large system, we have 10s os thousands of regions, so this could still be a lot of threads,
I think this could be an issue for Delete table scenario for large table where it actually archives all the regions at once in a loop, we might end up creating 16 X noOfRegions threads.
@stoty @mnpoonia
How about, keeping a flag on archiveFiles(boolean needsConcurrent), so that archival done as part of SplitProc does it concurrently and DeleteProc can still do it in existent sequential mode (since its not critical/timebound unlike SpitProcedure which can effect availability)
There was a problem hiding this comment.
Yes, that would alleviate some of the potential problems.
I really don't know the controll flow and parallel behaviour here, so I cannot be more definite.
I know we have seen issues where the (serial) cleanup took several days, and had trouble keeping up, but I haven't followed the exact circumstances, i.e. was it deletion or just simple compactions.
There was a problem hiding this comment.
To explain more of what i know.
Here the max number of threads that can be created after this change are hbase.hfilearchiver.per.region.thread.pool.max * hbase.hfilearchiver.thread.pool.max which by default comes to 8 * 16 = 128. But currently the defaults are 8 * 1 = 8 threads.
There was a problem hiding this comment.
@stoty I think i am missing something. Are you saying that if we have 10k files in one folder and single layer with 128 threads then that would perform better than having two level parallel threads? Can you elaborate on this a bit?
IUC The outer threadpool walks over directories, then feeds the archival operations into the inner threadpool.
If the outer pool has a size of 8, and the inner pools have a size of 16, then, since a single inner pool processes all files in a single directory, the parallelism will only be 16 (in the pathological example).
If the threads processing directories share a bigger threadpool (which is equal to the max possible in the current case, i.e. 128), then the parallelism for the arhichival file operation will be 128 even in the pathological case.
There was a problem hiding this comment.
I see. I was going after a case where number of regions to be cleaned are very small maybe 2 or 3 regions but number of hfiles in the regions is 50 to 100. So since we have parallelism at region layer and no parallelism within region to clean hfiles we get a performance hit.
So clearly we have two different problems and both are valid. I am just thinking now how we can solve both more generically.
There was a problem hiding this comment.
@mnpoonia , I'm a bit confused by how this is indeed impacting split times. My readings from the code is that this is only used from splits in the case of rollback. Am I missing something here?
Others parts this would run from master would be from delete table procedure, tuncate or the region catalog janitor. In all these cases, we would be creating one pool per region in the table being deleted/truncated, or to amount of regions to be cleaned by a given catalog janitor run.
There was a problem hiding this comment.
If we're splitting a region with store files which have been compacted but not yet archived. In that case region close will wait for these compacted files to be removed which goes through HFileArchiver.archive. In one of my test the unassign proc's were taking ~30 secs with S3 as rootdir.
Also adding the code flow from my IDE for reference
There was a problem hiding this comment.
@wchevreuil Was i able to answer what you asked for?
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
No description provided.