HADOOP-19354. S3AInputStream to be created by factory under S3AStore. #7237
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
First iteration * Factory interface with a parameter object creation method * Base class AbstractS3AInputStream for all streams to create * S3AInputStream subclasses that and has a factory * Production and test code to use it Not done * Input stream callbacks pushed down to S3Store * S3Store to dynamically choose factory at startup, stop in close() * S3Store to implement the factory interface, completing final binding operations (callbacks, stats) Change-Id: I8d0f86ca1f3463d4987a43924f155ce0c0644180
Revision API: Make clear this is part of the fundamental store Model: * abstract stream class is now ObjectInputStream * interface is ObjectInputStreamFactory * move to package org.apache.hadoop.fs.s3a.impl.model Implementation: Prefetching stream is created this way too; adds one extra parameter. Maybe we should pass conf down too Change-Id: I5bbb5dfe585528b047a649b6c82a9d0318c7e91e
|
💔 -1 overall
This message was automatically generated. |
There was a problem hiding this comment.
quick glance at my work from november (which I've forgotten already)
This should go into the new branch to give the option of changing things in a feature branch where patching it is possible -as we don't know exactly what is needed for the new work. Anyone going near the streams and the store just needs to know things will change in the near future
| @@ -1560,6 +1560,11 @@ private Constants() { | |||
| */ | |||
| public static final String AWS_AUTH_CLASS_PREFIX = "com.amazonaws.auth"; | |||
|
|
|||
|
|
|||
| public static final String INPUT_STREAM_TYPE = "fs.s3a.input.stream.type"; | |||
| return value; | ||
| } | ||
| } | ||
| LOG.warn("Unknown input stream type {}, using default classic stream.", inputStreamType); |
There was a problem hiding this comment.
log once at info. we aren't the AWS SDK here
|
Pulling to my PR. @ahmarsuhail can you be a bit more detailed on commit messages, and include the JIRA ID. now my commit log has the "initial commit" text in. Which I can't edit as it'll break the commit iD and so make merging my followup changes into your pR harder. thanks |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
This PR makes some additional changes to the initial PR from Steve: #7214.
Not sure if I got all of this right, but here are a few callouts:
Thought of moving InputStreamCallbacks into S3AStore, but I don't think that is the right place for it. InputStreamCallBacks uses
S3AFileSystemOperations, which usesS3AStore, so you end up in a dependency mess. Instead, move them to a separate classInputStreamCallbacksImpl, which keeps the code out of S3AFileSystem. Maybe there is a better way to do this, but I couldn't think of anything.Adds a new config,
fs.s3a.input.stream.type. This can be set toclassic,prefetch,analytics. Believe this is better than having multipleprefetch.enabledandanalytics.enabledflags.Could not figure out what was meant by "S3Store to implement the factory interface, completing final binding operations (callbacks, stats)" in Steve's original PR, let's discuss.
Let's merge this into trunk first, and then rebase https://github.com/apache/hadoop/tree/feature-HADOOP-19363-analytics-accelerator-s3 on top of it, and move the analytics stream creation code into the new factory.
This is a draft PR, was just attempting to complete the original PR.
How was this patch tested?
Not tested. Whoever picks this up for completion can test!