-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement Async for EMR Operators and Sensors #120
Comments
Both EmrCreateJobFlowOperator and EmrAddStepsOperator does not require the async implementation as an Operator. If you look at the boto3 documentation for run_job_flow, it instantaneously returns the response with Similarly in the |
Yes I agree, I think only the below sensors will take a while to run and will need async implementation
|
Is this completed, I don't see a PR for the following
|
I am working on |
I have started working |
PR opened |
Please look at docs to see if there are other operators we should make Async (Principle - Only create Async operators for the “sync-version” of operators that do some level of polling; take more than a few seconds to complete) for the EMR
Implement async versions for the following operators (Aligned on descending order of priority)
EmrCreateJobFlowOperator
EmrAddStepsOperator
EmrContainerOperator
EmrModifyClusterOperator
EmrTerminateJobFlowOperator
EmrContainerSensor
-> @bharanidharan14EmrJobFlowSensor
-> @bharanidharan14EmrStepSensor
-> @pankajastroAcceptance Criteria:
The text was updated successfully, but these errors were encountered: