Enable Streaming from SFTP to GCS in `SFTPToGCSOperator`

[rynmccrmck]

Description

The current SFTPToGCSOperator in Airflow does not support streaming objects directly from SFTP to Google Cloud Storage (GCS). This design poses a challenge since both Composer and Airflow best practices recommend against assigning large amounts of memory to workers. Transferring sizeable files from SFTP to GCS without streaming can potentially max out worker memory.

Use case/motivation

Prevent worker memory exhaustion, especially when dealing with large files.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[hegongwork]

hi @ferruzzi
yesterday in my project I wrote this code and my colleague @rynmccrmck told me I could added the code to Airflow and this PR is his draft for me. I will create a same PR.

[ferruzzi]

I'll assign the issue to you then. let me know if you need any help

[hegongwork]

update: I wrote a PR in my own repo: https://github.com/he-gong-sardine/airflow/pull/1/files
but I haven't set up airflow environment so I cannot test it.
I will send PR after I test it locally.

[Player00liv]

Vhuut

[elaye-canopy]

@he-gong-sardine did you get a chance to test it?
I am encountering a similar issue and was in the process of testing a fix independent of yours (I will also upload it later for comparison), but if yours already works I might just go with yours.
I've also left a comment (well, a question, but not a criticism) on your PR.

Also, not directly related but logically related - perhaps similar logic should be made for the converse operator as well (GCS to SFTP)