[source-google-drive] Unable to create new connector

[anchitbhatia]

Hi everyone, I am using open source airbyte and trying to create a google drive source. But i keep getting the following error:
Configuration check failed

'No streams are available for source SourceGoogleDrive. This is probably an issue with the connector. Please verify that your configuration provides permissions to list and read files from the source. Contact support if you are unable to resolve this issue.'

I am using Google service account for authentication. I have already verified that the service account has access to the google drive folder and the service key JSON is valid as i am able to use it to create a google sheets source. How can i fix this issue? Any help is appreciated. Thank you!

[rootless4real]

I'm having the same issue. Using the Airbytes Docker instance. Is there any solution for this problem?

Thank you.

[rwask]

Can you please send some more information regarding the configuration of the Google Drive Source connector if you are still having issues with this @anchitbhatia and @rootless4real?

Specifically a screenshot of the source configuration (including the streams config) and also give us an idea of what file types you intend to sync including the folder structure if possible in the GDrive?

[rootless4real]

I'm no longer having issues. You guys fixed it for me. I'm syncing pdf files.

…

On January 30, 2024 3:30:15 PM MST, Ryan Waskewich ***@***.***> wrote: Can you please send some more information regarding the configuration of the Google Drive Source connector if you are still having issues with this @anchitbhatia and @rootless4real? Specifically a screenshot of the source configuration (including the streams config) and also give us an idea of what file types you intend to sync including the folder structure if possible in the GDrive? -- Reply to this email directly or view it on GitHub: #33850 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[Mahmoud-Elbahnasawy]

How was it resolved.

[rwask]

Thanks @rootless4real.

@anchitbhatia, if you are still having issues please check the above reply and pass on that information and we will take a look! I will close this out meanwhile.

[vladaman]

We are having the same issue with S3 connector. Airbyte is running using Docker Compose.

2024-02-20 13:59:43 platform > Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@4aaff919[status=failed,message='No streams are available for source SourceS3. This is probably an issue with the connector. Please verify that your configuration provides permissions to list and read files from the source. Contact support if you are unable to resolve this issue.',additionalProperties={}]

[jmgayosso]

Hi @anchitbhatia @rwask @rootless4real I am having the same issue, could you help me please?

[rootless4real]

It is difficult to troubleshoot without the "Logs" portion visible. Please post with that visible. It will also help you to better understand what is happening/failing.

[jmgayosso]

I got it, thanks for replay I see this when expands Logs

How can I find the logs file? I am using the open source version with docker

[rootless4real]

Well, that is terse. Have you verified the permissions on the Service Account Key? Beyond that, I'm not sure what it would be. You will likely have to wait for someone from Airbyte to respond. I do wonder if there is more information in the log file that is being pointed to.

…

On Fri, Apr 19, 2024 at 3:17 PM Jose Maria ***@***.***> wrote: I got it, thanks for replay I see this when expands Logs Screenshot.2024-04-19.at.4.16.02.p.m.png (view on web) <https://github.com/airbytehq/airbyte/assets/58195358/f74f9d8c-9b23-438e-8b73-3069e463d588> How can I find the logs file? I am using the open source version with docker — Reply to this email directly, view it on GitHub <#33850 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHSKIPSRJ3TKLP2HOJBOFTY6GJYVAVCNFSM6AAAAABBHHS3KWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCNZRGMZDE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- James Fowler/方志宇 ***@***.*** ***@***.*** Public Key: https://sites.google.com/site/poorpermissions/pk/fowler_publickey?attredirects=0&d=1 <http://rootless.googlepages.com/fowler_publickey>

[mmicikevicius]

We are having the same issue with OneDrive connector. I am using open source airbyte. I know that the credentials are correct, since with the same details Microsoft Teams connector works fine.

Log:
2024-04-25 07:37:50 �[46mplatform�[0m > Docker volume job log path: /tmp/workspace/c2f6c357-d31f-46ed-8661-2e0d29de899a/0/logs.log
2024-04-25 07:37:50 �[46mplatform�[0m > Executing worker wrapper. Airbyte version: 0.50.43
2024-04-25 07:37:50 �[46mplatform�[0m > Attempt 0 to save workflow id for cancellation
2024-04-25 07:37:50 �[46mplatform�[0m >
2024-04-25 07:37:50 �[46mplatform�[0m > ----- START CHECK -----
2024-04-25 07:37:50 �[46mplatform�[0m >
2024-04-25 07:37:50 �[46mplatform�[0m > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-04-25 07:37:50 �[46mplatform�[0m > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-04-25 07:37:50 �[46mplatform�[0m > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-04-25 07:37:50 �[46mplatform�[0m > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-04-25 07:37:50 �[46mplatform�[0m > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-04-25 07:37:50 �[46mplatform�[0m > Checking if airbyte/source-microsoft-onedrive:0.1.0 exists...
2024-04-25 07:37:50 �[46mplatform�[0m > airbyte/source-microsoft-onedrive:0.1.0 was found locally.
2024-04-25 07:37:50 �[46mplatform�[0m > Creating docker container = source-microsoft-onedrive-check-c2f6c357-d31f-46ed-8661-2e0d29de899a-0-zxjcw with resources io.airbyte.config.ResourceRequirements@3bafd6ab[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2024-04-25 07:37:50 �[46mplatform�[0m > Preparing command: docker run --rm --init -i -w /data/c2f6c357-d31f-46ed-8661-2e0d29de899a/0 --log-driver none --name source-microsoft-onedrive-check-c2f6c357-d31f-46ed-8661-2e0d29de899a-0-zxjcw --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-microsoft-onedrive:0.1.0 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.43 -e WORKER_JOB_ID=c2f6c357-d31f-46ed-8661-2e0d29de899a airbyte/source-microsoft-onedrive:0.1.0 check --config source_config.json
2024-04-25 07:37:50 �[46mplatform�[0m > Reading messages from protocol version 0.2.0
2024-04-25 07:37:51 �[46mplatform�[0m > Check failed
2024-04-25 07:37:51 �[46mplatform�[0m > Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@5bb4ad6d[status=failed,message='No streams are available for source SourceMicrosoftOneDrive. This is probably an issue with the connector. Please verify that your configuration provides permissions to list and read files from the source. Contact support if you are unable to resolve this issue.',additionalProperties={}]
2024-04-25 07:37:51 �[46mplatform�[0m >
2024-04-25 07:37:51 �[46mplatform�[0m > ----- END CHECK -----
2024-04-25 07:37:51 �[46mplatform�[0m >

[rewelle-dic]

Hi @jmgayosso , @mmicikevicius I am having the same issue with open source sharepoint connector.
How did you find the name of the streams to add in the first place?

My credentials are good, I can list the lists, drives and libraries in calling MSgraph API directly via postman.

Any suggestions?

[mous222]

@rewelle-dic , you can try adding a stream to specify the type of document you want to include, such as setting it to 'Document.' I encountered a similar error with Google Drive Folder, and this resolved the issue for me. Hope this helps!

[rewelle-dic]

Thanks @mous222,
This is what I have when I try adding a stream.

But Credentials provided have full read access in graph API (Files.Read.All and Sites.Read.All).
Testing via Postman, the application can list and read lists, drives and libraries successfully.

Also In the connector documentation here: https://docs.airbyte.com/integrations/sources/microsoft-sharepoint#step-1-set-up-sharepoint-application, the point 5. Enter Drive Name, but I can't find in the interface where to enter the Drive Name.

Any suggestions?

[mikobits]

I am also having the same issue. I have written a python script that uses the exact same JSON credentials to look in the file folder. I did fix an error that the service account email did not have explicit share access to the folder, but now my python script sees files in the folder.

So now that I confirmed that this python script sees files in the folder using the credential, but the airbyte connector to Google Drive does not see any objects to stream.

Please help!

[mikobits]

[Elsayed91]

Also facing this issue with SharePoint as well as OneDrive connectors.

[gpetrucci93]

I have the same scenario, the connector with sharepoint does not work at all 2024-10-02 14:36:01 platform > Docker volume job log path: /tmp/workspace/45243053-e5b6-4568-875e-9c57a1919b76/0/logs.log
2024-10-02 14:36:01 platform > Executing worker wrapper. Airbyte version: 0.50.48
2024-10-02 14:36:01 platform > Attempt 0 to save workflow id for cancellation
2024-10-02 14:36:01 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-10-02 14:36:01 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-10-02 14:36:01 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-10-02 14:36:01 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-10-02 14:36:01 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-10-02 14:36:01 platform >
2024-10-02 14:36:01 platform > ----- START CHECK -----
2024-10-02 14:36:01 platform >
2024-10-02 14:36:01 platform > Checking if airbyte/source-microsoft-sharepoint:0.5.2 exists...
2024-10-02 14:36:01 platform > airbyte/source-microsoft-sharepoint:0.5.2 was found locally.
2024-10-02 14:36:01 platform > Creating docker container = source-microsoft-sharepoint-check-45243053-e5b6-4568-875e-9c57a1919b76-0-xpotk with resources io.airbyte.config.ResourceRequirements@2de27ca8[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2024-10-02 14:36:01 platform > Preparing command: docker run --rm --init -i -w /data/45243053-e5b6-4568-875e-9c57a1919b76/0 --log-driver none --name source-microsoft-sharepoint-check-45243053-e5b6-4568-875e-9c57a1919b76-0-xpotk --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-microsoft-sharepoint:0.5.2 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE=dev -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.48 -e WORKER_JOB_ID=45243053-e5b6-4568-875e-9c57a1919b76 airbyte/source-microsoft-sharepoint:0.5.2 check --config source_config.json
2024-10-02 14:36:01 platform > Reading messages from protocol version 0.2.0
2024-10-02 14:36:10 platform > Check failed
2024-10-02 14:36:11 platform > Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@2e3a7ce0[status=failed,message='No streams are available for source SourceMicrosoftSharePoint. This is probably an issue with the connector. Please verify that your configuration provides permissions to list and read files from the source. Contact support if you are unable to resolve this issue.',additionalProperties={}]
2024-10-02 14:36:11 platform >
2024-10-02 14:36:11 platform > ----- END CHECK -----
2024-10-02 14:36:11 platform >