-
Notifications
You must be signed in to change notification settings - Fork 30
Description
-
The CPS source connector calls GetSubscription to check for the existence + auth access to a subscription. The returned Subscription object is not used at all. This requires the user to grant the Pub/Sub Viewer role to the principal, in addition to the Pub/Sub Subscriber role.
-
Whether a subscription exists or not is already checked when we do the actually Pulls. Other client libraries (like the Python one) do not require this extra call. It's possible the Subscription could be useful for extra behavior - but here it's not used for anything.
-
The BQ sink connector does not check for BQ table existence upon startup. That's because it can be configured to create tables.
-
The GCS sink connector does not check for GCS bucket existence upon startup. It only creates blobs, not entire buckets. I expect this connector will fail upon first flush to the bucket, if the bucket does not exist.
-
The CPS sink connector does not check for CPS Topic existence upon startup. It creates a publisher at task startup. Task startup is separate from connector start up and happens afterwards, so I don't think the entire connector will necessarily fail to start up. I see a case for making the publisher at connector startup, so connector creation doesn't return success, only for task startup to ƒail later.
-
The Debezium Spanner source/CDC connector does not connect to Spanner upon startup. That happens at task startup.
Given that the problem of connector startup errors vs task startup errors is common in the ecosystem, we could just remove the GetSubscription check in the CPS source connector.