GRPC call fail with outstanding reads

[himanshu-patel-dev]

While fetching data using Google Ads Ruby Client - v18 I see the GRPC Streaming fails with grpc_call_start_batch failed with outstanding read or write present (code=8) (GRPC::Core::CallError)?

Version: v18
LIb: google-ads-googleads (31.0.0)

This happens only when the data pulled for an account is large (like more than a 1mil row). For other accounts with lesser data, I don't face the issue (same client code).

Query:

 SELECT asset.id, ad_group_ad.ad.type, ad_group_ad_asset_view.resource_name, ad_group_ad_asset_view.performance_label, ad_group_ad_asset_view.asset, asset.type, asset.text_asset.text, asset.sitelink_asset.link_text, asset.sitelink_asset.description1, asset.sitelink_asset.description2, asset.structured_snippet_asset.header, asset.structured_snippet_asset.values, asset.call_to_action_asset.call_to_action, asset.name, asset.youtube_video_asset.youtube_video_title, ad_group_ad.ad.demand_gen_carousel_ad.description, ad_group_ad.ad.demand_gen_carousel_ad.headline, ad_group_ad.ad.demand_gen_carousel_ad.call_to_action_text, asset.image_asset.full_size.url, asset.youtube_video_asset.youtube_video_id, asset.demand_gen_carousel_card_asset.square_marketing_image_asset, asset.demand_gen_carousel_card_asset.portrait_marketing_image_asset, asset.demand_gen_carousel_card_asset.marketing_image_asset, asset.image_asset.full_size.height_pixels, asset.image_asset.full_size.width_pixels
   FROM ad_group_ad_asset_view
  WHERE ad_group_ad.ad.type in ('RESPONSIVE_SEARCH_AD', 'APP_AD', 'DEMAND_GEN_CAROUSEL_AD', 'DEMAND_GEN_MULTI_ASSET_AD', 'DEMAND_GEN_VIDEO_RESPONSIVE_AD')
    AND asset.type in ('TEXT', 'IMAGE', 'YOUTUBE_VIDEO', 'CALL_TO_ACTION', 'DEMAND_GEN_CAROUSEL_CARD', 'SITELINK', 'STRUCTURED_SNIPPET', 'UNKNOWN')
    AND segments.date
BETWEEN '2025-01-13'
    AND '2025-01-13'
    AND metrics.impressions > 0

Logs:

W, [2025-01-14T08:36:27.928456 #296]  WARN -- : CID: <customer_id>, Host: [googleads.googleapis.com:443](http://googleads.googleapis.com:443/), Method: /google.ads.googleads.v16.services.GoogleAdsService/SearchStream, IsFault: yes, Request ID: N/A
I, [2025-01-14T08:36:27.928653 #296]  INFO -- : Outgoing request: Headers: {"developer-token":"REDACTED","login-customer-id":"3661020287","x-goog-api-client":"gl-ruby/3.1.4 gccl/27.0.0 gax/0.21.1 gapic/27.0.0 grpc/1.62.0 pb/3.25.5","x-goog-request-params":"customer_id=<customer_id>"} Payload: {"customerId":"<customer_id>","query":" SELECT asset.id, ad_group_ad.ad.type, ad_group_ad_asset_view.resource_name, ad_group_ad_asset_view.performance_label, ad_group_ad_asset_view.asset, asset.type, asset.text_asset.text, asset.sitelink_asset.link_text, asset.sitelink_asset.description1, asset.sitelink_asset.description2, asset.structured_snippet_asset.header, asset.structured_snippet_asset.values, asset.call_to_action_asset.call_to_action, [asset.name](http://asset.name/), asset.youtube_video_asset.youtube_video_title, ad_group_ad.ad.discovery_carousel_ad.description, ad_group_ad.ad.discovery_carousel_ad.headline, ad_group_ad.ad.discovery_carousel_ad.call_to_action_text, asset.image_asset.full_size.url, asset.youtube_video_asset.youtube_video_id, asset.discovery_carousel_card_asset.square_marketing_image_asset, asset.discovery_carousel_card_asset.portrait_marketing_image_asset, asset.discovery_carousel_card_asset.marketing_image_asset, asset.image_asset.full_size.height_pixels, asset.image_asset.full_size.width_pixels\n   FROM ad_group_ad_asset_view\n  WHERE ad_group_ad.ad.type in ('RESPONSIVE_SEARCH_AD', 'APP_AD', 'DISCOVERY_CAROUSEL_AD', 'DISCOVERY_MULTI_ASSET_AD', 'DISCOVERY_VIDEO_RESPONSIVE_AD')\n    AND asset.type in ('TEXT', 'IMAGE', 'YOUTUBE_VIDEO', 'CALL_TO_ACTION', 'DISCOVERY_CAROUSEL_CARD', 'SITELINK', 'STRUCTURED_SNIPPET', 'UNKNOWN')\n    AND segments.date\nBETWEEN '2025-01-13'\n    AND '2025-01-13'\n    AND metrics.impressions > 0\n AND campaign.id IN (<comma separated campaign ids>)"}
I, [2025-01-14T08:36:27.928759 #296]  INFO -- : Incoming response (errors):

  GRPC::Core::CallError(grpc_call_start_batch failed with outstanding read or write present (code=8)):
  called from: /home/deploy/.rvm/gems/ruby-3.1.4/gems/activesupport-6.1.7.8/lib/active_support/logger_thread_safe_level.rb:67:in `add'
/home/deploy/.rvm/gems/ruby-3.1.4/gems/grpc-1.62.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:169:in `run_batch': grpc_call_start_batch failed with outstanding read or write present (code=8) (GRPC::Core::CallError)
/home/deploy/.rvm/gems/ruby-3.1.4/gems/grpc-1.62.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:169:in `run_batch': grpc_call_start_batch failed with outstanding read or write present (code=8) (GRPC::Core::CallError)

When I do same query via cURL, I get data from API with next_page token. That means there is nothing wrong with query or credentials.

curl -f --request POST "https://googleads.googleapis.com/v18/customers/<CUSTOMER_ID>/googleAds:search" --header "Content-Type: application/json" --header "developer-token: <DEVELOPER_TOKEN>" --header "login-customer-id: <LOGIN_CUSTOMER_ID>" --header "Authorization: Bearer <TOKEN>" --data '{ 
"query": "
 SELECT asset.id, ad_group_ad.ad.type, ad_group_ad_asset_view.resource_name, ad_group_ad_asset_view.performance_label, ad_group_ad_asset_view.asset, asset.type, asset.text_asset.text, asset.sitelink_asset.link_text, asset.sitelink_asset.description1, asset.sitelink_asset.description2, asset.structured_snippet_asset.header, asset.structured_snippet_asset.values, asset.call_to_action_asset.call_to_action, asset.name, asset.youtube_video_asset.youtube_video_title, ad_group_ad.ad.demand_gen_carousel_ad.description, ad_group_ad.ad.demand_gen_carousel_ad.headline, ad_group_ad.ad.demand_gen_carousel_ad.call_to_action_text, asset.image_asset.full_size.url, asset.youtube_video_asset.youtube_video_id, asset.demand_gen_carousel_card_asset.square_marketing_image_asset, asset.demand_gen_carousel_card_asset.portrait_marketing_image_asset, asset.demand_gen_carousel_card_asset.marketing_image_asset, asset.image_asset.full_size.height_pixels, asset.image_asset.full_size.width_pixels
   FROM ad_group_ad_asset_view
  WHERE ad_group_ad.ad.type in (\"RESPONSIVE_SEARCH_AD\", \"APP_AD\", \"DEMAND_GEN_CAROUSEL_AD\", \"DEMAND_GEN_MULTI_ASSET_AD\", \"DEMAND_GEN_VIDEO_RESPONSIVE_AD\")
    AND asset.type in (\"TEXT\", \"IMAGE\", \"YOUTUBE_VIDEO\", \"CALL_TO_ACTION\", \"DEMAND_GEN_CAROUSEL_CARD\", \"SITELINK\", \"STRUCTURED_SNIPPET\", \"UNKNOWN\")
    AND segments.date
BETWEEN \"2025-01-13\"
    AND \"2025-01-13\"
    AND metrics.impressions > 0
"
}'

Same cURL query I tried with Ruby Client and end up with error.

3.1.4 :112 > response = api_client.get_google_ads_service_client.search(customer_id:, query:).to_a


W, [2025-01-20T13:14:00.049667 #337]  WARN -- : CID: <CUSTOMER_ID>, Host: googleads.googleapis.com:443, Method: /google.ads.googleads.v18.services.GoogleAdsService/Search, IsFault: yes, Request ID:
I, [2025-01-20T13:14:00.049888 #337]  INFO -- : Outgoing request: Headers: {"developer-token":"REDACTED","login-customer-id":"<CUSTOMER_ID>","x-goog-api-client":"gl-ruby/3.1.4 gccl/31.0.0 gax/0.21.1 gapic/31.0.0 grpc/1.62.0 pb/3.25.5","x-goog-request-params":"customer_id=<CUSTOMER_ID>"} Payload: {"customerId":"<CUSTOMER_ID>","query":"\n SELECT asset.id, ad_group_ad.ad.type, ad_group_ad_asset_view.resource_name, ad_group_ad_asset_view.performance_label, ad_group_ad_asset_view.asset, asset.type, asset.text_asset.text, asset.sitelink_asset.link_text, asset.sitelink_asset.description1, asset.sitelink_asset.description2, asset.structured_snippet_asset.header, asset.structured_snippet_asset.values, asset.call_to_action_asset.call_to_action, asset.name, asset.youtube_video_asset.youtube_video_title, ad_group_ad.ad.demand_gen_carousel_ad.description, ad_group_ad.ad.demand_gen_carousel_ad.headline, ad_group_ad.ad.demand_gen_carousel_ad.call_to_action_text, asset.image_asset.full_size.url, asset.youtube_video_asset.youtube_video_id, asset.demand_gen_carousel_card_asset.square_marketing_image_asset, asset.demand_gen_carousel_card_asset.portrait_marketing_image_asset, asset.demand_gen_carousel_card_asset.marketing_image_asset, asset.image_asset.full_size.height_pixels, asset.image_asset.full_size.width_pixels\n   FROM ad_group_ad_asset_view\n  WHERE ad_group_ad.ad.type in (\"RESPONSIVE_SEARCH_AD\", \"APP_AD\", \"DEMAND_GEN_CAROUSEL_AD\", \"DEMAND_GEN_MULTI_ASSET_AD\", \"DEMAND_GEN_VIDEO_RESPONSIVE_AD\")\n    AND asset.type in (\"TEXT\", \"IMAGE\", \"YOUTUBE_VIDEO\", \"CALL_TO_ACTION\", \"DEMAND_GEN_CAROUSEL_CARD\", \"SITELINK\", \"STRUCTURED_SNIPPET\", \"UNKNOWN\")\n    AND segments.date\nBETWEEN \"2025-01-13\"\n    AND \"2025-01-13\"\n    AND metrics.impressions > 0\n"}
I, [2025-01-20T13:14:00.050004 #337]  INFO -- : Incoming response (errors):
  GRPC::DeadlineExceeded(4:Deadline Exceeded. debug_error_string:{UNKNOWN:Error received from peer  {created_time:"2025-01-20T13:14:00.048977631+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}}):
  called from: /home/deploy/.rvm/gems/ruby-3.1.4/gems/activesupport-6.1.7.8/lib/active_support/logger_thread_safe_level.rb:67:in `add'
/home/deploy/.rvm/gems/ruby-3.1.4/gems/gapic-common-0.21.1/lib/gapic/grpc/service_stub/rpc_call.rb:127:in `rescue in call': 4:4:Deadline Exceeded. debug_error_string:{UNKNOWN:Error received from peer  {created_time:"2025-01-20T13:14:00.048977631+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}} (Gapic::GRPC::DeadlineExceededError)
/home/deploy/.rvm/gems/ruby-3.1.4/gems/grpc-1.62.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:29:in `check_status': 4:Deadline Exceeded. debug_error_string:{UNKNOWN:Error received from peer  {created_time:"2025-01-20T13:14:00.048977631+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}} (GRPC::DeadlineExceeded)

A Quick Snippet to reproduce the error

require 'google/ads/google_ads'
require 'googleauth'

client = Google::Ads::GoogleAds::GoogleAdsClient.new do |config|
  config.refresh_token = "TOKEN"
  config.client_id = "ID"
  config.client_secret = "SECRET"
  config.developer_token = "DEV_TOKEN"
  config.login_customer_id = "LOGIN_CUSTOMER_ID"
end
customer_id = 'CUSTOMER_ID'
query = "
 SELECT asset.id, ad_group_ad.ad.type, ad_group_ad_asset_view.resource_name, ad_group_ad_asset_view.performance_label, ad_group_ad_asset_view.asset, asset.type, asset.text_asset.text, asset.sitelink_asset.link_text, asset.sitelink_asset.description1, asset.sitelink_asset.description2, asset.structured_snippet_asset.header, asset.structured_snippet_asset.values, asset.call_to_action_asset.call_to_action, asset.name, asset.youtube_video_asset.youtube_video_title, ad_group_ad.ad.demand_gen_carousel_ad.description, ad_group_ad.ad.demand_gen_carousel_ad.headline, ad_group_ad.ad.demand_gen_carousel_ad.call_to_action_text, asset.image_asset.full_size.url, asset.youtube_video_asset.youtube_video_id, asset.demand_gen_carousel_card_asset.square_marketing_image_asset, asset.demand_gen_carousel_card_asset.portrait_marketing_image_asset, asset.demand_gen_carousel_card_asset.marketing_image_asset, asset.image_asset.full_size.height_pixels, asset.image_asset.full_size.width_pixels
   FROM ad_group_ad_asset_view
  WHERE ad_group_ad.ad.type in (\"RESPONSIVE_SEARCH_AD\", \"APP_AD\", \"DEMAND_GEN_CAROUSEL_AD\", \"DEMAND_GEN_MULTI_ASSET_AD\", \"DEMAND_GEN_VIDEO_RESPONSIVE_AD\")
    AND asset.type in (\"TEXT\", \"IMAGE\", \"YOUTUBE_VIDEO\", \"CALL_TO_ACTION\", \"DEMAND_GEN_CAROUSEL_CARD\", \"SITELINK\", \"STRUCTURED_SNIPPET\", \"UNKNOWN\")
    AND segments.date
BETWEEN \"2025-01-13\"
    AND \"2025-01-13\"
    AND metrics.impressions > 0
"

# Get the Google Ads Service

# Stream the results by using `search_stream`
stream = client.service.google_ads.search_stream(customer_id: customer_id, query: query)

# Handle the streamed results in chunks
data = []
stream.each do |batch|
  batch.results.each do |row|
    data << row
  end
end

/home/deploy/.rvm/gems/ruby-3.1.4/gems/grpc-1.62.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:169:in `run_batch': grpc_call_start_batch failed with outstanding read or write present (code=8) (GRPC::Core::CallError)

This Ad Account has many Ads (8M+) and fetching all of them always throws the above error. This behavior is not present for any other account, or even in the same account when we fetch a limited number of Ads we don't face this error.
I suspect, on fetching large amount of data, the server exhaust it's limit (file system, memory etc) and thus end up throwing this GRPC error (code = 8, RESOURCE_EXHAUSTED).

[mcloonan]

It looks like it's from one of our dependencies, not something that we can fix in our lib. I've reached out to one of the owners there for next steps. Thanks for reporting.

[mcloonan]

I missed the fact that this was RESOURCE_EXHAUSTED in my first look. You may be able to solve this by modifying the timeouts to allow the server more time to respond if you expect the result to be large. You can see an example of how to do that here: https://github.com/googleads/google-ads-ruby/blob/main/examples/misc/set_custom_client_timeouts.rb

You may have to try a few different values to get one that works. The 5 minutes shown in that example is clearly going to be too short.