Failed to open TCP connection - new item create/edit

[telezoic]

Hi again,

I'm setting up a new production build (DigitalOcean)

manager - v2.12.11
worker - v2.12.16

and I when I create/edit an 'object' (like a new parser, datasource, or contributor) in the manager, I consistently get a "We're sorry, something went wrong page" , but I can trim the url in the browser and go back and see that the item it created/edited does exist, what's more, I can use the item I created/edit. I thought this might be a route error but my manager log tells me there's a connection failure . . .

{"name":"rails","hostname":"SolumAuxiliator","pid":47743,"level":60,"time":"2022-06-02T21:44:19.981+00:00","v":0,"msg":"[6fd44b6c-546f-4611-9dc2-996e67f24e27] {:msg=>\"  \\n[6fd44b6c-546f-4611-9dc2-996e67f24e27] Errno::ECONNREFUSED (Failed to open TCP connection to 127.0.0.1:3000 (Connection refused - connect(2) for \\\"127.0.0.1\\\" port 3000)):\\n[6fd44b6c-546f-4611-9dc2-996e67f24e27]   \\n[6fd44b6c-546f-4611-9dc2-996e67f24e27] app/api/api/request.rb:45:in `execute'\\n[6fd44b6c-546f-4611-9dc2-996e67f24e27] app/api/api/request.rb:20:in `post'\\n[6fd44b6c-546f-4611-9dc2-996e67f24e27] app/api/api/partner.rb:6:in `post'\\n[6fd44b6c-546f-4611-9dc2-996e67f24e27] app/models/partner.rb:25:in `block in update_apis'\\n[6fd44b6c-546f-4611-9dc2-996e67f24e27] app/models/partner.rb:24:in `each'\\n[6fd44b6c-546f-4611-9dc2-996e67f24e27] app/models/partner.rb:24:in `update_apis'\\n[6fd44b6c-546f-4611-9dc2-996e67f24e27] app/controllers/parsers_controller.rb:65:in `update'\"}"}

to what looks like it should be the api at 127.0.0.1:3000 if I were running it in dev. My application.ymls all point to the API domain:

manager /config/application.yml

production:
  HOST: https://manager.auxiliator-digitalbc.ca
  WORKER_HOST: https://worker.auxiliator-digitalbc.ca
  WORKER_KEY: 959933ea2ead4b73a6d8415b4ddad7b931324501a379a9197ca8b38b90eb83f2b1d61bcbded14ddc8820d63f49f700fd6d8294f28e99b5bbcd71b960ab8aec29 
  API_HOST: https://api.auxiliator-digitalbc.ca
  MONGO_HOSTS: mongodb+srv://corymbusaux.d0rvn.mongodb.net:27017
  HARVESTER_API_KEY: LxPjQyKyBmo-emc7ATBL
  OTP_SECRET_KEY: YOUROTPSECRETKEY
  MFA_ENABLED: 'false'

worker /config/application.yml

production:
  HOST: 'https://worker.auxiliator-digitalbc.ca'
  API_HOST: 'https://api.auxiliator-digitalbc.ca'
  MANAGER_HOST: 'https://manager.auxiliator-digitalbc.ca'
  HARVESTER_API_KEY: LxPjQyKyBmo-emc7ATBL
  HARVESTER_CACHING_ENABLED: "true"
  LINK_CHECKING_ENABLED: "true"
  LINK_CHECKING_INTERVAL: 6
  LINKCHECKER_RECIPIENTS: "tech@digitalbc.ca"
  LINKCHECKER_EMAIL: "linkchecker@digitalbc.ca"
  SECRET_KEY_BASE: 'some long hash value' 
  DEVISE_MAILER: "info@digitalbc.ca"
  WORKER_KEY: 959933ea2ead4b73a6d8415b4ddad7b931324501a379a9197ca8b38b90eb83f2b1d61bcbded14ddc8820d63f49f700fd6d8294f28e99b5bbcd71b960ab8aec29 
  REDIS_URL: 'redis://localhost:6379/2'

api /config/application.yml

defaults: &defaults
  SOLR_PING: 'http://localhost:8982/solr/admin/ping'
  HTTP_HOST: 'https://api.auxiliator-digitalbc.ca'
  WORKER_API_URL: 'https://worker.auxiliator-digitalbc.ca'
  WWW_DOMAIN: 'www.dev'
  DEVISE_MAILER: 'tech@digitalbc.ca'
  RESQUE_USER: 'admin'
  RESQUE_PASS: 'password'
  HARVESTER_IPS: '127.0.0.1'
  REQUEST_LIMIT_MAILER: 'tech@digitalbc.ca'
  BLACKLIST_MAILER_TO: 'tech@digitalbc.ca'
  BLACKLIST_MAILER_FROM: 'tech@digitalbc.ca'
 # HTTP_HOST: 'http://test.host'
  MONGO_HOSTS: 'mongodb+srv://corymbusaux.d0rvn.mongodb.net:27017'
  SECRET_TOKEN: '456c9e5b80492fd22bbfba1c1a335913e12d9b06'
  WORKER_KEY: 959933ea2ead4b73a6d8415b4ddad7b931324501a379a9197ca8b38b90eb83f2b1d61bcbded14ddc8820d63f49f700fd6d8294f28e99b5bbcd71b960ab8aec29
  
  production:
  <<: *defaults

and of course my apache vhost has the HAproxy ref:

        ProxyPass / http://api.auxiliator-digitalbc.ca:81/ retry=0
       ProxyPassReverse / http://api.auxiliator-digitalbc.ca:81/

I can't locate any reference to 127.0.0.1:3000 in the app to see where this would be coming from, and I'm wondering how this info is generated in the hope that I can correct it.

Thanks for any assistance you could render.

[paul-mesnilgrente]

I can see 2 possible causes at the moment. Could you check you have the right application.yml in your manager production setup?

Also the manager should be running in production mode, ie: the environment variable RAILS_ENV should be equal to production.

I don't think this is related by you in api's application.yml. The production key has a wrong indentation.

Could you provide the code you have in config/application.rb in production?

[telezoic]

Thanks @paul-mesnilgrente.

My manager production.yml looks like:

production:
  HOST: https://manager.auxiliator-digitalbc.ca
  WORKER_HOST: https://worker.auxiliator-digitalbc.ca
  WORKER_KEY: 959933ea2ead4b73a6d8415b4ddad7b931324501a379a9197ca8b38b90eb83f2b1d61bcbded14ddc8820d63f49f700fd6d8294f28e99b5bbcd71b960ab8aec29
  API_HOST: https://api.auxiliator-digitalbc.ca
  MONGO_HOSTS: mongodb+srv://corymbusaux.d0rvn.mongodb.net:27017
  HARVESTER_API_KEY: LxPjQyKyBmo-emc7ATBL
  OTP_SECRET_KEY: YOUROTPSECRETKEY
  MFA_ENABLED: 'false'


development:
  WORKER_HOST: http://127.0.0.1:3002
  WORKER_KEY: 962c466b97d79d861f9dc03c350cc914bb9e9e4a57cbf80ac17e5273089e95d744ded9d021a170194805b7aa7f78771167c10346c2852b5bf05454e17f093fb0
  API_HOST: http://127.0.0.1:3000
  HARVESTER_API_KEY: huVPSc8H5uG4V3LGsoRM
  OTP_SECRET_KEY: YOUROTPSECRETKEY
  MFA_ENABLED: 'false'

test:
  WORKER_HOST: http://127.0.0.1:3002
  WORKER_KEY: WORKER_KEY
  API_HOST: http://127.0.0.1:3000
  HARVESTER_API_KEY: HARVESTER_KEY
  MFA_ENABLED: 'false'
  OTP_SECRET_KEY: c8e869e0ce9c139a53c0d0fcc43c5883ea784e874330c458c160b7f4a3d227c802439188f37695a1f023ef42ffc3568cbd28b8900d6a0daccdabc20f443e51d7

staging:
  WORKER_HOST: http://127.0.0.1:3002
  WORKER_KEY: WORKER_KEY
  API_HOST: http://127.0.0.1:3000
  HARVESTER_API_KEY: HARVESTER_KEY
  OTP_SECRET_KEY: YOUROTPSECRETKEY
  MFA_ENABLED: 'false'

I start it with bundle exec rails s -e production -b 0.0.0.0 -p 3001

I did get some moderate success editing these environment blocks in the manager.yml, I noticed i could create/edit new items when I removed all but the production block, but when the staging block is removed harvest previews fail. Do i need to generate keys staging for the block, even though I am not running the stager?

The api's application.yml indent error is mine here, just from my sloppy paste into this window.

For the manager config/application.rb I just noticed I can avoid the create/edit errors and get a preview, if I exclude the staging environment from loading in the manager via:

require_relative "boot"

require "rails"
# Pick the frameworks you want:
# require "active_model/railtie"
# require "active_job/railtie"
# require "active_record/railtie"
# require "active_storage/engine"
require "action_controller/railtie"
require "action_mailer/railtie"
# require "action_mailbox/engine"
# require "action_text/engine"
# require "action_view/railtie"
require "action_cable/engine"
# require "rails/test_unit/railtie"

# Require the gems listed in Gemfile, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(*Rails.groups)

APPLICATION_ENVS = YAML.load_file('config/application.yml').keys - ['development', 'test', 'staging'] rescue []
APPLICATION_ENVIRONMENT_VARIABLES = YAML.load(ERB.new(File.read('config/application.yml')).result)

begin
  ENV.update YAML.load(ERB.new(File.read('config/application.yml')).result)[Rails.env]
rescue StandardError
  {}
end

module HarvesterManager
  class Application < Rails::Application
    # Initialize configuration defaults for originally generated Rails version.
    config.load_defaults 7.0

    # Configuration for the application, engines, and railties goes here.
    #
    # These settings can be overridden in specific environments using the files
    # in config/environments, which are processed later.
    #
    # config.time_zone = "Central Time (US & Canada)"
    # config.eager_load_paths << Rails.root.join("extras")

    # CUSTOM CONFIG GOES UNDER, this helps the migration process
    config.autoload_paths << Rails.root.join("/app/models/concerns")

    config.time_zone = ENV['TIMEZONE']
  end
end

This would prevent me from running the staging API stuff in the manager . . .(but I'm not yet sure how to do that anyways, or how critical is is at this time :)

Thanks again.

[paul-mesnilgrente]

Ok, you've done all the right steps. I might have found the issue. Can you replace line 35 in app/controllers/previews_controller.rb:

# from 
params[:environment] ||= 'staging'
# to
params[:environment] ||= ENV['PREVIEW_WORKER_HOST']

And you should have only the production block in the manager's application.yml. Note that I added the PREVIEW_WORKER_HOST line.

production:
  HOST: https://manager.auxiliator-digitalbc.ca
  WORKER_HOST: https://worker.auxiliator-digitalbc.ca
  PREVIEW_WORKER_HOST: https://worker.auxiliator-digitalbc.ca
  WORKER_KEY: 959933ea2ead4b73a6d8415b4ddad7b931324501a379a9197ca8b38b90eb83f2b1d61bcbded14ddc8820d63f49f700fd6d8294f28e99b5bbcd71b960ab8aec29
  API_HOST: https://api.auxiliator-digitalbc.ca
  MONGO_HOSTS: mongodb+srv://corymbusaux.d0rvn.mongodb.net:27017
  HARVESTER_API_KEY: LxPjQyKyBmo-emc7ATBL
  OTP_SECRET_KEY: YOUROTPSECRETKEY
  MFA_ENABLED: 'false'

It it fixes your problem I'll add the fix at the same as #260

[telezoic]

Thanks again @paul-mesnilgrente

I made the above changes and I can create/edit items, but the harvester preview fails (the icons spins without end) and I get the following error in the console:

{"name":"rails","hostname":"SolumAuxiliator","pid":11026,"level":50,"time":"2022-06-09T19:05:50.032+00:00","v":0,"msg":"[0323c2ea-d2bf-44d0-aea6-0c46198d0e91] {:msg=>\"Exception caught while checking scheduled jobs. Exception is #<NoMethodError: undefined method `[]' for nil:NilClass>\"}"}

{"name":"rails","hostname":"SolumAuxiliator","pid":11026,"level":50,"time":"2022-06-09T19:05:50.034+00:00","v":0,"msg":"[0323c2ea-d2bf-44d0-aea6-0c46198d0e91] {:msg=>\"/data/sites/supplejack_manager/app/models/concerns/environment_helpers.rb:38:in `change_worker_env!'\\n/data/sites/supplejack_manager/app/models/harvest_schedule.rb:33:in `find_from_environment'\\n/data/sites/supplejack_manager/app/models/parser.rb:148:in `scheduled?'\\n/data/sites/supplejack_manager/app/views/parsers/_parser.html.erb:159:in `_app_views_parsers__parser_html_erb__1274573687748011140_88260'\\n/home/arquebusier/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/actionview-7.0.2.4/lib/action_view/base.rb:244:in `public_send'\\n/home/arquebusier/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/actionview-7.0.2.4/lib/action_view/base.rb:244:in `_run'\\n/home/arquebusier/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/actionview-7.0.2.4/lib/action_view/template.rb:157:in `block in render'\\n/home/arquebusier/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/activesupport-7.0.2.4/lib/active_support/notifications.rb:206:in `block in instrument'\\n/home/arquebusier/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/activesupport-7.0.2.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument'\\n/home/arquebusier/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/activesupport-7.0.2.4/lib/active_support/notifications.rb:206:in `instrument'\\n/home/arquebusier/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/actionview-7.0.2.4/lib/action_view/template.rb:361:in . . .

I tried a few variants of this: with the staging env excluded in application.rb and without. With and without all the env blocks inapplication.yml but get the same error each time. I see also that

env = params[:environment] || 'staging

exists at lines 9 , 14 in app/models/concerns/environment_helpers.rb I swapped out those too but still had the error

The only approach I can get to work for create/edit and previews is the one I outlined above with all the env blocks in application.yml, excluding the staging env from the manager via

APPLICATION_ENVS = YAML.load_file('config/application.yml').keys - ['development', 'test', 'staging'] rescue []

in application.rb

and keeping params[:environment] ||= 'staging' in app/controllers/previews_controller.rb which, on the surface at least, seems ok.

Thanks again for all your help with this,
D.

[paul-mesnilgrente]

Ok, the troubles are not on you. I'll try to get a fix prioritized soon.

If you want to keep investigating in the mean time, you can try modifying line 148 of app/models/parser.rb:

!HarvestSchedule.find_from_environment({ parser_id: self.id }, 'production').empty?

with the changes I gave you in the last message.

[telezoic]

Thanks @paul-mesnilgrente! Much appreciated :)

I'm happy to investigate.

Modifying 148 in app/models/parser.rb to:

!HarvestSchedule.find_from_environment({ parser_id: self.id }, 'production').empty?

coupled with:

app/controllers/previews_controller.rb:

# from 
params[:environment] ||= 'staging'
# to
params[:environment] ||= ENV['PREVIEW_WORKER_HOST']

and in the manager's application.yml.

production:
  HOST: https://manager.auxiliator-digitalbc.ca
  WORKER_HOST: https://worker.auxiliator-digitalbc.ca
  PREVIEW_WORKER_HOST: https://worker.auxiliator-digitalbc.ca
  WORKER_KEY: 959933ea2ead4b73a6d8415b4ddad7b931324501a379a9197ca8b38b90eb83f2b1d61bcbded14ddc8820d63f49f700fd6d8294f28e99b5bbcd71b960ab8aec29
  API_HOST: https://api.auxiliator-digitalbc.ca
  MONGO_HOSTS: mongodb+srv://corymbusaux.d0rvn.mongodb.net:27017
  HARVESTER_API_KEY: LxPjQyKyBmo-emc7ATBL
  OTP_SECRET_KEY: YOUROTPSECRETKEY
  MFA_ENABLED: 'false'

give me a error (and a failed harvest preview) :

{"name":"rails","hostname":"SolumAuxiliator","pid":74451,"level":50,"time":"2022-06-13T17:40:25.874+00:00","v":0,"msg":"[1eb9b980-ee24-48ed-86ff-80787ea2a92f] {:msg=>\"Exception caught while checking scheduled jobs. Exception is #<NoMethodError: undefined method `[]' for nil:NilClass>\"}"}

Thanks again for all the assistance.

[paul-mesnilgrente]

What's the back-trace of this error?

[telezoic]

Sorry, I gave you the wrong error:

it's this one . . .

{"name":"rails","hostname":"SolumAuxiliator","pid":82865,"level":60,"time":"2022-06-13T21:36:08.000+00:00","v":0,"msg":"[8e8ec515-20d1-4d5b-9815-384572f26920] {:msg=>\" \\n[8e8ec515-20d1-4d5b-9815-384572f26920] NoMethodError (undefined method `[]' for nil:NilClass):\\n[8e8ec515-20d1-4d5b-9815-384572f26920] \\n[8e8ec515-20d1-4d5b-9815-384572f26920] app/models/concerns/environment_helpers.rb:29:in `set_worker_environment_for'\\n[8e8ec515-20d1-4d5b-9815-384572f26920] app/controllers/previews_controller.rb:37:in `create'\"}"}

I tried replacing lines 9 and 14 in app/models/concerns/environment_helpers.rb
with
params[:environment] ||= ENV['PREVIEW_WORKER_HOST']
(seeing that they too referenced env = params[:environment] || 'staging' )

but I got the same error and a spinning logo in the preview window

[paul-mesnilgrente]

👍 We have a story in the backlog to fix it. I don't know how much code needs to be changed in the manager. We have a setup working at the moment in our cluster which is used for testing.

This setup is only using staging. You could try running the worker/manager/api in staging mode without modifying the manager. I'm not sure if it's worth investigating for you... Sorry for the inconvenience

[telezoic]

No trouble @paul-mesnilgrente I can work around this. Thanks for all your assistance!