Connection reset errors halting data processing

[ramyaragupathy]

Of late, I have been noticing app crashing on osm-changes container with the following error:

{ Error: connect ECONNRESET X.XXX.X.X:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1174:14)
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'connect',
  address: 'X.XXX.X.X',
  port: 5432 }
Error: connect ECONNRESET X.XXX.X.X:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1174:14)
{ Error: connect ECONNRESET X.XXX.X.X:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1174:14)
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'connect',
  address: 'X.XXX.X.X',
  port: 5432 }
events.js:165
      throw er; // Unhandled 'error' event
      ^

Error: connect ECONNRESET X.XXX.X.X:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1174:14)
Emitted 'error' event at:
    at osm-stats-workers/node_modules/highland/lib/index.js:2029:18
    at Stream.s._send (osm-stats-workers/node_modules/highland/lib/index.js:1560:9)
    at Stream.write (osm-stats-workers/node_modules/highland/lib/index.js:1658:18)
    at Stream._send (osm-stats-workers/node_modules/highland/lib/index.js:984:26)
    at Stream.write (osm-stats-workers/node_modules/highland/lib/index.js:1658:18)
    at osm-stats-workers/node_modules/highland/lib/index.js:686:15
    at osm-stats-workers/node_modules/highland/lib/index.js:3588:17
    at osm-stats-workers/node_modules/highland/lib/index.js:1604:9
    at Stream.s._send (osm-stats-workers/node_modules/highland/lib/index.js:1560:9)
    at Stream.write (osm-stats-workers/node_modules/highland/lib/index.js:1658:18)
    at Stream._send (osm-stats-workers/node_modules/highland/lib/index.js:984:26)
    at Stream.write (osm-stats-workers/node_modules/highland/lib/index.js:1658:18)
    at osm-stats-workers/node_modules/highland/lib/index.js:686:15
    at async.series.err (osm-stats-workers/src/stats.js:339:14)
    at osm-stats-workers/node_modules/async/dist/async.js:3888:9
    at osm-stats-workers/node_modules/async/dist/async.js:473:16
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] osm-changes: `node bin/osm-changes.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] osm-changes script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

As a temporary fix, we've tried updating the following server parameters on DB:

• max_wal_senders
• idle_in_transaction_session_timeout

This might require updates to the script where we set connection and idle timeouts for DB operations:

  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 30000,

This could also be related to node version. We're still investigating on the root cause.

cc @dakotabenjamin @bopercival-hot @danbjoseph @eternaltyro