Connection refused by noah

[jwokaty]

kjohnson didn't finish sending build products because the connection was refused by noah. Noah did not reboot. Is there a time limit for connections?

BBS> [STAGE5] DONE at Thu Oct  6 15:09:02 2022.
BBS> START writing BBS_EndOfRun.txt ticket.
BBS>   cd BBS_MEAT_PATH
BBS>   Sending BBS_EndOfRun.txt to BBS_CENTRAL_BASEURL/products-in/kjohnson/:
BBS>   runJob(): /usr/bin/rsync --rsh 'ssh -F /Users/biocbuild/.ssh/config' -rl --delete --delete-excluded --exclude='.svn' --exclude='.git' --exclude='.github' --exclude='.git_*' BBS_EndOfRun.txt biocbuild@nebbiolo2:/home/biocbuild/public_html/BBS/3.16/bioc-mac-arm64/products-in/kjohnson
BBS>     pid = 13576 / time [ssh: connect to host noah.dfci.harvard.edu port 22: Connection refused^M
kex_exchange_identification: Connection closed by remote host^M
Connection closed by UNKNOWN port 65535^M
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at /AppleInternal/Library/BuildRoots/a0876c02-1788-11ed-b9c4-96898e02b808/Library/Caches/com.apple.xbs/Sources/rsync/rsync/io.c(453) [sender=2.6.9]
] = 0.12 seconds / retcode = 255 / ERROR!
BBS>   runJob(): /usr/bin/rsync --rsh 'ssh -F /Users/biocbuild/.ssh/config' -rl --delete --delete-excluded --exclude='.svn' --exclude='.git' --exclude='.github' --exclude='.git_*' BBS_EndOfRun.txt biocbuild@nebbiolo2:/home/biocbuild/public_html/BBS/3.16/bioc-mac-arm64/products-in/kjohnson
BBS>     pid = 13725 / time [ssh: connect to host noah.dfci.harvard.edu port 22: Connection refused^M
kex_exchange_identification: Connection closed by remote host^M
Connection closed by UNKNOWN port 65535^M
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at /AppleInternal/Library/BuildRoots/a0876c02-1788-11ed-b9c4-96898e02b808/Library/Caches/com.apple.xbs/Sources/rsync/rsync/io.c(453) [sender=2.6.9]
] = 0.23 seconds / retcode = 255 / ERROR!
BBS>   runJob(): /usr/bin/rsync --rsh 'ssh -F /Users/biocbuild/.ssh/config' -rl --delete --delete-excluded --exclude='.svn' --exclude='.git' --exclude='.github' --exclude='.git_*' BBS_EndOfRun.txt biocbuild@nebbiolo2:/home/biocbuild/public_html/BBS/3.16/bioc-mac-arm64/products-in/kjohnson
BBS>     pid = 13856 / time [ssh: connect to host noah.dfci.harvard.edu port 22: Connection refused^M
kex_exchange_identification: Connection closed by remote host^M
Connection closed by UNKNOWN port 65535^M
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at /AppleInternal/Library/BuildRoots/a0876c02-1788-11ed-b9c4-96898e02b808/Library/Caches/com.apple.xbs/Sources/rsync/rsync/io.c(453) [sender=2.6.9]
] = 0.21 seconds / retcode = 255 / ERROR!
BBS>   runJob(): /usr/bin/rsync --rsh 'ssh -F /Users/biocbuild/.ssh/config' -rl --delete --delete-excluded --exclude='.svn' --exclude='.git' --exclude='.github' --exclude='.git_*' BBS_EndOfRun.txt biocbuild@nebbiolo2:/home/biocbuild/public_html/BBS/3.16/bioc-mac-arm64/products-in/kjohnson
BBS>     pid = 14023 / time [ssh: connect to host noah.dfci.harvard.edu port 22: Connection refused^M
kex_exchange_identification: Connection closed by remote host^M
Connection closed by UNKNOWN port 65535^M
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at /AppleInternal/Library/BuildRoots/a0876c02-1788-11ed-b9c4-96898e02b808/Library/Caches/com.apple.xbs/Sources/rsync/rsync/io.c(453) [sender=2.6.9]
] = 0.14 seconds / retcode = 255 / ERROR!
BBS>   runJob(): /usr/bin/rsync --rsh 'ssh -F /Users/biocbuild/.ssh/config' -rl --delete --delete-excluded --exclude='.svn' --exclude='.git' --exclude='.github' --exclude='.git_*' BBS_EndOfRun.txt biocbuild@nebbiolo2:/home/biocbuild/public_html/BBS/3.16/bioc-mac-arm64/products-in/kjohnson
BBS>     pid = 14197 / time [ssh: connect to host noah.dfci.harvard.edu port 22: Connection refused^M
kex_exchange_identification: Connection closed by remote host^M
Connection closed by UNKNOWN port 65535^M
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at /AppleInternal/Library/BuildRoots/a0876c02-1788-11ed-b9c4-96898e02b808/Library/Caches/com.apple.xbs/Sources/rsync/rsync/io.c(453) [sender=2.6.9]
] = 0.11 seconds / retcode = 255 / ERROR!
5 failed attempts => EXIT at 2022-10-06 15:11:33.

Also noticed

client_global_hostkeys_private_confirm: server gave bad signature for RSA key 0

See https://gitlab.com/gitlab-org/gitlab/-/issues/323700

[hpages]

FWIW the *-push.log files in /Users/biocbuild/bbs-3.16-bioc-mac-arm64/products-out on kjohnson have more information:

kjohnson:products-out biocbuild$ grep Connection install-push.log | wc
       0       0       0
kjohnson:products-out biocbuild$ grep Connection buildsrc-push.log | wc
      14      84     734
kjohnson:products-out biocbuild$ grep Connection buildbin-push.log | wc
     177    1233   10346

Lots of connection problems during buildbin! (STAGE5)

Note that these files will get wiped out at the beginning of the next build run.