Jenkins java.io.IOException: remote file operation failed

[orangemocha]

This is an intermittent failure that I have seen in Jenkins for a long time. One instance here: https://ci.nodejs.org/job/node-test-commit-arm/nodes=pi1-raspbian-wheezy-1_of_2/407/console

Started by upstream project "node-test-commit-arm" build number 407
originally caused by:
 Started by upstream project "node-test-commit" build number 402
 originally caused by:
  Started by upstream project "node-test-pull-request" build number 214
  originally caused by:
   Started by user Jeremiah Senkpiel
[EnvInject] - Loading node environment variables.
Building remotely on iojs-nodesource-raspbian-wheezy-pi1p-4 (pi1-raspbian-wheezy pi1-raspbian-wheezy-2_of_2 pi1-raspbian-wheezy-1_of_2) in workspace /home/iojs/build/workspace/node-test-commit-arm/nodes/pi1-raspbian-wheezy-1_of_2
java.io.IOException: remote file operation failed: /home/iojs/build/workspace/node-test-commit-arm/nodes/pi1-raspbian-wheezy-1_of_2 at hudson.remoting.Channel@c92e156:iojs-nodesource-raspbian-wheezy-pi1p-4: java.io.IOException: Remote call on iojs-nodesource-raspbian-wheezy-pi1p-4 failed
    at hudson.FilePath.act(FilePath.java:987)
    at hudson.FilePath.act(FilePath.java:969)
    at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:131)
    at hudson.plugins.git.GitSCM.createClient(GitSCM.java:705)
    at hudson.plugins.git.GitSCM.createClient(GitSCM.java:697)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1037)
    at hudson.scm.SCM.checkout(SCM.java:485)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1277)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:610)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532)
    at hudson.model.Run.execute(Run.java:1741)
    at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:408)
Caused by: java.io.IOException: Remote call on iojs-nodesource-raspbian-wheezy-pi1p-4 failed
    at hudson.remoting.Channel.call(Channel.java:786)
    at hudson.FilePath.act(FilePath.java:980)
    ... 14 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy9
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at java.lang.reflect.Proxy.newInstance(Proxy.java:764)
    at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:755)
    at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:111)
    at hudson.remoting.Channel.export(Channel.java:593)
    at hudson.remoting.Channel.export(Channel.java:562)
    at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:201)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1075)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
    at hudson.remoting.UserRequest._serialize(UserRequest.java:158)
    at hudson.remoting.UserRequest.serialize(UserRequest.java:167)
    at hudson.remoting.UserRequest.perform(UserRequest.java:129)
    at hudson.remoting.UserRequest.perform(UserRequest.java:49)
    at hudson.remoting.Request$2.run(Request.java:325)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at hudson.remoting.Engine$1$1.run(Engine.java:69)
    at java.lang.Thread.run(Thread.java:745)
    at ......remote call to iojs-nodesource-raspbian-wheezy-pi1p-4(Native Method)
    at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1413)
    at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
    at hudson.remoting.Channel.call(Channel.java:778)
    ... 15 more
TAP Reports Processing: START
Looking for TAP results report in workspace using pattern: test.tap
Did not find any matching files. Setting build result to FAILURE.
Build step 'Publish TAP Results' marked build as failure
Checking ^not ok
Jenkins Text Finder: File set 'test.tap' is empty
Notifying upstream projects of job completion
Finished: FAILURE

[orangemocha]

Could potentially be addressed by restarting Jenkins, like #169

[orangemocha]

@nodejs/jenkins-admins

[rvagg]

happening a lot recently, perhaps we need to update slave.jar

[jbergstroem]

Haven't seen these in a good while. Fwiw, we've updated slave.jar's throughout our fleet but is likely unrleated.

[jbergstroem]

This just occured again (spotted by @Trott): https://ci.nodejs.org/job/node-test-binary-arm/159/RUN_SUBSET=1,nodes=pi2-raspbian-wheezy/console

[Trott]

And again:

• https://ci.nodejs.org/job/node-test-binary-arm/160/RUN_SUBSET=addons,nodes=pi2-raspbian-wheezy/console
• https://ci.nodejs.org/job/node-test-binary-arm/160/RUN_SUBSET=6,nodes=pi2-raspbian-wheezy/console

Should this be re-opened?

[Trott]

Is it useful to keep reporting these here as they occur or is that just noise?

• https://ci.nodejs.org/job/node-test-binary-arm/168/RUN_SUBSET=0,nodes=pi2-raspbian-wheezy/console

[rvagg]

interesting they are just pi2-9 and pi2-2 so far

@Trott feel free to keep on reporting if they keep showing up, a pattern may be interesting

[jbergstroem]

@Trott - seen anything of this recently?

[Trott]

Haven't noticed it lately, but I haven't been looking for it either.

[jbergstroem]

Speaking of the devil: https://ci.nodejs.org/job/node-test-binary-arm/RUN_SUBSET=addons,nodes=pi2-raspbian-wheezy/902/console

Edit: again just on ARM it seems. Are these perhaps trying to all pull at the same time?

[jpsim]

Sorry to bump into this thread, this is a last resort... was this ever resolved? If so how?

I'm seeing the same exception stack trace in my Jenkins jobs, though running in OS X, so not ARM:

java.io.IOException: remote file operation failed: /Users/realm/workspace/objc_pr at hudson.remoting.Channel@4ec66dd4:sf_host1_3: java.io.IOException: Remote call on sf_host1_3 failed
    at hudson.FilePath.act(FilePath.java:1014)
    at hudson.FilePath.act(FilePath.java:996)
    at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:131)
    at hudson.plugins.git.GitSCM.createClient(GitSCM.java:741)
    at hudson.plugins.git.GitSCM.createClient(GitSCM.java:733)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1080)
    at hudson.scm.SCM.checkout(SCM.java:485)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
    at hudson.model.Run.execute(Run.java:1741)
    at hudson.matrix.MatrixBuild.run(MatrixBuild.java:301)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:410)
Caused by: java.io.IOException: Remote call on sf_host1_3 failed
    at hudson.remoting.Channel.call(Channel.java:789)
    at hudson.FilePath.act(FilePath.java:1007)
    ... 14 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy11
    at sun.reflect.GeneratedConstructorAccessor17.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739)
    at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:143)
    at hudson.remoting.Channel.export(Channel.java:621)
    at hudson.remoting.Channel.export(Channel.java:590)
    at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:201)
    at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1118)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    at hudson.remoting.UserRequest._serialize(UserRequest.java:157)
    at hudson.remoting.UserRequest.serialize(UserRequest.java:166)
    at hudson.remoting.UserRequest.perform(UserRequest.java:128)
    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    at hudson.remoting.Request$2.run(Request.java:326)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at hudson.remoting.Engine$1$1.run(Engine.java:62)
    at java.lang.Thread.run(Thread.java:745)
    at ......remote call to sf_host1_3(Native Method)
    at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
    at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
    at hudson.remoting.Channel.call(Channel.java:781)
    ... 15 more