I upgraded one of my build enviromments from Jenkins 2.346.2 to Jenkins 2.375.2 today, and upgraded all plugins installed on it to the latest versions (including, but not limited to bouncycastle-api and subversion). After doing so, all jobs running on agent nodes begun failing with the following error:
Started by user
Running as
[EnvInject] - Loading node environment variables.
Building remotely on in workspace
Cleaning up
Updating svn+ssh:// at revision ''
Using sole credentials in realm ‘’
FATAL: java.lang.ExceptionInInitializerError
java.lang.NullPointerException
at java.base/javax.crypto.ProviderVerifier.verify(ProviderVerifier.java:122)
at java.base/javax.crypto.JceSecurity.verifyProvider(JceSecurity.java:191)
at java.base/javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:217)
at java.base/javax.crypto.Cipher.getInstance(Cipher.java:688)
Caused: java.lang.SecurityException: JCE cannot authenticate the provider BC
at java.base/javax.crypto.Cipher.getInstance(Cipher.java:692)
at java.base/javax.crypto.Cipher.getInstance(Cipher.java:623)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.sshd.common.util.security.SecurityEntityFactory$2.getInstance(SecurityEntityFactory.java:130)
at org.apache.sshd.common.util.security.SecurityUtils.getCipher(SecurityUtils.java:748)
at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator$LazyKeyLengthsHolder.detectSupportedKeySizes(AESPrivateKeyObfuscator.java:134)
at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator$LazyKeyLengthsHolder.(AESPrivateKeyObfuscator.java:121)
Caused: java.lang.ExceptionInInitializerError
at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.getAvailableKeyLengths(AESPrivateKeyObfuscator.java:110)
at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.getSupportedKeySizes(AESPrivateKeyObfuscator.java:51)
at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.resolveKeyLength(AESPrivateKeyObfuscator.java:87)
at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.applyPrivateKeyCipher(AESPrivateKeyObfuscator.java:58)
at org.apache.sshd.common.config.keys.loader.pem.AbstractPEMResourceKeyPairParser.applyPrivateKeyCipher(AbstractPEMResourceKeyPairParser.java:227)
at org.apache.sshd.common.config.keys.loader.pem.AbstractPEMResourceKeyPairParser.extractKeyPairs(AbstractPEMResourceKeyPairParser.java:170)
at org.apache.sshd.common.config.keys.loader.AbstractKeyPairResourceParser.loadKeyPairs(AbstractKeyPairResourceParser.java:117)
at org.apache.sshd.common.config.keys.loader.KeyPairResourceParser$2.loadKeyPairs(KeyPairResourceParser.java:166)
at org.apache.sshd.common.config.keys.loader.pem.PEMResourceParserUtils$1.loadKeyPairs(PEMResourceParserUtils.java:53)
at org.apache.sshd.common.config.keys.loader.KeyPairResourceParser$2.loadKeyPairs(KeyPairResourceParser.java:166)
at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:157)
at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:148)
at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:139)
at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:130)
at org.apache.sshd.common.util.security.SecurityUtils.loadKeyPairIdentities(SecurityUtils.java:522)
at org.tmatesoft.svn.core.internal.io.svn.SVNSSHPrivateKeyUtil.isValidPrivateKey(SVNSSHPrivateKeyUtil.java:99)
at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:102)
at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:80)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1282)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:100)
at org.tmatesoft.svn.core.io.SVNRepository.getRepositoryUUID(SVNRepository.java:268)
at org.tmatesoft.svn.core.internal.wc2.SvnRepositoryAccess.createRepository(SvnRepositoryAccess.java:103)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.createRepository(SvnNgRepositoryAccess.java:211)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:210)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1000)
at hudson.FilePath.act(FilePath.java:1186)
at hudson.FilePath.act(FilePath.java:1175)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:970)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:892)
at hudson.scm.SCM.checkout(SCM.java:540)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1241)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
at hudson.model.Run.execute(Run.java:1900)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)
Caused: java.lang.RuntimeException
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.throwThrowable(SvnNgAbstractUpdate.java:918)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:125)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:40)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:18)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:165)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
at hudson.scm.SubversionSCM$CheckOutUpdateTask.perform(SubversionSCM.java:1086)
at hudson.scm.SubversionSCM$CheckOutUpdateTask.run(SubversionSCM.java:1067)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1037)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1020)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3492)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:377)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Archiving artifacts
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Started calculate disk usage of workspace
Finished Calculation of disk usage of workspace in 0 seconds
Extended Email Publisher is currently disabled in project settings
Finished: FAILURE
This impacts agents on Windows, Linux, and macOS whether launched via JNLP or SSH.
The remoting logs don't indicate any issues:
Feb 01, 2023 3:04:31 PM org.apache.sshd.common.util.security.AbstractSecurityProviderRegistrar getOrCreateProvider
INFO: getOrCreateProvider(BC) created instance of org.bouncycastle.jce.provider.BouncyCastleProvider
Feb 01, 2023 3:04:31 PM org.apache.sshd.common.util.security.AbstractSecurityProviderRegistrar getOrCreateProvider
INFO: getOrCreateProvider(EdDSA) created instance of net.i2p.crypto.eddsa.EdDSASecurityProvider
Enabling debugging of jar sigignature validation (-Djavax.net.debug=jar) indicates no issues. Jobs run successfully on the built-in executors on the master instance. They also run successfully if the SSH key is decrypted and loaded into Jenkins with no password.
I theorize that this is somehow an artifact of the migration from the prior SSH plugin to the newer Mina-based solution. Not having any familiarity with the plumbing responsible for shipping BC to the nodes and loading it, nor the wiring of BC and Mina, this is only a theory.
Originally reported by mattkosem, imported from: Cannot execute builds using SVN SCM with SVN+SSH using private key with passphrase
- assignee:
alobato
- status: Open
- priority: Major
- component(s): ssh-credentials-plugin, ssh-plugin
- resolution: Unresolved
- votes: 0
- watchers: 1
- imported: 20251215-193512
Raw content of original issue
I upgraded one of my build enviromments from Jenkins 2.346.2 to Jenkins 2.375.2 today, and upgraded all plugins installed on it to the latest versions (including, but not limited to bouncycastle-api and subversion). After doing so, all jobs running on agent nodes begun failing with the following error:
Started by user <REDACTED>
Running as <REDACTED>
[EnvInject] - Loading node environment variables.
Building remotely on <REDACTED> in workspace <REDACTED>
Cleaning up <REDACTED>
Updating svn+ssh://<REDACTED> at revision '<REDACTED>'
Using sole credentials <REDACTED> in realm ‘<REDACTED>’
FATAL: java.lang.ExceptionInInitializerError
java.lang.NullPointerException
at java.base/javax.crypto.ProviderVerifier.verify(ProviderVerifier.java:122)
at java.base/javax.crypto.JceSecurity.verifyProvider(JceSecurity.java:191)
at java.base/javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:217)
at java.base/javax.crypto.Cipher.getInstance(Cipher.java:688)
Caused: java.lang.SecurityException: JCE cannot authenticate the provider BC
at java.base/javax.crypto.Cipher.getInstance(Cipher.java:692)
at java.base/javax.crypto.Cipher.getInstance(Cipher.java:623)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.sshd.common.util.security.SecurityEntityFactory$2.getInstance(SecurityEntityFactory.java:130)
at org.apache.sshd.common.util.security.SecurityUtils.getCipher(SecurityUtils.java:748)
at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator$LazyKeyLengthsHolder.detectSupportedKeySizes(AESPrivateKeyObfuscator.java:134)
at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator$LazyKeyLengthsHolder.<clinit>(AESPrivateKeyObfuscator.java:121)
Caused: java.lang.ExceptionInInitializerError
at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.getAvailableKeyLengths(AESPrivateKeyObfuscator.java:110)
at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.getSupportedKeySizes(AESPrivateKeyObfuscator.java:51)
at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.resolveKeyLength(AESPrivateKeyObfuscator.java:87)
at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.applyPrivateKeyCipher(AESPrivateKeyObfuscator.java:58)
at org.apache.sshd.common.config.keys.loader.pem.AbstractPEMResourceKeyPairParser.applyPrivateKeyCipher(AbstractPEMResourceKeyPairParser.java:227)
at org.apache.sshd.common.config.keys.loader.pem.AbstractPEMResourceKeyPairParser.extractKeyPairs(AbstractPEMResourceKeyPairParser.java:170)
at org.apache.sshd.common.config.keys.loader.AbstractKeyPairResourceParser.loadKeyPairs(AbstractKeyPairResourceParser.java:117)
at org.apache.sshd.common.config.keys.loader.KeyPairResourceParser$2.loadKeyPairs(KeyPairResourceParser.java:166)
at org.apache.sshd.common.config.keys.loader.pem.PEMResourceParserUtils$1.loadKeyPairs(PEMResourceParserUtils.java:53)
at org.apache.sshd.common.config.keys.loader.KeyPairResourceParser$2.loadKeyPairs(KeyPairResourceParser.java:166)
at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:157)
at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:148)
at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:139)
at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:130)
at org.apache.sshd.common.util.security.SecurityUtils.loadKeyPairIdentities(SecurityUtils.java:522)
at org.tmatesoft.svn.core.internal.io.svn.SVNSSHPrivateKeyUtil.isValidPrivateKey(SVNSSHPrivateKeyUtil.java:99)
at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:102)
at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:80)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1282)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:100)
at org.tmatesoft.svn.core.io.SVNRepository.getRepositoryUUID(SVNRepository.java:268)
at org.tmatesoft.svn.core.internal.wc2.SvnRepositoryAccess.createRepository(SvnRepositoryAccess.java:103)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.createRepository(SvnNgRepositoryAccess.java:211)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:210)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to <REDACTED>
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1000)
at hudson.FilePath.act(FilePath.java:1186)
at hudson.FilePath.act(FilePath.java:1175)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:970)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:892)
at hudson.scm.SCM.checkout(SCM.java:540)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1241)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
at hudson.model.Run.execute(Run.java:1900)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)
Caused: java.lang.RuntimeException
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.throwThrowable(SvnNgAbstractUpdate.java:918)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:125)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:40)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:18)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:165)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
at hudson.scm.SubversionSCM$CheckOutUpdateTask.perform(SubversionSCM.java:1086)
at hudson.scm.SubversionSCM$CheckOutUpdateTask.run(SubversionSCM.java:1067)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1037)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1020)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3492)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:377)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Archiving artifacts
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Started calculate disk usage of workspace
Finished Calculation of disk usage of workspace in 0 seconds
Extended Email Publisher is currently disabled in project settings
Finished: FAILURE
This impacts agents on Windows, Linux, and macOS whether launched via JNLP or SSH.
The remoting logs don't indicate any issues:
Feb 01, 2023 3:04:31 PM org.apache.sshd.common.util.security.AbstractSecurityProviderRegistrar getOrCreateProvider
INFO: getOrCreateProvider(BC) created instance of org.bouncycastle.jce.provider.BouncyCastleProvider
Feb 01, 2023 3:04:31 PM org.apache.sshd.common.util.security.AbstractSecurityProviderRegistrar getOrCreateProvider
INFO: getOrCreateProvider(EdDSA) created instance of net.i2p.crypto.eddsa.EdDSASecurityProvider
Enabling debugging of jar sigignature validation (-Djavax.net.debug=jar) indicates no issues. Jobs run successfully on the built-in executors on the master instance. They also run successfully if the SSH key is decrypted and loaded into Jenkins with no password.
I theorize that this is somehow an artifact of the migration from the prior SSH plugin to the newer Mina-based solution. Not having any familiarity with the plumbing responsible for shipping BC to the nodes and loading it, nor the wiring of BC and Mina, this is only a theory.
environment
Oracle Linux 7.9<br/>
JDK 11.0.18<br/>
Jenkins 2.375.2<br/>
Credentials plugin 1214.v1de940103927<br/>
SSH Credentials plugin 305.v8f4381501156<br/>
SSH plugin 2.6.1<br/>
bouncycastle-api plugin 2.27
I upgraded one of my build enviromments from Jenkins 2.346.2 to Jenkins 2.375.2 today, and upgraded all plugins installed on it to the latest versions (including, but not limited to bouncycastle-api and subversion). After doing so, all jobs running on agent nodes begun failing with the following error:
This impacts agents on Windows, Linux, and macOS whether launched via JNLP or SSH.
The remoting logs don't indicate any issues:
Enabling debugging of jar sigignature validation (-Djavax.net.debug=jar) indicates no issues. Jobs run successfully on the built-in executors on the master instance. They also run successfully if the SSH key is decrypted and loaded into Jenkins with no password.
I theorize that this is somehow an artifact of the migration from the prior SSH plugin to the newer Mina-based solution. Not having any familiarity with the plumbing responsible for shipping BC to the nodes and loading it, nor the wiring of BC and Mina, this is only a theory.
Originally reported by mattkosem, imported from: Cannot execute builds using SVN SCM with SVN+SSH using private key with passphrase
Raw content of original issue
environment