[JENKINS-67663] Pipeline checkout fails jenkins-gitclient-pass.sh: line 3: cat: command not found

[jenkins-infra-bot]

---

Originally reported by ffsetit, imported from: Pipeline checkout fails jenkins-gitclient-pass.sh: line 3: cat: command not found

• status: Open
• priority: Critical
• component(s): nodelabelparameter-plugin
• resolution: Unresolved
• votes: 2
• watchers: 5
• imported: 20251210-235911

environment

Jenkins: 2.331<br/>
OS: Linux - 3.10.0-1160.53.1.el7.x86_64<br/>
---<br/>
Office-365-Connector:4.15.2<br/>
ace-editor:1.1<br/>
analysis-core:1.96<br/>
ansicolor:1.0.1<br/>
ant:1.13<br/>
antisamy-markup-formatter:2.7<br/>
apache-httpcomponents-client-4-api:4.5.13-1.0<br/>
artifactory:3.15.2<br/>
authentication-tokens:1.4<br/>
authorize-project:1.4.0<br/>
badge:1.9.1<br/>
bitbucket:214.v2fd4234d0554<br/>
block-queued-job:0.2.0<br/>
blueocean:1.25.2<br/>
blueocean-autofavorite:1.2.4<br/>
blueocean-bitbucket-pipeline:1.25.2<br/>
blueocean-commons:1.25.2<br/>
blueocean-config:1.25.2<br/>
blueocean-core-js:1.25.2<br/>
blueocean-dashboard:1.25.2<br/>
blueocean-display-url:2.4.1<br/>
blueocean-events:1.25.2<br/>
blueocean-executor-info:1.25.2<br/>
blueocean-git-pipeline:1.25.2<br/>
blueocean-github-pipeline:1.25.2<br/>
blueocean-i18n:1.25.2<br/>
blueocean-jira:1.25.2<br/>
blueocean-jwt:1.25.2<br/>
blueocean-personalization:1.25.2<br/>
blueocean-pipeline-api-impl:1.25.2<br/>
blueocean-pipeline-editor:1.25.2<br/>
blueocean-pipeline-scm-api:1.25.2<br/>
blueocean-rest:1.25.2<br/>
blueocean-rest-impl:1.25.2<br/>
blueocean-web:1.25.2<br/>
bootstrap4-api:4.6.0-3<br/>
bootstrap5-api:5.1.3-4<br/>
bouncycastle-api:2.25<br/>
branch-api:2.7.0<br/>
build-blocker-plugin:1.7.7<br/>
build-failure-analyzer:2.1.0<br/>
build-monitor-plugin:1.13+build.202201221819<br/>
build-timeout:1.20<br/>
build-token-root:1.9<br/>
build-user-vars-plugin:1.8<br/>
caffeine-api:2.9.2-29.v717aac953ff3<br/>
checkmarx:2022.1.2<br/>
checks-api:1.7.2<br/>
claim:2.18.2<br/>
cloudbees-bitbucket-branch-source:751.vda_24678a_f781<br/>
cloudbees-folder:6.17<br/>
command-launcher:1.6<br/>
conditional-buildstep:1.4.1<br/>
config-file-provider:3.8.2<br/>
configurationslicing:1.52<br/>
copyartifact:1.46.2<br/>
credentials:1074.v60e6c29b_b_44b_<br/>
credentials-binding:1.27.1<br/>
custom-tools-plugin:0.8<br/>
cvs:2.19<br/>
dashboard-view:2.18<br/>
dependency-check-jenkins-plugin:5.1.2<br/>
dependency-track:4.0.0<br/>
discard-old-build:1.05<br/>
display-url-api:2.3.5<br/>
docker-commons:1.18<br/>
docker-java-api:3.1.5.2<br/>
docker-swarm:1.11<br/>
docker-workflow:1.27<br/>
durable-task:493.v195aefbb0ff2<br/>
echarts-api:5.2.2-2<br/>
email-ext:2.87<br/>
embeddable-build-status:2.0.3<br/>
envinject:2.4.0<br/>
envinject-api:1.8<br/>
extended-choice-parameter:0.82<br/>
external-monitor-job:1.7<br/>
favorite:2.3.3<br/>
file-operations:1.11<br/>
font-awesome-api:5.15.4-5<br/>
git:4.10.3<br/>
git-changelog:3.18<br/>
git-client:3.11.1-SNAPSHOT (private-b07504bc-)<br/>
git-server:1.10<br/>
github:1.34.1<br/>
github-api:1.301-378.v9807bd746da5<br/>
github-branch-source:2.11.4<br/>
google-metadata-plugin:0.3.1<br/>
google-oauth-plugin:1.0.6<br/>
google-storage-plugin:1.5.4.1<br/>
gradle:1.38<br/>
greenballs:1.15.1<br/>
groovy:2.4<br/>
groovy-postbuild:2.5<br/>
h2-api:1.4.199<br/>
handlebars:3.0.8<br/>
handy-uri-templates-2-api:2.1.8-1.0<br/>
htmlpublisher:1.28<br/>
http_request:1.13<br/>
icon-shim:3.0.0<br/>
ivy:2.1<br/>
jackson2-api:2.13.1-246.va8a9f3eaf46a<br/>
javadoc:1.6<br/>
javax-activation-api:1.2.0-2<br/>
javax-mail-api:1.6.2-5<br/>
jaxb:2.3.0.1<br/>
jdk-tool:1.5<br/>
jenkins-design-language:1.25.2<br/>
jira:3.6<br/>
jjwt-api:0.11.2-9.c8b45b8bb173<br/>
job-dsl:1.78.3<br/>
jobConfigHistory:2.31-rc1098.b666422863b2<br/>
jquery:1.12.4-1<br/>
jquery-detached:1.2.1<br/>
jquery-ui:1.0.2<br/>
jquery3-api:3.6.0-2<br/>
jsch:0.1.55.2<br/>
junit:1.53<br/>
label-linked-jobs:6.0.1<br/>
labelled-steps:1.0<br/>
ldap:2.7<br/>
leastload:3.0.0<br/>
lockable-resources:2.13<br/>
locks-and-latches:0.6<br/>
mailer:408.vd726a_1130320<br/>
mapdb-api:1.0.9.0<br/>
matrix-auth:3.0<br/>
matrix-project:1.20<br/>
maven-plugin:3.16<br/>
mercurial:2.16<br/>
metrics:4.0.2.8.1<br/>
momentjs:1.1.1<br/>
monitoring:1.90.0<br/>
msbuild:1.30<br/>
nested-view:1.24<br/>
newrelic-deployment-notifier:1.8<br/>
nodejs:1.5.1<br/>
nodelabelparameter:1.10.3<br/>
oauth-credentials:0.5<br/>
okhttp-api:4.9.3-105.vb96869f8ac3a<br/>
pam-auth:1.6.1<br/>
parameterized-trigger:2.43<br/>
pipeline-build-step:2.15<br/>
pipeline-graph-analysis:188.v3a01e7973f2c<br/>
pipeline-input-step:446.vf27b_0b_83500e<br/>
pipeline-maven:3.10.0<br/>
pipeline-milestone-step:1.3.2<br/>
pipeline-model-api:1.9.3<br/>
pipeline-model-declarative-agent:1.1.1<br/>
pipeline-model-definition:1.9.3<br/>
pipeline-model-extensions:1.9.3<br/>
pipeline-rest-api:2.20<br/>
pipeline-stage-step:291.vf0a8a7aeeb50<br/>
pipeline-stage-tags-metadata:1.9.3<br/>
pipeline-stage-view:2.20<br/>
pipeline-utility-steps:2.12.0<br/>
plain-credentials:1.7<br/>
plugin-util-api:2.12.0<br/>
poll-mailbox-trigger-plugin:1.026<br/>
popper-api:1.16.1-2<br/>
popper2-api:2.11.2-1<br/>
powershell:1.7<br/>
preSCMbuildstep:0.3<br/>
promoted-builds:3.11<br/>
publish-over:0.22<br/>
publish-over-ftp:1.16<br/>
pubsub-light:1.16<br/>
pyenv-pipeline:2.1.2<br/>
resource-disposer:0.17<br/>
role-strategy:3.2.0<br/>
run-condition:1.5<br/>
saml:2.1.1-275.va_5718591a_999<br/>
scm-api:595.vd5a_df5eb_0e39<br/>
scmskip:1.0.3<br/>
script-security:1131.v8b_b_5eda_c328e<br/>
simple-theme-plugin:0.7<br/>
snakeyaml-api:1.29.1<br/>
sonar:2.14<br/>
sse-gateway:1.24<br/>
ssh-credentials:1.19<br/>
ssh-slaves:1.33.0<br/>
sshd:3.1.0<br/>
stashNotifier:1.25<br/>
strict-crumb-issuer:2.1.0<br/>
structs:308.v852b473a2b8c<br/>
subversion:2.15.2<br/>
test-results-aggregator:1.2.6<br/>
thinBackup:1.10<br/>
timestamper:1.16<br/>
token-macro:267.vcdaea6462991<br/>
trilead-api:1.0.13<br/>
variant:1.4<br/>
view-job-filters:2.3<br/>
windows-slaves:1.8<br/>
workflow-aggregator:2.6<br/>
workflow-api:1136.v7f5f1759dc16<br/>
workflow-basic-steps:2.24<br/>
workflow-cps:2648.va9433432b33c<br/>
workflow-cps-global-lib:552.vd9cc05b8a2e1<br/>
workflow-durable-task-step:1121.va_65b_d2701486<br/>
workflow-job:1156.v7539182e7b_d5<br/>
workflow-multibranch:706.vd43c65dec013<br/>
workflow-scm-step:2.13<br/>
workflow-step-api:622.vb_8e7c15b_c95a_<br/>
workflow-support:813.vb_d7c3d2984a_0

1 attachment

• job-config.png

  

[jenkins-infra-bot]

markewaite:

• Original comment link
• Raw content of original comment:

  I'm not understanding how the failure occurs.
  
  As far as I can tell, the operation in the log that you provided is running the command /usr/bin/git on an agent that looks like it is a Unix style path (directory name starts with /apps/jenkins).  
  
  The git process that is attempting to perform the checkout is calling a shell script because it is configured with the ASKPASS authentication method that uses a script.  
  
  Inside that script, the cat command is not available on the path.  
  
  When you modified the script in your custom build of the git client plugin to use a hard coded path to the cat command, the script succeeds.
  
  That seems to indicate that there is a problem in the path setting on the location where the script is being run.  I'd rather not change that script to hard code the path to the cat command because missing entries in the path are likely to cause all sorts of other problems and the location of cat seems to vary from system to system (some in /bin/cat others in /usr/bin/cat)
  
  Do Pipelines fail consistently when attempting to use the agent with the label MASTER_NODE or is the failure limited to a specific Pipeline?
  
  I have many Pipelines that run in various configurations of Linux controller with agents on many different platforms, including Windows.  I've not seen this failure in those configurations as far as I can tell.  Any guesses what might be different between my environment and yours?
  

I'm not understanding how the failure occurs.

As far as I can tell, the operation in the log that you provided is running the command /usr/bin/git on an agent that looks like it is a Unix style path (directory name starts with /apps/jenkins).

The git process that is attempting to perform the checkout is calling a shell script because it is configured with the ASKPASS authentication method that uses a script.

Inside that script, the cat command is not available on the path.

When you modified the script in your custom build of the git client plugin to use a hard coded path to the cat command, the script succeeds.

That seems to indicate that there is a problem in the path setting on the location where the script is being run. I'd rather not change that script to hard code the path to the cat command because missing entries in the path are likely to cause all sorts of other problems and the location of cat seems to vary from system to system (some in /bin/cat others in /usr/bin/cat)

Do Pipelines fail consistently when attempting to use the agent with the label MASTER_NODE or is the failure limited to a specific Pipeline?

I have many Pipelines that run in various configurations of Linux controller with agents on many different platforms, including Windows. I've not seen this failure in those configurations as far as I can tell. Any guesses what might be different between my environment and yours?

[jenkins-infra-bot]

ffsetit:

• Original comment link
• Raw content of original comment:

  Its not clear to me either, everything was working correctly until we upgraded Jenkins and all the plugins on Sunday. We made a significant jump on multiple plugins including the git ones.
  All the testing we did in advance passed successfully, however we didn't have a test case for this particular scenario.
  
  All of our pipelines that specify the global context of a windows agent fail with this error, however the following are all ok
  
  	
  No global context + windows agents
  	
  No global context + linux agents
  	
  Global context (linux) + linux agents
  
  
  
  It seems the only thing thats not working is specifying a windows agent as the global context.
  
  I'm definitely not promoting the hack that i did, that was just to get back to a running state i would much rather understand how this is happening. From a cursory review of the code there are multiple calls to build the EnvVars that get passed to the FetchCommand, i'm wondering if one of those is getting the path from the node specified in the pipeline context which is overwriting the path for the controller itself.
  

Its not clear to me either, everything was working correctly until we upgraded Jenkins and all the plugins on Sunday. We made a significant jump on multiple plugins including the git ones.
All the testing we did in advance passed successfully, however we didn't have a test case for this particular scenario.

All of our pipelines that specify the global context of a windows agent fail with this error, however the following are all ok

• No global context + windows agents
• No global context + linux agents
• Global context (linux) + linux agents

It seems the only thing thats not working is specifying a windows agent as the global context.

I'm definitely not promoting the hack that i did, that was just to get back to a running state i would much rather understand how this is happening. From a cursory review of the code there are multiple calls to build the EnvVars that get passed to the FetchCommand, i'm wondering if one of those is getting the path from the node specified in the pipeline context which is overwriting the path for the controller itself.

[jenkins-infra-bot]

ffsetit:

• Original comment link
• Raw content of original comment:

  I managed to replicate this with a very simple pipeline
  
  
  #!groovy
  package pipelines
  
  library identifier: "jenkins-pipeline@${scm.branches[0]}", changelog: false
  
  LABEL = env.LABEL ?: ''
  NODES = LABEL ? utils.getAgentsByLabel(LABEL, false) : env.NODES?.tokenize(',')
  MASTER_NODE = NODES ? NODES[0] : 'master'
  
  assert NODES || LABEL: 'NODES or LABEL param is required'
  
  pipeline {
      agent {
          node {
              label MASTER_NODE
          }
      }
      stages {
          stage('Echo something') {
              steps {
                  script {
                      echo("Something here")
                  }
              }
          }
      }
  }
  
  
  The Library isn't strictly necessary but its part of our production, the relevant method used in it for this is
  
  List<String> getAgentsByLabel(String label, Boolean offline) {
      assert label: 'Label must be provided'
      log.info(this, "started: params: ${params}")
  def pattern = ~/[(\!\w\d\-_)]+ *( *[&amp;\|]{2} *[(\!\w\d\-_)]+)*/
  <span class="code-keyword">assert</span> pattern <span class="code-keyword">instanceof</span> java.util.regex.Pattern
  
  <span class="code-keyword">assert</span> label ==~ pattern: <span class="code-quote">"Invalid label expression specified. Usage a&amp;&amp;b a||b a&amp;&amp;(b||c)"</span>
  def nodes = nodesByLabel label: label, offline: offline
  
  <span class="code-keyword">assert</span> nodes: <span class="code-quote">"No Nodes found"</span>
  <span class="code-keyword">return</span> nodes.sort()
  
  }
  
  
  
  The issue seems to be related to the Project parameterization, when adding the param 
    
  the error occurs seemingly only with the "Label" param type
  

I managed to replicate this with a very simple pipeline

#!groovy
package pipelines

library identifier: "jenkins-pipeline@${scm.branches[0]}", changelog: false

LABEL = env.LABEL ?: ''
NODES = LABEL ? utils.getAgentsByLabel(LABEL, false) : env.NODES?.tokenize(',')
MASTER_NODE = NODES ? NODES[0] : 'master'

assert NODES || LABEL: 'NODES or LABEL param is required'

pipeline {
    agent {
node {
    label MASTER_NODE
}
    }
    stages {
stage('Echo something') {
    steps {
script {
    echo("Something here")
}
    }
}
    }
}

The Library isn't strictly necessary but its part of our production, the relevant method used in it for this is

List<String> getAgentsByLabel(String label, Boolean offline) {
    assert label: 'Label must be provided'
    log.info(this, "started: params: ${params}")

    def pattern = ~/[(\!\w\d\-_)]+ *( *[&\|]{2} *[(\!\w\d\-_)]+)*/
    assert pattern instanceof java.util.regex.Pattern

    assert label ==~ pattern: "Invalid label expression specified. Usage a&&b a||b a&&(b||c)"
    def nodes = nodesByLabel label: label, offline: offline

    assert nodes: "No Nodes found"
    return nodes.sort()
}

The issue seems to be related to the Project parameterization, when adding the param

the error occurs seemingly only with the "Label" param type

[jenkins-infra-bot]

markewaite:

• Original comment link
• Raw content of original comment:

  I should have read the description more carefully.  The problem appears to be in the Label parameter that is being provided by the node label parameter plugin.  The issue is not in the git client plugin, but is very much an issue.  The only workaround currently is to not use the label or the node parameter, replacing them with choice parameters or string parameters.
  

I should have read the description more carefully. The problem appears to be in the Label parameter that is being provided by the node label parameter plugin. The issue is not in the git client plugin, but is very much an issue. The only workaround currently is to not use the label or the node parameter, replacing them with choice parameters or string parameters.

[glennra]

This error can happen on MacOS (and I presume Linux/Unix) agents if the server is hosted on Windows and a Windows path is configured in "Manage Jenkins>System>Global Properties>Environment variables>PATH". It looks like that path is used to replace the agent's default PATH.
This is not really a bug but it would be helpful if there were better diagnostics to indicate that this has happened.