peadm:convert fails with puppetlabs/node_manager v1.0.1

[bastelfreak]

Describe the Bug

I've a PE 2019.12.8 installation. With puppetlabs/node_manager v1.0.1 the peadm:convert doesn't work on a simple/empty PE installation. Used pe.conf:

{
  "puppet_enterprise::profile::master::r10k_remote": "https://github.com/bastelfreak/testcontrolrepo.git",
  "puppet_enterprise::profile::master::code_manager_auto_configure": true,
  "console_admin_password": "***",
  "puppet_enterprise::puppet_master_host": "%{trusted.certname}"
}

This fails with:

{
  "level": "err",
  "message": "node_manager failed with error type 'missing-referents': The group being edited or created makes reference to the following missing classes:\n\"puppet_enterprise\" class in the \"production\" environment\n\nSee the details key for complete information on where every missing reference originated.",
  "source": "Puppet",
  "tags": [
    "err"
  ],
  "time": "2024-07-30T16:02:45.302802570+00:00",
  "file": null,
  "line": null
},

It works with node_manager v0.8.0. I can also reproduce the issue by running the convert successfully with v0.8.0, then delete a node group that's managed by PEADM, for example PE Primary B. Then running the convert again it fails with above error.

Expected Behavior

It's supposed to pass with node_manager v1.0.1 because that's listed in metadata.json.

Steps to Reproduce

• Install PE 2019.8.12 or 2021.7.8 or 2023.7.0
• install latest peadm & node_manager
• run peadm::convert

Environment

• AlmaLinux 8

Additional Context

• I am not sure if that's an issue within the node_manager module itself or how peadm uses it, so I raised the issue here instead of https://github.com/puppetlabs/puppetlabs-node_manager

[ragingra]

Thanks for raising the issue! I'm looking into this now.

[ragingra]

Hey @bastelfreak, I've tried a few different variations of OS and PE versions along with your pe.conf, but haven't been able replicate the issue myself. Are you using the latest PEADM release?

[bastelfreak]

I will try to explain my environment and provide a way to replicate this. I installed PE 2019.8.12 by hand. my agent is running in the environment peadm, not production. available modules:

root@pe ~ # puppet module list --environment peadm --modulepath /etc/puppetlabs/code/environments/peadm/modules
Warning: Missing dependency 'puppetlabs-apt':
  'puppetlabs-puppet_agent' (v4.20.1) requires 'puppetlabs-apt' (>= 7.7.1 < 10.0.0)
Warning: Missing dependency 'puppetlabs-bolt_shim':
  'puppetlabs-peadm' (v3.21.0) requires 'puppetlabs-bolt_shim' (>= 0.3.0 < 2.0.0)
Warning: Missing dependency 'puppetlabs-node_manager':
  'puppetlabs-peadm' (v3.21.0) requires 'puppetlabs-node_manager' (>= 1.0.1 < 2.0.0)
/etc/puppetlabs/code/environments/peadm/modules
├── WhatsARanjit-node_manager (v0.8.0)
├── ipcrm-echo (v0.1.8)
├── puppet-bolt (v1.1.1)
├── puppet-extlib (v7.2.0)
├── puppet-format (v1.1.1)
├── puppet-systemd (v7.1.0)
├── puppetlabs-apply_helpers (v0.3.0)
├── puppetlabs-exec (v3.0.0)
├── puppetlabs-facter_task (v2.0.1)
├── puppetlabs-facts (v1.4.0)
├── puppetlabs-inifile (v6.1.1)
├── puppetlabs-package (v3.0.1)
├── puppetlabs-pe_status_check (v4.3.0)
├── puppetlabs-peadm (v3.21.0)
├── puppetlabs-puppet_agent (v4.20.1)
├── puppetlabs-puppet_conf (v2.0.0)
├── puppetlabs-ruby_task_helper (v0.6.1)
├── puppetlabs-service (v3.0.0)
├── puppetlabs-stdlib (v9.6.0)
└── saz-sudo (v8.0.0)
root@pe ~ #

My Puppetfile:

mod 'WhatsARanjit/node_manager', '0.8.0'
#mod 'puppetlabs/node_manager', '1.0.1' # to cleanup existing node groups, was migrated to Perforce but not released yet

mod 'ipcrm/echo', '0.1.8' # for debug output during testing

mod 'puppetlabs/pe_status_check', # check if infra is healthy, https://github.com/puppetlabs/puppetlabs-pe_status_check/pull/226
  git: 'https://github.com/bastelfreak/puppetlabs-pe_status_check',
  branch: 'plans'

mod 'puppetlabs/stdlib', '9.6.0' # various datatypes used in the other modules

#mod 'puppetlabs-peadm', '3.21.0' # provides the peadm::convert and peadm::upgrade plans
mod 'puppetlabs-peadm',
  git: 'https://github.com/bastelfreak/puppetlabs-peadm',
  branch: 'url'

mod 'puppetlabs-apply_helpers', '0.3.0'     # peadm dependency
mod 'puppet-format', '1.1.1'                # peadm dependency
mod 'puppetlabs-service', '3.0.0'           # peadm dependency
mod 'puppetlabs-package', '3.0.1'           # peadm dependency
mod 'puppetlabs-inifile', '6.1.1'           # peadm dependency
mod 'puppetlabs-ruby_task_helper', '0.6.1'  # peadm dependency

mod 'puppetlabs-puppet_agent', '4.20.1' # configures puppet agent
mod 'puppetlabs-facts', '1.4.0'         # puppet_agent dependency
mod 'puppetlabs/puppet_conf', '2.0.0'   # retrieves/updates the environment option in puppet.conf
mod 'puppet/extlib', '7.2.0'            # retrieves/updates the environment option in puppet.conf

mod 'puppet/bolt', '1.1.1'    # installs bolt
mod 'puppet/systemd', '7.1.0' # required to write the bolt unit
mod 'saz-sudo', '8.0.0'       # required because we call peadm::* as normal user

mod 'puppetlabs-facter_task', '2.0.1' # for debugging
mod 'puppetlabs-exec', '3.0.0'        # for debugging

for the peadm module I'm referencing this branch: #465

My bolt-project.yaml:

---
analytics: false
name: peadmmig
modulepath:
- "/etc/puppetlabs/code/environments/peadm/modules"
- "/etc/puppetlabs/code/environments/peadm/site"
- "/opt/puppetlabs/puppet/modules"
stream: true
puppetdb:
  server_urls:
  - http://127.0.0.1:8080

After running the peadm::convert plan with node_manager 0.8.0 everything this fine:

root@pe ~ # puppet resource node_group 'PE Primary B'
node_group { 'PE Primary B':
  ensure               => 'present',
  data                 => {
  'puppet_enterprise::profile::primary_master_replica' => {
    'database_host_puppetdb' => 'not-configured'
  },
  'puppet_enterprise::profile::puppetdb' => {
    'database_host' => 'not-configured'
  }
},
  environment          => 'production',
  id                   => 'f0fcb7f2-a506-4a74-9ac7-8e45e8c377e9',
  override_environment => 'false',
  parent               => 'PE Infrastructure',
  provider             => 'https',
  rule                 => ['and',
  ['=',
    ['trusted', 'extensions', '1.3.6.1.4.1.34380.1.1.9812'],
    'puppet/server'],
  ['=',
    ['trusted', 'extensions', '1.3.6.1.4.1.34380.1.1.9813'],
    'B']],
}
root@pe ~ #

Now I switch to to node_manager 1.0.1:

root@pe ~ # puppet module list --environment peadm --modulepath /etc/puppetlabs/code/environments/peadm/modules
Warning: Missing dependency 'puppetlabs-apt':
  'puppetlabs-puppet_agent' (v4.20.1) requires 'puppetlabs-apt' (>= 7.7.1 < 10.0.0)
Warning: Missing dependency 'puppetlabs-bolt_shim':
  'puppetlabs-peadm' (v3.21.0) requires 'puppetlabs-bolt_shim' (>= 0.3.0 < 2.0.0)
/etc/puppetlabs/code/environments/peadm/modules
├── ipcrm-echo (v0.1.8)
├── puppet-bolt (v1.1.1)
├── puppet-extlib (v7.2.0)
├── puppet-format (v1.1.1)
├── puppet-systemd (v7.1.0)
├── puppetlabs-apply_helpers (v0.3.0)
├── puppetlabs-exec (v3.0.0)
├── puppetlabs-facter_task (v2.0.1)
├── puppetlabs-facts (v1.4.0)
├── puppetlabs-inifile (v6.1.1)
├── puppetlabs-node_manager (v1.0.1)
├── puppetlabs-package (v3.0.1)
├── puppetlabs-pe_status_check (v4.3.0)
├── puppetlabs-peadm (v3.21.0)
├── puppetlabs-puppet_agent (v4.20.1)
├── puppetlabs-puppet_conf (v2.0.0)
├── puppetlabs-ruby_task_helper (v0.6.1)
├── puppetlabs-service (v3.0.0)
├── puppetlabs-stdlib (v9.6.0)
└── saz-sudo (v8.0.0)
root@pe ~ #

Now I purge the node group:

root@pe ~ # puppet resource node_group 'PE Primary B' ensure=absent
Notice: /Node_group[PE Primary B]/ensure: removed
node_group { 'PE Primary B':
  ensure   => 'absent',
  provider => 'https',
}
root@pe ~ # puppet resource node_group 'PE Primary B'
node_group { 'PE Primary B':
  ensure   => 'absent',
  provider => 'https',
}
root@pe ~ #

now running peadm again, it also works: https://gist.github.com/bastelfreak/5f75d165e3159594813cf9103542c22a

my /etc/puppetlabs/enterprise/conf.d/pe.conf:

{
  "puppet_enterprise::profile::master::code_manager_auto_configure": true,
  "console_admin_password": "***",
  "puppet_enterprise::puppet_master_host": "%{trusted.certname}",
  "puppet_enterprise::send_analytics_data": false,
  "pe_install::install::classification::pe_node_group_environment": "peadm",
  "puppet_enterprise::master::recover_configuration::pe_environment": "peadm"
}

This "minimal" example worked. I will try around to reproduce it

[bastelfreak]

okay another try! this time it failed.

• I started a vanilla/minimal alma linux 8 VM
  run:

dnf update -y --quiet; \
dnf install -y --quiet langpacks-de tar epel-release bash-completion vim jq git; \
dnf install -y --quiet htop ccze tree screen; \
dnf update -y --quiet

then for PE, create /root/pe.conf:

{
  "puppet_enterprise::profile::master::r10k_remote": "https://github.com/bastelfreak/testcontrolrepo.git",
  "puppet_enterprise::profile::master::code_manager_auto_configure": true,
  "console_admin_password": "***",
  "puppet_enterprise::puppet_master_host": "%{trusted.certname}"
}

Install PE

curl --silent --remote-header-name --location --remote-name 'https://pm.puppet.com/cgi-bin/download.cgi?dist=el&rel=8&arch=x86_64&ver=2019.8.12'
tar xf puppet-enterprise-*-x86_64.tar.gz
rm -f puppet-enterprise-*.tar.gz
cd puppet-enterprise-*-x86_64/
DISABLE_ANALYTICS=1 ./puppet-enterprise-installer -c ~/pe.conf -y
puppet agent -t
puppet agent -t
puppet agent -t
puppet access login --verbose --username admin --lifetime=4y
puppet code deploy peadm

• create an environment group for peadm
• assign the profiles::boltproject class (basically installs bolt)
• assign the primary to that group
• run puppet agent, it will install bolt and write an inventory.yaml and bolt-project.yaml to /opt/peadmmig

inventory.yaml:

---
groups:
- name: primary
  targets:
  - name: puppet.local
    uri: local://localhost

bolt-project.yaml:

---
analytics: false
name: peadmmig
modulepath:
- "/etc/puppetlabs/code/environments/peadm/modules"
- "/etc/puppetlabs/code/environments/peadm/site"
- "/opt/puppetlabs/puppet/modules"
stream: true
puppetdb:
  server_urls:
  - http://127.0.0.1:8080

Then run the convert:

cd /opt/peadmmig
bolt plan run peadm::convert --params=@/opt/peadmmig/profiles::convert.json

This fails for me with:

Finished: apply catalog with 1 failure in 12.33 sec
Finished: plan peadm::convert in 55.22 sec
Failed on puppet.local:
  Resources failed to apply for puppet.local
    Node_group[PE Primary A]: change from 'absent' to 'present' failed: Could not set 'present' on ensure: Unable to create node_group 'PE Primary A' (file: /etc/puppetlabs/code/environments/peadm/modules/peadm/manifests/setup/node_manager.pp, line: 93)
    Node_group[PE Compiler Group A]: change from 'absent' to 'present' failed: Could not set 'present' on ensure: Unable to create node_group 'PE Compiler Group A' (file: /etc/puppetlabs/code/environments/peadm/modules/peadm/manifests/setup/node_manager.pp, line: 112)
    Node_group[PE HA Replica]: change from 'absent' to 'present' failed: Could not set 'present' on ensure: Unable to create node_group 'PE HA Replica' (file: /etc/puppetlabs/code/environments/peadm/modules/peadm/manifests/setup/node_manager.pp, line: 144)
    Node_group[PE Primary B]: change from 'absent' to 'present' failed: Could not set 'present' on ensure: Unable to create node_group 'PE Primary B' (file: /etc/puppetlabs/code/environments/peadm/modules/peadm/manifests/setup/node_manager.pp, line: 153)
    Node_group[PE Compiler Group B]: change from 'absent' to 'present' failed: Could not set 'present' on ensure: Unable to create node_group 'PE Compiler Group B' (file: /etc/puppetlabs/code/environments/peadm/modules/peadm/manifests/setup/node_manager.pp, line: 170)
  changed: 4, failed: 5, unchanged: 0 skipped: 2, noop: 0
Failed on 1 target: puppet.local
Ran on 1 target

full output: https://gist.github.com/bastelfreak/fb42f9411a2f5221823ef6a42e360638

[bastelfreak]

And when I run it a second time it passes:

https://gist.github.com/bastelfreak/ae524a0064c9ad8e84570e28855ea3bb

[bastelfreak]

@ragingra I'm wondering if this is actually related to the changed environment and not the node_manager version. My customers don't have a production environment and they only deploy one environment, named after their team or product. In my above tests it's peadm. When I do the first deployment with puppet code deploy peadm, code-manager will purge /etc/puppetlabs/code/environments/production. On the first puppet run afterwards, puppet will recreate /etc/puppetlabs/code/environments/production. Maybe this is related.

[bastelfreak]

I want to point out that I'm able to reproduce this, as described in #469 (comment), but with the v3.21.0 release of PEADM. This issue isn't related to my feature branch from #465

[ragingra]

Hi @bastelfreak, thank you for the extra detail. I was able to replicate the same error using your environment, and found that adding the wait flag to the code deploy seemed to fix the issue for me. puppet code deploy peadm --wait In the usual scenario, convert was working fine for me and I don't believe there is an issue with node_manager or PEADM.

[bastelfreak]

I tried it again today with a fresh VM and I'm able to reproduce the problem, even with puppet code deploy peadm --wait.

Modulelist:

root@pe /opt/peadmmig # puppet module list --environment peadm
Warning: Missing dependency 'puppetlabs-apt':
  'puppetlabs-puppet_agent' (v4.20.1) requires 'puppetlabs-apt' (>= 7.7.1 < 10.0.0)
Warning: Missing dependency 'puppetlabs-bolt_shim':
  'puppetlabs-peadm' (v3.21.0) requires 'puppetlabs-bolt_shim' (>= 0.3.0 < 2.0.0)
/etc/puppetlabs/code/environments/peadm/site
└── puppet-profiles (v1.0.0)
/etc/puppetlabs/code/environments/peadm/modules
├── ipcrm-echo (v0.1.8)
├── puppet-bolt (v1.1.1)
├── puppet-extlib (v7.2.0)
├── puppet-format (v1.1.1)
├── puppet-systemd (v7.1.0)
├── puppetlabs-apply_helpers (v0.3.0)
├── puppetlabs-exec (v3.0.0)
├── puppetlabs-facter_task (v2.0.1)
├── puppetlabs-facts (v1.4.0)
├── puppetlabs-inifile (v6.1.1)
├── puppetlabs-node_manager (v1.0.1)
├── puppetlabs-package (v3.0.1)
├── puppetlabs-pe_status_check (v4.3.0)
├── puppetlabs-peadm (v3.21.0)
├── puppetlabs-puppet_agent (v4.20.1)
├── puppetlabs-puppet_conf (v2.0.0)
├── puppetlabs-ruby_task_helper (v0.6.1)
├── puppetlabs-service (v3.0.0)
├── puppetlabs-stdlib (v9.6.0)
└── saz-sudo (v8.0.0)
/opt/puppetlabs/puppet/modules
├── puppetlabs-bash_task_helper (v2.0.0)
├── puppetlabs-cd4pe_jobs (v1.5.0)
├── puppetlabs-enterprise_tasks (v0.1.0)
├── puppetlabs-facter_task (v1.1.0)
├── puppetlabs-facts (v1.4.0)
├── puppetlabs-package (v2.2.0)
├── puppetlabs-pe_bootstrap (v0.3.0)
├── puppetlabs-pe_concat (v1.1.1)
├── puppetlabs-pe_databases (v2.2.0)
├── puppetlabs-pe_hocon (v2019.0.0)
├── puppetlabs-pe_infrastructure (v2018.1.0)
├── puppetlabs-pe_inifile (v1.1.3)
├── puppetlabs-pe_install (v2019.8.10)
├── puppetlabs-pe_java_ks (v2016.4.0)
├── puppetlabs-pe_nginx (v2017.1.0)
├── puppetlabs-pe_patch (v0.13.0)
├── puppetlabs-pe_postgresql (v2016.5.0)
├── puppetlabs-pe_puppet_authorization (v2016.2.0)
├── puppetlabs-pe_r10k (v2016.2.0)
├── puppetlabs-pe_repo (v2018.1.0)
├── puppetlabs-pe_staging (v0.3.3)
├── puppetlabs-pe_support_script (v3.2.0)
├── puppetlabs-powershell_task_helper (v0.1.0)
├── puppetlabs-puppet_conf (v1.3.0)
├── puppetlabs-puppet_enterprise (v2018.1.0)
├── puppetlabs-puppet_metrics_collector (v7.1.0)
├── puppetlabs-python_task_helper (v0.5.0)
├── puppetlabs-reboot (v4.2.0)
├── puppetlabs-ruby_task_helper (v0.6.1)
└── puppetlabs-service (v2.2.0)
root@pe /opt/peadmmig #

full bolt output: https://gist.github.com/bastelfreak/1ae222ea9ab4cb388e725c37e81cb954.

I'm wondering if this is somehow related to the empty production environment. the puppet_enterprise module is always available at /opt/puppetlabs/puppet/modules. My puppet.conf is default:

root@pe ~ # cat /etc/puppetlabs/puppet/puppet.conf
# This file can be used to override the default puppet settings.
# See the following links for more details on what settings are available:
# - https://puppet.com/docs/puppet/latest/config_important_settings.html
# - https://puppet.com/docs/puppet/latest/config_about_settings.html
# - https://puppet.com/docs/puppet/latest/config_file_main.html
# - https://puppet.com/docs/puppet/latest/configuration.html

[main]
certname = ***
server = ***
user = pe-puppet
group = pe-puppet
environment_timeout = unlimited
module_groups = base+pe_only

[agent]
graph = true
server_list = ***:8140

[master]
node_terminus = classifier
storeconfigs = true
storeconfigs_backend = puppetdb
reports = puppetdb
certname = ***
always_retry_plugins = false
disable_i18n = false
versioned_environment_dirs = true
root@pe ~ #

and modulepath is okay as well:

root@pe ~ # puppet config print modulepath
ca/etc/puppetlabs/code/environments/production/modules:/etc/puppetlabs/code/modules:/opt/puppetlabs/puppet/modules
root@pe ~ #

and production is empty because I don't deploy it and then puppet agent creates it:

root@pe ~ # ls -la /etc/puppetlabs/code/environments/production/
total 8.0K
drwxr-x---. 2 pe-puppet pe-puppet 4.0K  8. Aug 07:46 .
drwxr-xr-x. 4 pe-puppet pe-puppet 4.0K  8. Aug 07:46 ..
root@pe ~ #

I can even start bolt multiple times now and always get the same error: https://gist.github.com/bastelfreak/70cf3693342f8ce46e307f5fa13a5e6c

I now added the environment to puppt.conf:

# This file can be used to override the default puppet settings.
# See the following links for more details on what settings are available:
# - https://puppet.com/docs/puppet/latest/config_important_settings.html
# - https://puppet.com/docs/puppet/latest/config_about_settings.html
# - https://puppet.com/docs/puppet/latest/config_file_main.html
# - https://puppet.com/docs/puppet/latest/configuration.html

[main]
certname = pe.tim.betadots.training
server = pe.tim.betadots.training
user = pe-puppet
group = pe-puppet
environment_timeout = unlimited
module_groups = base+pe_only

[agent]
graph = true
server_list = pe.tim.betadots.training:8140
environment = peadm

[master]
node_terminus = classifier
storeconfigs = true
storeconfigs_backend = puppetdb
reports = puppetdb
certname = pe.tim.betadots.training
always_retry_plugins = false
disable_i18n = false
versioned_environment_dirs = true

And now bolt passes: https://gist.github.com/bastelfreak/fdb7b0cbacaa1d2aad61a28f82b24b3b (but I don't know yet if that's a timing accident or an actual fix)

[bastelfreak]

I tried this on a new VM and added the environment to puppet.conf, but that also produces the same error. full output: https://gist.github.com/bastelfreak/25fe80e420537d8b60420914ef6d2161

Afterwards I run bolt plan run peadm::convert --params=@/opt/peadmmig/profiles::convert.json 9 times in a row, they all failed with the same error

[bastelfreak]

I was away for an hour, run bolt again, and this time it just worked. I guess this is a weird timing issue, but I'm not sure yet what would cause it.

[GSPatton]

Hi @bastelfreak. I've created a ticket for this and have prioritised it for the team to work on as soon as possible. Thank you once again for your contributions!

[bastelfreak]

ohhhh I maybe found the problem:

Flow

• the control-repo has no production branch or the production branch isn't deployed
• A deployment of any other environment triggers r10k/code-manager to cleanup unmanaged environments, e.g. production
• the classifier API periodically requests a list of environments/classes from the puppetserver
• I usually have a code-manager hook to reread the classes into classifier (and that probab):

puppet_enterprise::master::code_manager::post_environment_hooks:
  - url: "https://%{trusted.certname}:4433/classifier-api/v1/update-classes"
    use-client-ssl: true

• you cannot assign class data nor classes to a node group when they don't exist in the environment mentioned in the node group
• PEADM doesn't set the environment in https://github.com/puppetlabs/puppetlabs-peadm/blob/main/manifests/setup/node_manager.pp
• the default is production: https://github.com/puppetlabs/puppetlabs-node_manager/blob/main/lib/puppet/type/node_group.rb#L124
• puppet agent has some internal logic to recreate the production environment if it's missing

root@pe ~ # rm -rf /etc/puppetlabs/code/environments/production/
root@pe ~ # puppet agent -t
Notice: /File[/etc/puppetlabs/code/environments/production]/ensure: created
Info: Using environment 'peadm'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for pe.tim.betadots.training
Info: Applying configuration version '1724075688'
Notice: Applied catalog in 20.77 seconds
root@pe ~ #

• puppet agent of course doesn't inform the classifier to reread the classes
• there's a timewindow after a code deployment up to the next agent run + periodic classifier class import, where production environment is missing
• when you run peadm::convert in this time, it will fail with the error I posted earlier

[puppet.local] out:       "message": "node_manager failed with error type 'missing-referents': The group being edited or created makes reference to the following missing classes:\n\"puppet_enterprise\" class in the \"production\" environment\n\nSee the details key for complete information on where every missing reference originated.",

Root cause

PEADM always assumes that an production environment is present and I think that's a bug.

Possible workaround 1

Running PE nodes in a different environment is documented at https://www.puppet.com/docs/pe/2023.7/upgrade_pe#update_environment . Maybe PEADM should check if those values are set and then use that environment.

Possible workaround 2

An alternative could be that people that use a different environment have to update the environment for the PE Infrastructure node group + all childs. And PEADM has to check for the environment of the parent and inherit it for new node groups.

Possible solution

We talked a lot in the past about how Puppet Enterprise handles the modules in /opt/puppetlabs/puppet/modules. I think PE should have it's own environment. And PE specific node groups should use that environment, and not production. Another benefit is that vendored modules don't need to be renamed to pe_* anymore.