MCO-1976: [Dev] Fix RHEL9-specific MCD Logic for RHEL10/CentOS10 Compatibility

[dkhater-redhat]

- What I did
Added RHEL10 support to the MCD for SSH key path selection and binary compatibility across RHEL versions.

Changes made:

1. SSH Key Path Selection (pkg/daemon/update.go:2291): Added IsEL10() check to useNewSSHKeyPath() so RHEL10 correctly uses the new SSH key path (/home/core/.ssh/authorized_keys.d/ignition) instead of the legacy path (/home/core/.ssh/authorized_keys).

2. Binary Selection (pkg/daemon/daemon.go:540-548): Extended ReexecuteForTargetRoot() to handle RHEL10 containers managing RHEL8 or RHEL9 hosts by selecting the appropriate binary suffix (.rhel8 or .rhel9).

This addresses items [2] and [4] from the RHEL10 MCD compatibility spike, ensuring the MCD works correctly on RHEL10 and can manage older RHEL versions during dual-stream operations.

- How to verify it
For SSH Key Path (Item 2):

On a RHEL10 cluster:

1. Deploy the updated MCO image
2. Debug into a node and check MCD logs:

   oc logs -n openshift-machine-config-operator -l k8s-app=machine-config-daemon | grep "ssh"

3. Should see: Writing SSH keys to "/home/core/.ssh/authorized_keys.d/ignition"
4. Should NOT see: Writing SSH keys to "/home/core/.ssh/authorized_keys"
5. Verify SSH keys are written to the correct location:
   oc debug node/
   chroot /host
   ls -la /home/core/.ssh/authorized_keys.d/ignition

For Binary Selection (Item 4):

When RHEL10 container images are available:

1. Deploy RHEL10 MCO container to a cluster with RHEL9 or RHEL8 nodes
2. Check MCD logs during re-execution:
   oc logs -n openshift-machine-config-operator -l k8s-app=machine-config-daemon | grep "container is rhel10"
3. Should see appropriate log:
   - "container is rhel10, target is rhel9" (for RHEL9 nodes)
   - "container is rhel10, target is rhel8" (for RHEL8 nodes)

Note: Item 4 also requires Dockerfile updates to include compiled .rhel9 binary when building RHEL10 container images. When the MCO team builds RHEL10-based container images, we will be able to see this change.
- Description for the changelog

[openshift-ci-robot]

@dkhater-redhat: This pull request references MCO-1976 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

In response to this:

- What I did
Added RHEL10 support to the MCD for SSH key path selection and binary compatibility across RHEL versions.

Changes made:

1. SSH Key Path Selection (pkg/daemon/update.go:2291): Added IsEL10() check to useNewSSHKeyPath() so
   RHEL10 correctly uses the new SSH key path (/home/core/.ssh/authorized_keys.d/ignition) instead of the legacy
   path (/home/core/.ssh/authorized_keys).

2. Binary Selection (pkg/daemon/daemon.go:540-548): Extended ReexecuteForTargetRoot() to handle RHEL10
   containers managing RHEL8 or RHEL9 hosts by selecting the appropriate binary suffix (.rhel8 or .rhel9).

This addresses items [2] and [4] from the RHEL10 MCD compatibility spike, ensuring the MCD works correctly on
RHEL10 and can manage older RHEL versions during dual-stream operations.
- How to verify it
For SSH Key Path (Item 2):

On a RHEL10 cluster:

1. Deploy the updated MCO image
2. Debug into a node and check MCD logs:

   oc logs -n openshift-machine-config-operator -l k8s-app=machine-config-daemon | grep "Writing SSH keys"

3. Should see: Writing SSH keys to "/home/core/.ssh/authorized_keys.d/ignition"
4. Should NOT see: Writing SSH keys to "/home/core/.ssh/authorized_keys"
5. Verify SSH keys are written to the correct location:
   oc debug node/
   chroot /host
   ls -la /home/core/.ssh/authorized_keys.d/ignition

For Binary Selection (Item 4):

When RHEL10 container images are available:

1. Deploy RHEL10 MCO container to a cluster with RHEL9 or RHEL8 nodes
2. Check MCD logs during re-execution:
   oc logs -n openshift-machine-config-operator -l k8s-app=machine-config-daemon | grep "container is rhel10"
3. Should see appropriate log:

• "container is rhel10, target is rhel9" (for RHEL9 nodes)
• "container is rhel10, target is rhel8" (for RHEL8 nodes)

Note: Item 4 also requires Dockerfile updates to include compiled .rhel9 binary when building RHEL10 container
images.
- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@dkhater-redhat: This pull request references MCO-1976 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

In response to this:

- What I did
Added RHEL10 support to the MCD for SSH key path selection and binary compatibility across RHEL versions.

Changes made:

1. SSH Key Path Selection (pkg/daemon/update.go:2291): Added IsEL10() check to useNewSSHKeyPath() so RHEL10 correctly uses the new SSH key path (/home/core/.ssh/authorized_keys.d/ignition) instead of the legacy path (/home/core/.ssh/authorized_keys).

2. Binary Selection (pkg/daemon/daemon.go:540-548): Extended ReexecuteForTargetRoot() to handle RHEL10 containers managing RHEL8 or RHEL9 hosts by selecting the appropriate binary suffix (.rhel8 or .rhel9).

This addresses items [2] and [4] from the RHEL10 MCD compatibility spike, ensuring the MCD works correctly on RHEL10 and can manage older RHEL versions during dual-stream operations.

- How to verify it
For SSH Key Path (Item 2):

On a RHEL10 cluster:

1. Deploy the updated MCO image
2. Debug into a node and check MCD logs:

   oc logs -n openshift-machine-config-operator -l k8s-app=machine-config-daemon | grep "Writing SSH keys"

3. Should see: Writing SSH keys to "/home/core/.ssh/authorized_keys.d/ignition"
4. Should NOT see: Writing SSH keys to "/home/core/.ssh/authorized_keys"
5. Verify SSH keys are written to the correct location:
   oc debug node/
   chroot /host
   ls -la /home/core/.ssh/authorized_keys.d/ignition

For Binary Selection (Item 4):

When RHEL10 container images are available:

1. Deploy RHEL10 MCO container to a cluster with RHEL9 or RHEL8 nodes
2. Check MCD logs during re-execution:
   oc logs -n openshift-machine-config-operator -l k8s-app=machine-config-daemon | grep "container is rhel10"
3. Should see appropriate log:

• "container is rhel10, target is rhel9" (for RHEL9 nodes)
• "container is rhel10, target is rhel8" (for RHEL8 nodes)

Note: Item 4 also requires Dockerfile updates to include compiled .rhel9 binary when building RHEL10 container
images.
- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@dkhater-redhat: This pull request references MCO-1976 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

In response to this:

- What I did
Added RHEL10 support to the MCD for SSH key path selection and binary compatibility across RHEL versions.

Changes made:

1. SSH Key Path Selection (pkg/daemon/update.go:2291): Added IsEL10() check to useNewSSHKeyPath() so RHEL10 correctly uses the new SSH key path (/home/core/.ssh/authorized_keys.d/ignition) instead of the legacy path (/home/core/.ssh/authorized_keys).

2. Binary Selection (pkg/daemon/daemon.go:540-548): Extended ReexecuteForTargetRoot() to handle RHEL10 containers managing RHEL8 or RHEL9 hosts by selecting the appropriate binary suffix (.rhel8 or .rhel9).

This addresses items [2] and [4] from the RHEL10 MCD compatibility spike, ensuring the MCD works correctly on RHEL10 and can manage older RHEL versions during dual-stream operations.

- How to verify it
For SSH Key Path (Item 2):

On a RHEL10 cluster:

1. Deploy the updated MCO image
2. Debug into a node and check MCD logs:

   oc logs -n openshift-machine-config-operator -l k8s-app=machine-config-daemon | grep "ssh"

3. Should see: Writing SSH keys to "/home/core/.ssh/authorized_keys.d/ignition"
4. Should NOT see: Writing SSH keys to "/home/core/.ssh/authorized_keys"
5. Verify SSH keys are written to the correct location:
   oc debug node/
   chroot /host
   ls -la /home/core/.ssh/authorized_keys.d/ignition

For Binary Selection (Item 4):

When RHEL10 container images are available:

1. Deploy RHEL10 MCO container to a cluster with RHEL9 or RHEL8 nodes
2. Check MCD logs during re-execution:
   oc logs -n openshift-machine-config-operator -l k8s-app=machine-config-daemon | grep "container is rhel10"
3. Should see appropriate log:

• "container is rhel10, target is rhel9" (for RHEL9 nodes)
• "container is rhel10, target is rhel8" (for RHEL8 nodes)

Note: Item 4 also requires Dockerfile updates to include compiled .rhel9 binary when building RHEL10 container
images.
- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@dkhater-redhat: This pull request references MCO-1976 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

In response to this:

- What I did
Added RHEL10 support to the MCD for SSH key path selection and binary compatibility across RHEL versions.

Changes made:

1. SSH Key Path Selection (pkg/daemon/update.go:2291): Added IsEL10() check to useNewSSHKeyPath() so RHEL10 correctly uses the new SSH key path (/home/core/.ssh/authorized_keys.d/ignition) instead of the legacy path (/home/core/.ssh/authorized_keys).

2. Binary Selection (pkg/daemon/daemon.go:540-548): Extended ReexecuteForTargetRoot() to handle RHEL10 containers managing RHEL8 or RHEL9 hosts by selecting the appropriate binary suffix (.rhel8 or .rhel9).

This addresses items [2] and [4] from the RHEL10 MCD compatibility spike, ensuring the MCD works correctly on RHEL10 and can manage older RHEL versions during dual-stream operations.

- How to verify it
For SSH Key Path (Item 2):

On a RHEL10 cluster:

1. Deploy the updated MCO image
2. Debug into a node and check MCD logs:

   oc logs -n openshift-machine-config-operator -l k8s-app=machine-config-daemon | grep "ssh"

3. Should see: Writing SSH keys to "/home/core/.ssh/authorized_keys.d/ignition"
4. Should NOT see: Writing SSH keys to "/home/core/.ssh/authorized_keys"
5. Verify SSH keys are written to the correct location:
   oc debug node/
   chroot /host
   ls -la /home/core/.ssh/authorized_keys.d/ignition

For Binary Selection (Item 4):

When RHEL10 container images are available:

1. Deploy RHEL10 MCO container to a cluster with RHEL9 or RHEL8 nodes
2. Check MCD logs during re-execution:
   oc logs -n openshift-machine-config-operator -l k8s-app=machine-config-daemon | grep "container is rhel10"
3. Should see appropriate log:

• "container is rhel10, target is rhel9" (for RHEL9 nodes)
• "container is rhel10, target is rhel8" (for RHEL8 nodes)

Note: Item 4 also requires Dockerfile updates to include compiled .rhel9 binary when building RHEL10 container. When the MCO team builds RHEL10-based container images, we will be able to see this change.
images.
- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@dkhater-redhat: This pull request references MCO-1976 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

In response to this:

- What I did
Added RHEL10 support to the MCD for SSH key path selection and binary compatibility across RHEL versions.

Changes made:

1. SSH Key Path Selection (pkg/daemon/update.go:2291): Added IsEL10() check to useNewSSHKeyPath() so RHEL10 correctly uses the new SSH key path (/home/core/.ssh/authorized_keys.d/ignition) instead of the legacy path (/home/core/.ssh/authorized_keys).

2. Binary Selection (pkg/daemon/daemon.go:540-548): Extended ReexecuteForTargetRoot() to handle RHEL10 containers managing RHEL8 or RHEL9 hosts by selecting the appropriate binary suffix (.rhel8 or .rhel9).

This addresses items [2] and [4] from the RHEL10 MCD compatibility spike, ensuring the MCD works correctly on RHEL10 and can manage older RHEL versions during dual-stream operations.

- How to verify it
For SSH Key Path (Item 2):

On a RHEL10 cluster:

1. Deploy the updated MCO image
2. Debug into a node and check MCD logs:

   oc logs -n openshift-machine-config-operator -l k8s-app=machine-config-daemon | grep "ssh"

3. Should see: Writing SSH keys to "/home/core/.ssh/authorized_keys.d/ignition"
4. Should NOT see: Writing SSH keys to "/home/core/.ssh/authorized_keys"
5. Verify SSH keys are written to the correct location:
   oc debug node/
   chroot /host
   ls -la /home/core/.ssh/authorized_keys.d/ignition

For Binary Selection (Item 4):

When RHEL10 container images are available:

1. Deploy RHEL10 MCO container to a cluster with RHEL9 or RHEL8 nodes
2. Check MCD logs during re-execution:
   oc logs -n openshift-machine-config-operator -l k8s-app=machine-config-daemon | grep "container is rhel10"
3. Should see appropriate log:

• "container is rhel10, target is rhel9" (for RHEL9 nodes)
• "container is rhel10, target is rhel8" (for RHEL8 nodes)

Note: Item 4 also requires Dockerfile updates to include compiled .rhel9 binary when building RHEL10 container images. When the MCO team builds RHEL10-based container images, we will be able to see this change.
- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[dkhater-redhat]

/jira refresh

[openshift-ci-robot]

@dkhater-redhat: This pull request references MCO-1976 which is a valid jira issue.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[sergiordlr]

All password test cases passed using rhel9 and rhel10

After configuring RHEL10 in worker nodes and RHEL9 in master nodes we can see this;

Both, master and worker nodes are using the right binary (the already existing one)

$ nodelogs ip-10-0-15-243.us-east-2.compute.internal  |grep "using ap"
I1119 16:15:42.890891  115901 daemon.go:551] using appropriate binary for source=rhel-9 target=rhel-10
$ nodelogs ip-10-0-31-0.us-east-2.compute.internal |grep "using ap"
I1119 09:30:45.092246    5869 daemon.go:551] using appropriate binary for source=rhel-9 target=rhel-9

We have no access to a rhel10 payload, so we couldnt test the changes rhel10 -> rhel10 and rhel10 -> rhel9

No issues where found.

/label qe-approved
/label cherry-pick-approved

[openshift-ci-robot]

@dkhater-redhat: This pull request references MCO-1976 which is a valid jira issue.

In response to this:

- What I did
Added RHEL10 support to the MCD for SSH key path selection and binary compatibility across RHEL versions.

Changes made:

1. SSH Key Path Selection (pkg/daemon/update.go:2291): Added IsEL10() check to useNewSSHKeyPath() so RHEL10 correctly uses the new SSH key path (/home/core/.ssh/authorized_keys.d/ignition) instead of the legacy path (/home/core/.ssh/authorized_keys).

2. Binary Selection (pkg/daemon/daemon.go:540-548): Extended ReexecuteForTargetRoot() to handle RHEL10 containers managing RHEL8 or RHEL9 hosts by selecting the appropriate binary suffix (.rhel8 or .rhel9).

This addresses items [2] and [4] from the RHEL10 MCD compatibility spike, ensuring the MCD works correctly on RHEL10 and can manage older RHEL versions during dual-stream operations.

- How to verify it
For SSH Key Path (Item 2):

On a RHEL10 cluster:

1. Deploy the updated MCO image
2. Debug into a node and check MCD logs:

   oc logs -n openshift-machine-config-operator -l k8s-app=machine-config-daemon | grep "ssh"

3. Should see: Writing SSH keys to "/home/core/.ssh/authorized_keys.d/ignition"
4. Should NOT see: Writing SSH keys to "/home/core/.ssh/authorized_keys"
5. Verify SSH keys are written to the correct location:
   oc debug node/
   chroot /host
   ls -la /home/core/.ssh/authorized_keys.d/ignition

For Binary Selection (Item 4):

When RHEL10 container images are available:

1. Deploy RHEL10 MCO container to a cluster with RHEL9 or RHEL8 nodes
2. Check MCD logs during re-execution:
   oc logs -n openshift-machine-config-operator -l k8s-app=machine-config-daemon | grep "container is rhel10"
3. Should see appropriate log:

• "container is rhel10, target is rhel9" (for RHEL9 nodes)
• "container is rhel10, target is rhel8" (for RHEL8 nodes)

Note: Item 4 also requires Dockerfile updates to include compiled .rhel9 binary when building RHEL10 container images. When the MCO team builds RHEL10-based container images, we will be able to see this change.
- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[sergiordlr]

/remove-label cherry-pick-approved
/verified by @sergiordlr

[openshift-ci-robot]

@sergiordlr: This PR has been marked as verified by @sergiordlr.

In response to this:

/remove-label cherry-pick-approved
/verified by @sergiordlr

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[umohnani8]

/lgtm

[isabella-janssen]

/lgtm

Change looks clear & change was verified by QE.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dkhater-redhat, isabella-janssen, umohnani8

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [dkhater-redhat,isabella-janssen,umohnani8]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dkhater-redhat, isabella-janssen, umohnani8

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [dkhater-redhat,isabella-janssen,umohnani8]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 759466d and 2 for PR HEAD 2cabd9c in total

[openshift-ci]

@dkhater-redhat: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/bootstrap-unit	2cabd9c	link	false	/test bootstrap-unit

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[pablintino]

Just my 2 cents, but I don't think continuing with version-specific conditionals is the right direction. This piece of code will need a revisit each time there's a new major, like now.
WDYT about something like this?

diff --git a/pkg/daemon/daemon.go b/pkg/daemon/daemon.go
index e990021ed..01170f701 100644
--- a/pkg/daemon/daemon.go
+++ b/pkg/daemon/daemon.go
@@ -14,7 +14,9 @@ import (
        "os/exec"
        "os/signal"
        "path/filepath"
+       "regexp"
        "slices"
+       "strconv"
        "strings"
        "sync"
        "syscall"
@@ -531,20 +533,34 @@ func ReexecuteForTargetRoot(target string) error {
                return fmt.Errorf("failed to get target OS: %w", err)
        }
 
-       var sourceBinarySuffix string
+       sourceBinary := "/usr/bin/machine-config-daemon"
        if sourceOsVersion.IsLikeRHEL() && targetOsVersion.IsLikeRHEL() {
-               sourceMajor := sourceOsVersion.BaseVersionMajor()
-               targetMajor := targetOsVersion.BaseVersionMajor()
-               if sourceMajor == "9" && targetMajor == "8" {
-                       sourceBinarySuffix = ".rhel8"
-                       klog.Info("container is rhel9, target is rhel8")
+               sourceVersion, err := strconv.Atoi(sourceOsVersion.BaseVersionMajor())
+               if err != nil {
+                       return fmt.Errorf("failed to convert source major version: %w", err)
+               }
+               targetVersion, err := strconv.Atoi(targetOsVersion.BaseVersionMajor())
+               if err != nil {
+                       return fmt.Errorf("failed to convert target major version: %w", err)
+               }
+               if sourceVersion > targetVersion {
+                       suffixes, err := getVersionedBinarySuffixes(sourceBinary + "*")
+                       if err != nil {
+                               return fmt.Errorf("failed to get machine-config-daemon versioned binaries: %w", err)
+                       }
+                       suffix, exists := suffixes[targetVersion]
+                       if exists {
+                               sourceBinary += suffix
+                       }
+                       klog.Infof("container is rhel%d, target is rhel%d", sourceVersion, targetVersion)
                } else {
-                       klog.Infof("using appropriate binary for source=rhel-%s target=rhel-%s", sourceMajor, targetMajor)
+                       klog.Infof("using appropriate binary for source=rhel-%d target=rhel-%d", sourceVersion, targetVersion)
                }
+
        } else {
                klog.Info("assuming we can use container binary chroot() to host")
        }
-       sourceBinary := "/usr/bin/machine-config-daemon" + sourceBinarySuffix
+
        src, err := os.Open(sourceBinary)
        if err != nil {
                return fmt.Errorf("opening %s: %w", sourceBinary, err)
@@ -596,6 +612,33 @@ func ReexecuteForTargetRoot(target string) error {
        return syscall.Exec(targetBin, newArgv, newEnv)
 }
 
+var regexDigits = regexp.MustCompile(`(\d)*$`)
+
+func getVersionedBinarySuffixes(basePath string) (map[int]string, error) {
+       result := map[int]string{}
+       matches, err := filepath.Glob(basePath)
+       if err != nil {
+               return nil, err
+       }
+       for _, match := range matches {
+               extension := filepath.Ext(match)
+               if extension != "" {
+                       versionBytes := regexDigits.Find([]byte(extension))
+                       if versionBytes == nil {
+                               // Skip, is a binary that doesn't have the pattern we are looking for
+                               continue
+                       }
+
+                       version, err := strconv.Atoi(string(versionBytes))
+                       if err != nil {
+                               return nil, err
+                       }
+                       result[version] = extension
+               }
+       }
+       return result, nil
+}
+
 // worker runs a worker thread that just dequeues items, processes them, and marks them done.
 // It enforces that the syncHandler is never invoked concurrently with the same key.
 func (dn *Daemon) worker() {

[isabella-janssen]

/retest-required

[dkhater-redhat]

Spoke with Pablo and I completely agree (and actually considered that when creating this PR, but opted for this method because we have time constraints and the if statement isn't too bad for now). We agreed that we will address this for RHEL 11. We are in a time crunch due to the soft freeze.

[isabella-janssen]

We agreed that we will address this for RHEL 11. We are in a time crunch due to the soft freeze.

@dkhater-redhat Do you think it's worth creating a tech debt card in Jira to track this to make the proposal is not lost?

[dkhater-redhat]

tech debt card to address pablo's callout https://issues.redhat.com/browse/MCO-1989