Dev: sbd: Improve the process of leveraging maintenance mode

[liangxin1300]

Problem

#1744 leverage maintenance mode when needs to restart cluster, but there are still some problems when resources are running:

Configuration changed before hinting, might lead to inconsistent

# crm sbd configure watchdog-timeout=45
INFO: No 'msgwait-timeout=' specified in the command, use 2*watchdog timeout: 90
INFO: Configuring disk-based SBD
INFO: Initializing SBD device /dev/sda5
INFO: Update SBD_WATCHDOG_DEV in /etc/sysconfig/sbd: /dev/watchdog0
INFO: Sync file /etc/sysconfig/sbd to sle16-2
INFO: Already synced /etc/sysconfig/sbd to all nodes
INFO: Update SBD_DELAY_START in /etc/sysconfig/sbd: 131
INFO: Sync file /etc/sysconfig/sbd to sle16-2
INFO: Already synced /etc/sysconfig/sbd to all nodes
WARNING: "stonith-timeout" in crm_config is set to 119, it was 71
INFO: Sync directory /etc/systemd/system/sbd.service.d to sle16-2
WARNING: Resource is running, need to restart cluster service manually on each node
WARNING: Or, run with `crm -F` or `--force` option, the `sbd` subcommand will leverage maintenance mode for any changes that require restarting sbd.service
WARNING: Understand risks that running RA has no cluster protection while the cluster is in maintenance mode and restarting

# crm sbd purge 
INFO: Stop sbd resource 'stonith-sbd'(stonith:fence_sbd)
INFO: Remove sbd resource 'stonith-sbd'
INFO: Disable sbd.service on node sle16-1
INFO: Disable sbd.service on node sle16-2
INFO: Move /etc/sysconfig/sbd to /etc/sysconfig/sbd.bak on all nodes
INFO: Delete cluster property "stonith-timeout" in crm_config
INFO: Delete cluster property "priority-fencing-delay" in crm_config
WARNING: "stonith-enabled" in crm_config is set to false, it was true
WARNING: Resource is running, need to restart cluster service manually on each node
WARNING: Or, run with `crm -F` or `--force` option, the `sbd` subcommand will leverage maintenance mode for any changes that require restarting sbd.service
WARNING: Understand risks that running RA has no cluster protection while the cluster is in maintenance mode and restarting

Pacemaker fatal exit when adding diskless sbd on a running cluster with resources running

 # crm cluster init sbd -S -y
INFO: Loading "default" profile from /etc/crm/profiles.yml
INFO: Loading "knet-default" profile from /etc/crm/profiles.yml
INFO: Configuring diskless SBD
WARNING: Diskless SBD requires cluster with three or more nodes. If you want to use diskless SBD for 2-node cluster, should be combined with QDevice.
INFO: Update SBD_WATCHDOG_TIMEOUT in /etc/sysconfig/sbd: 15
INFO: Update SBD_WATCHDOG_DEV in /etc/sysconfig/sbd: /dev/watchdog0
INFO: Sync file /etc/sysconfig/sbd to sle16-2
INFO: Already synced /etc/sysconfig/sbd to all nodes
INFO: Enable sbd.service on node sle16-1
INFO: Enable sbd.service on node sle16-2
WARNING: Resource is running, need to restart cluster service manually on each node
WARNING: Or, run with `crm -F` or `--force` option, the `sbd` subcommand will leverage maintenance mode for any changes that require restarting sbd.service
WARNING: Understand risks that running RA has no cluster protection while the cluster is in maintenance mode and restarting
WARNING: "stonith-watchdog-timeout" in crm_config is set to 30, it was 0

Broadcast message from systemd-journald@sle16-1 (Thu 2025-10-23 10:54:11 CEST):

pacemaker-controld[5674]:  emerg: Shutting down: stonith-watchdog-timeout configured (30) but SBD not active


Message from syslogd@sle16-1 at Oct 23 10:54:11 ...
 pacemaker-controld[5674]:  emerg: Shutting down: stonith-watchdog-timeout configured (30) but SBD not active
ERROR: cluster.init: Failed to run 'crm configure property stonith-watchdog-timeout=30': ERROR: Failed to run 'crm_mon -1rR': crm_mon: Connection to cluster failed: Connection refused

Solution

• Drop the function restart_cluster_if_possible
• Introduced a new function utils.able_to_restart_cluster to check if the cluster can be restarted. Call it before changing any configurations.
• Add leverage maintenance mode in sbd device remove and sbd purge commands

Add sbd via sbd stage while resource is running

 # crm cluster init sbd -S -y
INFO: Loading "default" profile from /etc/crm/profiles.yml
INFO: Loading "knet-default" profile from /etc/crm/profiles.yml
WARNING: Please stop all running resources and try again
WARNING: Or use 'crm -F/--force' option to leverage maintenance mode
WARNING: Understand risks that running RA has no cluster protection while the cluster is in maintenance mode and restarting
INFO: Aborting the configuration change attempt
INFO: Done (log saved to /var/log/crmsh/crmsh.log on sle16-1)

# Leverage maintenance mode
# crm -F cluster init sbd -S -y
INFO: Loading "default" profile from /etc/crm/profiles.yml
INFO: Loading "knet-default" profile from /etc/crm/profiles.yml
INFO: Set cluster to maintenance mode
WARNING: "maintenance-mode" in crm_config is set to true, it was false
INFO: Configuring diskless SBD
WARNING: Diskless SBD requires cluster with three or more nodes. If you want to use diskless SBD for 2-node cluster, should be combined with QDevice.
INFO: Update SBD_WATCHDOG_TIMEOUT in /etc/sysconfig/sbd: 15
INFO: Update SBD_WATCHDOG_DEV in /etc/sysconfig/sbd: /dev/watchdog0
INFO: Sync file /etc/sysconfig/sbd to sle16-2
INFO: Already synced /etc/sysconfig/sbd to all nodes
INFO: Enable sbd.service on node sle16-1
INFO: Enable sbd.service on node sle16-2
INFO: Restarting cluster service
INFO: BEGIN Waiting for cluster
...........                                                                                                                                                                                                                            
INFO: END Waiting for cluster
WARNING: "stonith-watchdog-timeout" in crm_config is set to 30, it was 0
WARNING: "stonith-enabled" in crm_config is set to true, it was false
INFO: Update SBD_DELAY_START in /etc/sysconfig/sbd: 41
INFO: Sync file /etc/sysconfig/sbd to sle16-2
INFO: Already synced /etc/sysconfig/sbd to all nodes
WARNING: "stonith-timeout" in crm_config is set to 71, it was 60s
INFO: Set cluster from maintenance mode to normal
INFO: Delete cluster property "maintenance-mode" in crm_config
INFO: Done (log saved to /var/log/crmsh/crmsh.log on sle16-1)

Purge sbd while resource is running

# crm sbd purge 
WARNING: Please stop all running resources and try again
WARNING: Or use 'crm -F/--force' option to leverage maintenance mode
WARNING: Understand risks that running RA has no cluster protection while the cluster is in maintenance mode and restarting
INFO: Aborting the configuration change attempt

Add device

 # crm sbd device add /dev/sda6
INFO: Configured sbd devices: /dev/sda5
INFO: Append devices: /dev/sda6
WARNING: Please stop all running resources and try again
WARNING: Or use 'crm -F/--force' option to leverage maintenance mode
WARNING: Understand risks that running RA has no cluster protection while the cluster is in maintenance mode and restarting
INFO: Aborting the configuration change attempt

Remove device

# crm sbd device remove /dev/sda6
INFO: Configured sbd devices: /dev/sda5;/dev/sda6
INFO: Remove devices: /dev/sda6
WARNING: Please stop all running resources and try again
WARNING: Or use 'crm -F/--force' option to leverage maintenance mode
WARNING: Understand risks that running RA has no cluster protection while the cluster is in maintenance mode and restarting
INFO: Aborting the configuration change attempt

Configure sbd while DLM is running

# crm sbd configure watchdog-timeout=40
INFO: No 'msgwait-timeout=' specified in the command, use 2*watchdog timeout: 80
WARNING: Please stop all running resources and try again
WARNING: Or use 'crm -F/--force' option to leverage maintenance mode
WARNING: Understand risks that running RA has no cluster protection while the cluster is in maintenance mode and restarting
INFO: Aborting the configuration change attempt

# Leverage maintenance mode
# crm -F sbd configure watchdog-timeout=40
INFO: No 'msgwait-timeout=' specified in the command, use 2*watchdog timeout: 80
INFO: Set cluster to maintenance mode
WARNING: "maintenance-mode" in crm_config is set to true, it was false
WARNING: Please stop DLM related resources (gfs2-clone) and try again
INFO: Set cluster from maintenance mode to normal
INFO: Delete cluster property "maintenance-mode" in crm_config

[codecov]

Codecov Report

❌ Patch coverage is 36.73469% with 31 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.70%. Comparing base (722ae58) to head (b387201).
⚠️ Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
crmsh/utils.py	17.64%	14 Missing ⚠️
crmsh/ui_sbd.py	48.00%	13 Missing ⚠️
crmsh/sbd.py	50.00%	2 Missing ⚠️
crmsh/xmlutil.py	33.33%	2 Missing ⚠️

Additional details and impacted files

Flag	Coverage Δ
integration	55.14% <4.08%> (-0.05%)	⬇️
unit	52.88% <36.73%> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crmsh/sbd.py	83.65% <50.00%> (-0.34%)	⬇️
crmsh/xmlutil.py	70.39% <33.33%> (-0.12%)	⬇️
crmsh/ui_sbd.py	83.13% <48.00%> (-0.64%)	⬇️
crmsh/utils.py	67.14% <17.64%> (-0.43%)	⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[liangxin1300]

• Missing: need to hide the whole process of configuring crashdump watchdog timeout into init_and_deploy
  See the below: sbd sysconfig changed before raising maintenance hint

# crm sbd configure crashdump-watchdog-timeout=40
...
INFO: Set crashdump option for fence_sbd resource
INFO: Set msgwait-timeout to 2*watchdog-timeout + crashdump-watchdog-timeout: 70
INFO: Update SBD_TIMEOUT_ACTION in /etc/sysconfig/sbd: flush,crashdump
WARNING: Please stop all running resources and try again
WARNING: Or run this command with -F/--force option to leverage maintenance mode
WARNING: Understand risks that running RA has no cluster protection while the cluster is in maintenance mode and restarting

Improve this in #1978