sapstartsrv cannot start in NetWeaver-Distributed

[tstaerk]

I deleted all VMs installed by Terraform so far and now I run:

./ansible-wrapper ./stacks/NetWeaver-Distributed/playbook.yml --extra-vars "@./stacks/NetWeaver-Distributed/vars/deploy-vars.yml"

and I get the error message

TASK [sap-hana-install : Upgrade SAP HANA started] *****************************
fatal: [10.164.0.2]: FAILED! => {"changed": true, "cmd": ["./hdblcm", "--configfile=/tmp/ansible.zkqCjmhanaconfig/configfile.cfg", "--action=update", "--ignore=check_signature_file", "--update_execution_mode=optimized", "-b"], "delta": "0:03:34.141952", "end": "2022-03-25 14:32:53.250182", "msg": "non-zero return code", "rc": 1, "start": "2022-03-25 14:29:19.108230", "stderr": "Update of SAP HANA Database components failed.\n Update of SAP HANA Database failed\n Upgrade failed\n Error checking Upgrade\n Checking upgrade failed\n Cannot start local sapstartsrv\n Waiting for sapstartsrv failed: timeout reached (120)\n Looking for error messages in sapstartsrv trace file\n sapstartsrv.log: Root PSE verification failed (ssfPkiCheckOrCreateRootPSE): Could not write to Secure Storage\n sapstartsrv.log: SapSSLInit failed => https support disabled\n Tcp port '50013' is already in use by process:\n /usr/sap/H01/HDB00/exe/sapstartsrv pf=/hana/shared/H01/profile/H01_HDB00_hana-dev -D -u h01adm (pid = 30821, user = h01adm)", "stderr_lines": ["Update of SAP HANA Database components failed.", " Update of SAP HANA Database failed", " Upgrade failed", " Error checking Upgrade", " Checking upgrade failed", " Cannot start local sapstartsrv", " Waiting for sapstartsrv failed: timeout reached (120)", " Looking for error messages in sapstartsrv trace file", " sapstartsrv.log: Root PSE verification failed (ssfPkiCheckOrCreateRootPSE): Could not write to Secure Storage", " sapstartsrv.log: SapSSLInit failed => https support disabled", " Tcp port '50013' is already in use by process:", " /usr/sap/H01/HDB00/exe/sapstartsrv pf=/hana/shared/H01/profile/H01_HDB00_hana-dev -D -u h01adm (pid = 30821, user = h01adm)"], "stdout": "\n\nSAP HANA Lifecycle Management - SAP HANA Database 2.00.037.01.1557479495\n************************************************************************\n\n\nScanning software locations...\nDetected components:\n SAP HANA Database (2.00.037.01.1557479495) in /hana/shared/software/SAP_HANA_DATABASE/server\n\nSummary before execution:\n=========================\n\nSAP HANA Database\n Update Parameters\n SAP HANA System ID: H01\n Remote Execution: ssh\n Enable the installation or upgrade of the SAP Host Agent: No\n Update Execution Mode: optimized\n System Database User Name: SYSTEM\n Software Components\n SAP HANA Database\n Update from version 2.00.037.01.1557479495 to 2.00.037.01.1557479495\n Location: /hana/shared/software/SAP_HANA_DATABASE/server\n\nNote: Cannot verify database user (SYSTEM) password in advance: no connection available.\nNote: Cannot perform license check: no connection available\nNote: Parameter 'root_password' will not be used. It is not applicable in the current context.\nNote: Parameter 'sapadm_password' will not be used. It is not applicable in the current context.\nNote: Parameter 'password' will not be used. It is not applicable in the current context.\nNote: Parameter 'xs_components' will not be used. It is not applicable in the current context.\nUpdating components...\nPreparing update of SAP HANA Database...\n Starting local sapstartsrv...\nYou can send feedback to SAP with this form: https://hana-dev:1129/lmsl/HDBLCM/H01/feedback/feedback.html\nLog file written to '/var/tmp/hdb_H01_hdblcm_update_2022-03-25_14.29.19/hdblcm.log' on host 'hana-dev'.", "stdout_lines": ["", "", "SAP HANA Lifecycle Management - SAP HANA Database 2.00.037.01.1557479495", "************************************************************************", "", "", "Scanning software locations...", "Detected components:", " SAP HANA Database (2.00.037.01.1557479495) in /hana/shared/software/SAP_HANA_DATABASE/server", "", "Summary before execution:", "=========================", "", "SAP HANA Database", " Update Parameters", " SAP HANA System ID: H01", " Remote Execution: ssh", " Enable the installation or upgrade of the SAP Host Agent: No", " Update Execution Mode: optimized", " System Database User Name: SYSTEM", " Software Components", " SAP HANA Database", " Update from version 2.00.037.01.1557479495 to 2.00.037.01.1557479495", " Location: /hana/shared/software/SAP_HANA_DATABASE/server", "", "Note: Cannot verify database user (SYSTEM) password in advance: no connection available.", "Note: Cannot perform license check: no connection available", "Note: Parameter 'root_password' will not be used. It is not applicable in the current context.", "Note: Parameter 'sapadm_password' will not be used. It is not applicable in the current context.", "Note: Parameter 'password' will not be used. It is not applicable in the current context.", "Note: Parameter 'xs_components' will not be used. It is not applicable in the current context.", "Updating components...", "Preparing update of SAP HANA Database...", " Starting local sapstartsrv...", "You can send feedback to SAP with this form: https://hana-dev:1129/lmsl/HDBLCM/H01/feedback/feedback.html", "Log file written to '/var/tmp/hdb_H01_hdblcm_update_2022-03-25_14.29.19/hdblcm.log' on host 'hana-dev'."]}

Looking into /var/tmp/...hdblcm.log, I see:

14:32:53.067 - ERR : Update of SAP HANA Database components failed.
14:32:53.082 - INFO: Summary of critical errors
14:32:53.067 - ERR : Update of SAP HANA Database components failed.
14:32:51.970 - ERR : Update of SAP HANA Database failed
14:32:51.969 - INFO: Upgrade failed
14:32:51.969 - INFO: Error checking Upgrade
14:32:51.969 - INFO: Checking upgrade failed
14:32:51.969 - INFO: Cannot start local sapstartsrv
14:32:51.969 - INFO: Waiting for sapstartsrv failed: timeout reached (120)
14:32:51.969 - INFO: Looking for error messages in sapstartsrv trace file
14:32:51.969 - INFO: sapstartsrv.log: Root PSE verification failed (ssfPkiCheckOrCreateRootPSE): Could not write to Secure Storage
14:32:51.969 - INFO: sapstartsrv.log: SapSSLInit failed => https support disabled
14:32:51.970 - INFO: Tcp port '50013' is already in use by process:
14:32:51.970 - INFO: /usr/sap/H01/HDB00/exe/sapstartsrv pf=/hana/shared/H01/profile/H01_HDB00_hana-dev -D -u h01adm (pid = 30821, user = h01adm)

[tstaerk]

Looking at the hana-dev server, indeed, Port 50013 is blocked by sapstartsrv:

hana-dev:/var/tmp/hdb_H01_hdblcm_update_2022-03-29_08.08.18 # netstat -putan | grep 500
tcp        0      0 0.0.0.0:50013           0.0.0.0:*               LISTEN      30540/sapstartsrv   

I wonder if the error "Could not write to Secure Storage" is just an errror message resulting from the fact that ssfPkiCheckOrCreateRootPSE tried to call sapstartsrv and failed.

[tstaerk]

I could manually start sapstartsrv, here is how:

h01adm@hana-dev:~> export LD_LIBRARY_PATH=/hana/shared/H01/exe/linuxx86_64/HDB_2.00.030.00.1522209842_c6e59dfe8322e948bc9bec3b1ef59da7708f0f4e/
h01adm@hana-dev:~> /usr/sap/H01/HDB00/exe/sapstartsrv pf=/hana/shared/H01/profile/H01_HDB00_hana-dev -D -u h01adm
Impromptu CCC initialization by 'rscpCInit'.
  See SAP note 1266393.

[tstaerk]

I could work around this issue by using SLES15 in the bastion host, but I would be thankful for any hint of systematically troubleshooting what actually happened.