MDBF-960: Document environment variables in master-libvirt #706
Conversation
cvicentiu
force-pushed
the
refactor_master_libvirt
branch
3 times, most recently
from
7ae4362 to
74c34d8
Compare
cvicentiu
changed the title
cvicentiu
force-pushed
the
refactor_master_libvirt
branch
from
74c34d8 to
86d1a4c
Compare
|
@fauust I need your eyes on the bash_lib.sh changes. I'm no expert in bash writing, but I have tested that the URLs are generated properly on almalinux8. |
|
@RazvanLiviuVarzaru I need your review on the other code changes. Again, all these were tested locally and I don't expect surprises, but I may have missed something. |
There was a problem hiding this comment.
There are a lot of changes so this is quite complex to review without testing.
Your key is not deployed on VMs but I will deploy them today and explain you how you can manually test your changes on the DEV platform.
For now, error is:
INFO: setup MariaDB repository for 11.7 branch
+ command -v wget
+ local baseurl
+ wget -q --spider https://rpm.mariadb.org/11.7/almalinux9-almalinux9-amd64
+ wget -q --spider https://archive.mariadb.org/mariadb-11.7/yum//almalinux9-almalinux9-amd64
./bash_lib.sh: line 372: baseurl: unbound variableAlmalinux 9 AMD64 running :
./rpm-upgrade.sh https://buildbot.mariadb.org/#/builders/612/builds/1817.
|
Ok, your keys are deployed (or currently being deployed on DEV), here is how to test those scripts: 1/ start a VM on the host, example with AMD64 and almalinux 9: ❯ ssh ro-apexis-bbw03-x64
faust@ro-apexis-bbw03-x64:~$ sudo virsh list --all
Id Name State
-----------------------------------------------------------------------
1 bb-rhel9-docker running
260 bb-ro-apexis-bbw03-x64-openeuler-2403-amd64_install running
- bb-ro-apexis-bbw03-x64-almalinux-8-amd64 shut off
- bb-ro-apexis-bbw03-x64-almalinux-9-amd64 shut off
- bb-ro-apexis-bbw03-x64-centos-7-amd64 shut off
- bb-ro-apexis-bbw03-x64-centos-stream9-amd64 shut off
- bb-ro-apexis-bbw03-x64-debian-10-amd64 shut off
- bb-ro-apexis-bbw03-x64-debian-11-amd64 shut off
- bb-ro-apexis-bbw03-x64-debian-12-amd64 shut off
- bb-ro-apexis-bbw03-x64-debian-sid-amd64 shut off
- bb-ro-apexis-bbw03-x64-fedora-38-amd64 shut off
- bb-ro-apexis-bbw03-x64-fedora-39-amd64 shut off
- bb-ro-apexis-bbw03-x64-fedora-40-amd64 shut off
- bb-ro-apexis-bbw03-x64-fedora-41-amd64 shut off
- bb-ro-apexis-bbw03-x64-opensuse-15-amd64 shut off
- bb-ro-apexis-bbw03-x64-opensuse-1505-amd64 shut off
- bb-ro-apexis-bbw03-x64-opensuse-1506-amd64 shut off
- bb-ro-apexis-bbw03-x64-rhel-8-amd64 shut off
- bb-ro-apexis-bbw03-x64-rhel-9-amd64 shut off
- bb-ro-apexis-bbw03-x64-rockylinux-8-amd64 shut off
- bb-ro-apexis-bbw03-x64-rockylinux-9-amd64 shut off
- bb-ro-apexis-bbw03-x64-sles-1506-amd64 shut off
- bb-ro-apexis-bbw03-x64-ubuntu-1804-amd64 shut off
- bb-ro-apexis-bbw03-x64-ubuntu-2004-amd64 shut off
- bb-ro-apexis-bbw03-x64-ubuntu-2204-amd64 shut off
- bb-ro-apexis-bbw03-x64-ubuntu-2404-amd64 shut off
- bb-ro-apexis-bbw03-x64-ubuntu-2410-amd64 shut off
faust@ro-apexis-bbw03-x64:~$ sudo virsh start bb-ro-apexis-bbw03-x64-almalinux-9-amd642/ find the VM IP: faust@ro-apexis-bbw03-x64:~$ sudo virsh net-dhcp-leases default
Expiry Time MAC address Protocol IP address Hostname Client ID or DUID
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2025-02-12 15:57:57 52:54:00:6f:7f:d8 ipv4 192.168.122.244/24 bb-ro-apexis-bbw03-x64-almalinux-9-amd64 01:52:54:00:6f:7f:d83/ make sure your local ssh config contain something similar to: host bb-vm
hostname 192.168.122.244
User buildbot
ProxyJump ro-apexis-bbw03-x644/ scp the scripts to the VM and run a BB build: ~/buildbot refactor_master_libvirt
.venv ❯ scp -r scripts/ bb-vm:
deb-install.sh 100% 5705 52.1KB/s 00:00
docker-library-build.sh 100% 5319 48.8KB/s 00:00
pam-test.sh 100% 2289 21.5KB/s 00:00
aix-build-and-test.sh 100% 4896 35.6KB/s 00:00
rpm-install.sh 100% 3206 30.1KB/s 00:00
rpm-upgrade.sh 100% 8763 80.4KB/s 00:00
deb-upgrade.sh 100% 7851 73.5KB/s 00:00
docker-library-test.sh 100% 637 6.0KB/s 00:00
service-test.sh 100% 7331 69.4KB/s 00:00
bb_package_rotation.sh 100% 1488 14.1KB/s 00:00
docker-library-manifest.sh 100% 6322 46.4KB/s 00:00
bash_lib.sh 100% 25KB 157.0KB/s 00:00
~/buildbot refactor_master_libvirt 9s
.venv ❯ ssh bb-vm
[buildbot@bb-ro-apexis-bbw03-x64-almalinux-9-amd64 ~]$ cd scripts/
[buildbot@bb-ro-apexis-bbw03-x64-almalinux-9-amd64 scripts]$ ./rpm-upgrade.sh https://buildbot.mariadb.org/#/builders/612/builds/18175/ power off the VM ( |
cvicentiu
force-pushed
the
refactor_master_libvirt
branch
from
86d1a4c to
0042bfc
Compare
Fixed this issue. |
@fauust Don't we need buildbot key-pair? It doesn't work for me. |
| "dist_name": os_name, | ||
| "version_name": OS_INFO[os_info_name]["version_name"], | ||
| "arch": build_arch, | ||
| "arch": platform, |
There was a problem hiding this comment.
@cvicentiu We need to test on Dev if a re-build is causing a re-calculation of the "arch" property.
My guess is Yes, but if "arch" is not updated on re-build then we end up with an incompatible change with the scripts.
We need to be sure that we can re-build runs at least for a couple of days in the past.
There was a problem hiding this comment.
It would be logical for properties to be recomputed, but yes, I agree, we should check this.
There was a problem hiding this comment.
I tested this now, the properties are re-computed on rebuild but there are 2 conditions for a production release:
https://buildbot.dev.mariadb.org/#/builders/231/builds/41
arch: "almalinux8-aarch64"
re-run: https://buildbot.dev.mariadb.org/#/builders/231/builds/42
arch: "aarch64"
- After this patch is merged to Main, we need to immediately deploy the buildbot configuration changes to PROD and restart
master-libvirt - No libvirt builders running at the time of the merge to Main
Why?
- the worker is downloading the script from the main branch
- bash_lib changes are incompatible with the current buildbot configuration so both pieces (script,bb config) need to be deployed together
@cvicentiu @fauust I will test this PR on VM's by replacing "arch" value from the build instance with what will result from the changes in this patch. Will post the results soon. |
Scripts OK! |
* Also make the env variables uniform, a list shared by all builders.
cvicentiu
force-pushed
the
refactor_master_libvirt
branch
2 times, most recently
from
6bc982e to
526d8eb
Compare
…scripts The arch environment variable was overused to generate proper repository paths, when the whole information was already available as part of dist_name and version_name. This patch moves the URL transformation logic within bash_lib.sh, allowing master_libvirt to pass in fewer environment variables.
* Eliminate statically hardcoding environment variables such as tarballnum * DEVELOPMENT_BRANCH is not used / maintained anymore, so should be dropped.
cvicentiu
force-pushed
the
refactor_master_libvirt
branch
from
526d8eb to
86aef1c
Compare
|
@RazvanLiviuVarzaru Ready to be merged to dev. You mentioned the need to push some changes together to prevent breakages, so I'll let you merge this when you see fit. |
3 participants
This pull request cleans up and documents a lot of variables in master-libvirt.
Additionally it refactors bash_lib to handle URL archive & logic instead of having it coded into the master.cfg.