From 3218c5d75bb81201bf7e2372c97146a88ed4b217 Mon Sep 17 00:00:00 2001 From: Bryan Vaz Date: Thu, 25 Nov 2021 17:23:50 -0500 Subject: [PATCH 1/3] docs: add PCI passthrough removal instructions Signed-off-by: Bryan Vaz --- docs/compute.md | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/docs/compute.md b/docs/compute.md index b281ec81..2a057c97 100644 --- a/docs/compute.md +++ b/docs/compute.md @@ -103,6 +103,54 @@ This kernel parameter is not retained when you upgrade an XCP-ng host [using the `[root@xen ~]# xe vm-start uuid=` +## PCI Passthrough Removal +To remove a PCI device that has been attached to a VM via PCI passthrough: + +### 1. List PCI Devices with `lspci` +_This is to make sure you remove the correct PCI device_ +``` +[root@xen ~]# lspci +... +04:01.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05) +``` + +### 2. Make sure your VM has halted +``` +[root@xen ~]# xe vm-list uuid= +... +uuid ( RO) : 740e9485-8e1a-da89-2561-d75bcede3fc9 + name-label ( RW): VM with PCI Passthrough + power-state ( RO): halted +``` + +### 3. Confirm the VM has the list of attached PCI devices +Run +`xe vm-param-list uuid= | grep -i other-config` +``` +[root@xen ~]# xe vm-param-list uuid=740e9485-8e1a-da89-2561-d75bcede3fc9 | grep -i other-config +other-config (MRW): pci: 0/0000:04:01.0; base_template_name: Other install media; import_task: OpaqueRef:46e1e0db-f7fa-4131-8539-0f82f0336d7c; mac_seed: ebd99219-9881-b965-610e-342087ec6606; install-methods: cdrom +``` +> If you have multiple devices, note down their PCI addresses as you may have to add them back after removing your device. + +### 4. Remove the PCI device from the VM config using `vm-param-remove` +Run `xe vm-param-remove param-name=other-config param-key=pci uuid=`. +``` +[root@xen ~]# xe vm-param-remove param-name=other-config param-key=pci uuid=740e9485-8e1a-da89-2561-d75bcede3fc9 +``` + +### 5. Confirm the PCI device is no longer attached +Run `xe vm-param-list uuid= | grep -i other-config` again. +``` +[root@xen ~]# xe vm-param-list uuid=740e9485-8e1a-da89-2561-d75bcede3fc9 | grep -i other-config +other-config (MRW): base_template_name: Other install media; import_task: OpaqueRef:46e1e0db-f7fa-4131-8539-0f82f0336d7c; mac_seed: ebd99219-9881-b965-610e-342087ec6606; install-methods: cdrom +``` +> At this point, if you had multiple devices and they were removed, you should add them back again using the `vm-param-set` command`. + +You can now attach the device to another VM or add it back to the `dom0` control domain. + +### 6. Reattach PCI device to control domain +Run `/opt/xensource/libexec/xen-cmdline --delete-dom0 xen-pciback.hide` + ## GPU Passthrough To passthrough a complete graphics card to a VM (not virtualize it into multiple virtual vGPUs, which is different, see the vGPU section below), just follow the regular PCI passthrough instructions, no special steps are needed. Most Nvidia and AMD video cards should work without issue. From 9715cd6fae13fa2cbff5a04267a252ad5a2b8c2c Mon Sep 17 00:00:00 2001 From: Bryan Vaz Date: Fri, 26 Nov 2021 12:11:18 -0500 Subject: [PATCH 2/3] docs: update PCI unhide instructions with reboot instructions Signed-off-by: Bryan Vaz --- docs/compute.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/docs/compute.md b/docs/compute.md index 2a057c97..97bb1a7e 100644 --- a/docs/compute.md +++ b/docs/compute.md @@ -148,8 +148,24 @@ other-config (MRW): base_template_name: Other install media; import_task: Opaque You can now attach the device to another VM or add it back to the `dom0` control domain. -### 6. Reattach PCI device to control domain -Run `/opt/xensource/libexec/xen-cmdline --delete-dom0 xen-pciback.hide` +### 6. Reattach PCI device to control domain (if desired) +Run `/opt/xensource/libexec/xen-cmdline --delete-dom0 xen-pciback.hide`. + +_Note: this command removes all PCI devices hidden from the control domain (`dom0`). If you have multiple devices, you can either: (a) add them back if they are inadvertently removed; or (b) use the previous `--set-dom0` command, to update the list of hidden PCI devices, excluding the device address that you want to add back to the control domain._ + +:::tip +You can use `cat /boot/efi/EFI/xenserver/grub.cfg` see a list of devices currently hidden under during a normal boot (`menuentry 'XCP-ng'`). The grub location will vary depending on how you setup your bootloader._ +::: + +:::tip +This is a kernel parameter, and thus is not retained when you upgrade an XCP-ng host [using the installation ISO](upgrade.md#upgrade-via-installation-iso-recommended). Remember to re-do this step after the upgrade, if you still have some PCI devices you want to hide from the control domain. +::: + +### 7. Reboot the XCP-ng host + +`[root@xen ~]# reboot`. +_Note: Reboot is only necessary if you changed the kernel parameters, i.e. by adding or hiding devices from the control domain._ + ## GPU Passthrough To passthrough a complete graphics card to a VM (not virtualize it into multiple virtual vGPUs, which is different, see the vGPU section below), just follow the regular PCI passthrough instructions, no special steps are needed. Most Nvidia and AMD video cards should work without issue. From baaa0e3fbd0f9999fd52e61c6e9a311f3eb25843 Mon Sep 17 00:00:00 2001 From: Bryan Vaz Date: Fri, 26 Nov 2021 12:13:29 -0500 Subject: [PATCH 3/3] fix: removed erroneously included trailing underscore Signed-off-by: Bryan Vaz --- docs/compute.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/compute.md b/docs/compute.md index 97bb1a7e..94ee91c6 100644 --- a/docs/compute.md +++ b/docs/compute.md @@ -154,7 +154,7 @@ Run `/opt/xensource/libexec/xen-cmdline --delete-dom0 xen-pciback.hide`. _Note: this command removes all PCI devices hidden from the control domain (`dom0`). If you have multiple devices, you can either: (a) add them back if they are inadvertently removed; or (b) use the previous `--set-dom0` command, to update the list of hidden PCI devices, excluding the device address that you want to add back to the control domain._ :::tip -You can use `cat /boot/efi/EFI/xenserver/grub.cfg` see a list of devices currently hidden under during a normal boot (`menuentry 'XCP-ng'`). The grub location will vary depending on how you setup your bootloader._ +You can use `cat /boot/efi/EFI/xenserver/grub.cfg` see a list of devices currently hidden under during a normal boot (`menuentry 'XCP-ng'`). The grub location will vary depending on how you setup your bootloader. ::: :::tip