Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use direct I/O for loop devices #7332

Open
DemiMarie opened this issue Mar 9, 2022 · 16 comments · Fixed by QubesOS/qubes-vmm-xen#127
Open

Use direct I/O for loop devices #7332

DemiMarie opened this issue Mar 9, 2022 · 16 comments · Fixed by QubesOS/qubes-vmm-xen#127
Labels
C: storage P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. r4.1-bookworm-stable r4.1-bullseye-stable r4.1-buster-stable r4.1-centos-stream8-stable r4.1-dom0-stable r4.2-host-cur-test r4.2-vm-bookworm-cur-test r4.2-vm-bullseye-cur-test

Comments

@DemiMarie
Copy link

How to file a helpful issue

The problem you're addressing (if any)

Qubes OS currently does not use direct I/O for the pools that use loop devices This leads to double-caching and poor I/O performance. I believe it is the cause of the anomalies in this benchmark.

The solution you'd like

Use O_DIRECT for loop devices.

The value to a user, and who that user might be

Users of the reflink pool will experience improved and more consistent performance.
@DemiMarie DemiMarie added T: enhancement P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. labels Mar 9, 2022
@andrewdavidwong andrewdavidwong added this to the Release TBD milestone Mar 10, 2022
@AlxHnr
Copy link

AlxHnr commented Mar 10, 2022

Could this be causing #5086?

@DemiMarie
Copy link
Author

Possibly

DemiMarie added a commit to DemiMarie/qubes-vmm-xen that referenced this issue Mar 11, 2022
@DemiMarie
Use direct I/O for loop devices
e879ec2 
This is a huge performance improvement for two reasons:

1. It uses the filesystem’s asynchronous I/O support, rather than using
   synchronous I/O.
2. It bypasses the page cache, removing a redundant layer of caching and
   associated overhead.

I also took the opportunity to rip out some cruft related to old losetup
versions, which Qubes OS doesn't need to support anymore.

Fixes QubesOS/qubes-issues#7332.
@DemiMarie DemiMarie mentioned this issue Mar 11, 2022
Merged
DemiMarie added a commit to DemiMarie/qubes-vmm-xen that referenced this issue Mar 11, 2022
@DemiMarie
Use direct I/O for loop devices
2044fa8 
This is a huge performance improvement for two reasons:

1. It uses the filesystem’s asynchronous I/O support, rather than using
   synchronous I/O.
2. It bypasses the page cache, removing a redundant layer of caching and
   associated overhead.

I also took the opportunity to rip out some cruft related to old losetup
versions, which Qubes OS doesn't need to support anymore.

Fixes QubesOS/qubes-issues#7332.
DemiMarie added a commit to DemiMarie/qubes-vmm-xen that referenced this issue Apr 14, 2022
@DemiMarie
Use direct I/O for loop devices
cced625 
This is a huge performance improvement for two reasons:

1. It uses the filesystem’s asynchronous I/O support, rather than using
   synchronous I/O.
2. It bypasses the page cache, removing a redundant layer of caching and
   associated overhead.

Fixes QubesOS/qubes-issues#7332.
DemiMarie added a commit to DemiMarie/qubes-vmm-xen that referenced this issue May 12, 2022
@DemiMarie
Use direct I/O for loop devices
e4df455 
This is a huge performance improvement for two reasons:

1. It uses the filesystem’s asynchronous I/O support, rather than using
   synchronous I/O.
2. It bypasses the page cache, removing a redundant layer of caching and
   associated overhead.

Fixes QubesOS/qubes-issues#7332.
DemiMarie added a commit to DemiMarie/qubes-vmm-xen that referenced this issue May 12, 2022
@DemiMarie
Use direct I/O for loop devices
3f25f85 
This is a huge performance improvement for two reasons:

1. It uses the filesystem’s asynchronous I/O support, rather than using
   synchronous I/O.
2. It bypasses the page cache, removing a redundant layer of caching and
   associated overhead.

Fixes QubesOS/qubes-issues#7332.
DemiMarie added a commit to DemiMarie/qubes-vmm-xen that referenced this issue May 12, 2022
@DemiMarie
Use direct I/O for loop devices
bfe5ffd 
This is a huge performance improvement for two reasons:

1. It uses the filesystem’s asynchronous I/O support, rather than using
   synchronous I/O.
2. It bypasses the page cache, removing a redundant layer of caching and
   associated overhead.

Fixes QubesOS/qubes-issues#7332.
DemiMarie added a commit to DemiMarie/qubes-vmm-xen that referenced this issue Aug 17, 2022
@DemiMarie
Use direct I/O for loop devices
23d5b18 
This is a huge performance improvement for two reasons:

1. It uses the filesystem’s asynchronous I/O support, rather than using
   synchronous I/O.
2. It bypasses the page cache, removing a redundant layer of caching and
   associated overhead.

Fixes QubesOS/qubes-issues#7332.
@qubesos-bot qubesos-bot mentioned this issue Oct 16, 2022
Closed
@qubesos-bot
Copy link

Automated announcement from builder-github

The component vmm-xen (including package python3-xen-4.14.5-9.fc32) has been pushed to the r4.1 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

@qubesos-bot qubesos-bot mentioned this issue Oct 16, 2022
Closed
@qubesos-bot
Copy link

Automated announcement from builder-github

The package vmm-xen has been pushed to the r4.1 testing repository for the CentOS centos-stream8 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.1-current-testing

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package xen_4.14.5-9 has been pushed to the r4.1 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing buster-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

@andrewdavidwong andrewdavidwong modified the milestones: Release TBD, Release 4.1 updates Oct 16, 2022
@qubesos-bot qubesos-bot mentioned this issue Oct 20, 2022
Closed
@qubesos-bot
Copy link

Automated announcement from builder-github

The package vmm-xen has been pushed to the r4.2 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing bookworm-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

@DemiMarie
Copy link
Author

Reopening as QubesOS/qubes-vmm-xen#127 had to be reverted (#7828).

@DemiMarie DemiMarie reopened this Oct 20, 2022
@marmarek
Copy link
Member

For the record, the next iteration of this patch (if it comes), I'm going to block on actually proving it helps with performance. We see it's more risky change than it seems to be, so lets be sure it's actually worth doing.

@DemiMarie
Copy link
Author

I agree. Someone on the forum seems to have a benchmark setup that could be used.

@rustybird
Copy link

rustybird commented Oct 21, 2022 •
edited
Loading

If someone wants to play around with this on file(-reflink) without having to build a bleeding edge util-linux losetup (v2.38.1 includes another relevant patch that avoids a kernel race only fixed in kernel-latest), here's a small Python piccolosetup that's good enough to do one proper LOOP_CONFIGURE ioctl from a patched /etc/xen/scripts/block.

Usage: piccolosetup [--sector-size=BYTES] [--direct-io=on] [-r] LOOPDEV FILE

@andrewdavidwong andrewdavidwong modified the milestones: Release 4.1 updates, Release TBD Oct 21, 2022
@brendanhoar
Copy link

Looks like #7828 now includes a clarification that the failure of the first iteration of the patch (on 4Kn drives) applied to LVM thick, not LVM thin. For the record.

B

@rustybird
Copy link

rustybird commented Oct 22, 2022 •
edited
Loading

In the LVM Thin installation layout it would still have affected the legacy 'file' driver varlibqubes pool on the ext4 dom0 filesystem. (But the lvm_thin driver vm pool would usually be broken anyway on a 4Kn drive.)

@rustybird rustybird mentioned this issue Oct 25, 2022
Closed
@qubesos-bot
Copy link

Automated announcement from builder-github

The component vmm-xen (including package python3-xen-4.14.5-10.fc32) has been pushed to the r4.1 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package vmm-xen has been pushed to the r4.1 stable repository for the CentOS centos-stream8 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package xen_4.14.5-10+deb10u1 has been pushed to the r4.1 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

@andrewdavidwong andrewdavidwong modified the milestones: Release TBD, Release 4.1 updates Oct 28, 2022
@andrewdavidwong andrewdavidwong removed this from the Release 4.1 updates milestone Aug 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C: storage P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. r4.1-bookworm-stable r4.1-bullseye-stable r4.1-buster-stable r4.1-centos-stream8-stable r4.1-dom0-stable r4.2-host-cur-test r4.2-vm-bookworm-cur-test r4.2-vm-bullseye-cur-test
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use direct I/O for loop devices DemiMarie/qubes-vmm-xen
7 participants
@marmarek @DemiMarie @rustybird @AlxHnr @brendanhoar @andrewdavidwong @qubesos-bot