NVMe RAID support

[isudos]

Hi!
Does it suppose to work with NVMe RAID?
My attempt to install it failed.
If doesn't - are there any chances to add such support?

[Hubcapp]

This is a duplicate of #7. TL;DR is that the author of this software is not able to support NVMe RAID at this time.

I can also personally confirm that NVMe RAID is not supported. :-(

[thopiekar]

Finally, bought myself a NVMe drive. My drive is also not managed by RAIDxpert, but have the same on Windows, too. (Dual-boot) Therefore, I wouldn't blame it on AMD and their driver, since Windows behaves the same.
So in my opinion the issue comes from somewhere below OS layer - BIOS level.
I'll try to get in contact with ASUS and ask there for help based on my Windows installation (which should be better supported). Hope it will resolve the issue on my Linux installation, too.

[kubuntugamer]

To have the NVMe drives managed by RAIDXpert, there is a setting somewhere in UEFI to enable NVMe RAID(it isn't enabled by default). The system has to boot via UEFI with no Legacy BIOS options enabled. I am currently not at my x399 system to go thru the options and look for the specific settings, but i do remember having to enable RAID support for NVMe(Asrock Phantom Gaming 6 mobo).
However, even after having created arrays with NVMe drives, they still won't show any sort of array in linux. linux just sees them as 2 seperate block devices.

[Hubcapp]

Yea, there's an option on my asus x570 board specifically to enable NVMe RAID. Then it's still fake RAID handled in software. You'll see two devices during Windows installation until you load AMD's fake raid driver, (actually using the drivers included on the CD like it's 2004), then it's just one device.

[thopiekar]

Can confirm your problem here. I had to upgrade my Windows drivers and my NVMe appeared as "Array 6". However, going back to Linux and blacklisting ahci and nvme still didn't get me the drive. "Array 6" was still listed, but marked as offline.

[thopiekar]

I'm going to write an support ticket tomorrow. I see nothing in the driver that mentions NVMe support.
On the other hand there are documents already, which mention NVMe support on Linux.
So I have the following theories:

• AMD missed to publish the driver
• AMD is testing the driver with partners or internally still

[thopiekar]

But it is promising.. Just found a PDF with the latest instructions.

There must be a driver 🤔

[kubuntugamer]

But it is promising.. Just found a PDF with the latest instructions.

There must be a driver 🤔

this blacklisting of the linux nvme driver is information i was not aware of in my prior attempts. later today i will make a backup of my $HOME directory and experiment with this boot option and the live kubuntu installer. i will update this post when i know something.

[thopiekar]

Yes, this is new and needs to be done for the new driver I think. You will see no difference now.
When looking at the sources you will see no nvme implementation. ;)

[thopiekar]

Some news: First I sent a support ticket just some days before. I was a bit too busy recently.
Anyway, got a reply from the support telling me that they don't provide updated drivers.

However, in the meantime I made some research and found binary drivers distributed by Lenovo for their workstations for Redhat Linux.
I'm going to confront them with this fact and question them whether they think there are second class clients, or what?!

Alternatively we can experiment using the same version as Redhat Linux and try to load the blob, but hope this won't be needed to waste time on.

[thopiekar]

@ducphuc Took a look over your repo. Can you tell me where you've got the sources from? I see some indications that NVMe support is included there. Can you confirm this?

Others: The driver is 9.3.0 😱

[ducphuc]

I got the sources from the RAIDXpert2 application install for RHEL, Ubuntu, and Debian which came with my Lenovo ThinkStation P620. It broke with the changes in Linux kernel 5.6 so I patched it with fixes in this repo.

Yes, I can confirm that NVMe support is included. I have my system booting from a 4 x Samsung 980 PRO PCIe 4.0 NVMe SSD RAID0 array using a PCIe x16 Gen 4 AIC. I get over 21GB/s sequential reads and over 18GB/s sequential writes.

From the AMD RAIDXpert2 User Guide:

RC-8.1 and RC8.2 branches support only SATA. RC-9.1 and above release branches support SATA, M.2 NVMe SSD and HHHL NVMe SSD

[kubuntugamer]

does this mean that support will be coming in the near future? will i finally be able to use the hardware i paid for?

edit: just in case: the drivers and sources can be found here
https://pcsupport.lenovo.com/us/en/products/workstations/thinkstation-p-series-workstations/thinkstation-p620/downloads/driver-list/component?name=RAID%2FSAS%20HBA%20Controllers,%20Backplanes,%20Storage%20Expanders%2FSwitches

i attempted to do a patch and compile myself using the patch file from Martin Karl Weber's github repo, but it fails to patch 3 files and fails to compile

[thopiekar]

Will verify the files tomorrow and test the driver myself. Great findings! I was looking for the sources but only saw the binaries so far.
Many thanks!

PS: You didn't merge the code correctly. Your overwrote Martin's patches and reapplied them later. Will prepare something cleaner once I got it working here.

[kubuntugamer]

Will verify the files tomorrow and test the driver myself. Great findings! I was looking for the sources but only saw the binaries so far.
Many thanks!

PS: You didn't merge the code correctly. Your overwrote Martin's patches and reapplied them later. Will prepare something cleaner once I got it working here.

That sounds AWESOME!!!

edit: do u have a staging ppa available that i can also help test before you push it to the stable ppa?

[ducphuc]

@thopiekar If you referring to my fork with not merging correctly then you're right. The last commit wasn't supposed to be pushed so I reverted it. I was re-applying the unofficial patches(Martin's patches) to see what the differences were. Sorry about the confusion.

What I did was just copy the official sources over the sources in the repo then applied the fix to get Linux Kernel 5.6 working. What I am trying to do is get DKMS working and using what you have in this repo to get started. I am not planning to submit a PR nor planning to keep my repo around once you saw the official sources were available. I will just be using the OEM vendor supplied packages. I am just testing some unsupported distros. The problems I am facing is from the platform being just released and the AMD support for Linux is rough around the edges still. I suspect things will be different now that AMD is trying to get into the workstation market.

[thopiekar]

Alright. From what I saw in the latest sources, it seems that AMD took over the source code. At least there are copyright tags stating that AMD owns the code beginning from 2020. Hope they are going to support us better now 🤞

[ducphuc]

Support will come from the OEM partners at least. AMD RAID is currently certified and supported on a Lenovo ThinkStation P620 with Ubuntu 20.04 or RHEL 8.2. This workstation is a great product and I highly recommend it. I'll be submitting a bug report that AMD RAID needs to be fixed for Linux 5.6 kernels with them.

Lenovo has made some big announcements regarding both Linux and AMD(ThreadRipper Pro workstations) these last few months. The work they are doing to bring support to Linux is amazing. For example, they are requiring their component suppliers, such as the fingerprint readers, provide Linux drivers or they will use one that will. My next laptop will definitely be a Lenovo ThinkPad with Linux preinstalled.

• Lenovo Brings Linux® Certification to ThinkPad and ThinkStation Workstation Portfolio, Easing Deployment for Developers & Data Scientists
• Lenovo Launches Linux-Ready ThinkPad and ThinkStation PCs Preinstalled with Ubuntu
• Lenovo ThinkStation P620: World’s First AMD Ryzen™ Threadripper™ PRO Workstation

[thopiekar]

After migrating the old to the new code and getting the driver running, I got messages like "license level to low to run Array X". However, I have/had no complaints about missing licenses or permissions on Windows.

After about a week of fixing and a messed up hard drive (yeah good timing -.-' ), I gave up with AMD RaidXpress.
The Windows and Linux drivers are not fitting at all. If I set in my BIOS NVMe mode to "normal" (I would expect it to be not managed by RaidXpress) it won't appear on Windows at all, but on Linux I can access it from 8.1.0 when not blacklisting nvme.
When using 8.1.0 on Linux and 9.3.0 on Windows plus I lost my hard drive with non-critical data for Linux (trash store for building Android firmware or whatever else) I had a "biting" effect. I not only lost the specific drive but also the relevant data. So I lost again days to use recovery tools to find my partitions on the drive(s).

I just don't have time for further experiments and crossed fingers. Regarding the licensing errors reported by the driver I doubt the driver is ready for everyone now. I don't trust this software as it is now.

[ducphuc]

That's not a good experience. I feel your pain. It was a struggle for me to get things working like I wanted too. It seems the firmware version needs be 9.3.x as well to support NVMe RAID. My AMR RAID firmware is 9.3. I would check with your motherboard vendor for updated firmware. The firmware writes metadata onto the drives. The firmware will also store information about the drives in it's memory. The manual states that the drive will not be visible to the operating systems when deleted from an array. I experienced this myself and is by design. You need to add it an array. Clearing the metadata from the drive and/or from firmware would get it back. There are some quirks like this and others like needing to delete a drive from an array(the automatic legacy one created for it) before you can add it to an array is shown in AMD RAID reviews and many forums. It is not the easiest thing to work with at all. But once you get it working, 21 GB/s is very nice to have.

Also the manual states specifically that dual booting is not supported. It jacks up my system too. Two separate legacy RAID controllers are created for a drive. It would flip flop which one is working and visible between Windows and Linux.

[MarkLTZ]

Hello folks,

any news about the NVMe AMD RAID support? I have the same issue discussed here and I would like to fix it on my Ubuntu 20.10

Is a fix on the roadmap or does an alternative solution exist?

Thanks,
Mark

[Fuzzo]

I also found a driver raid_linux_driver_930_00276.zip named that says:

[kubuntugamer]

I also found a driver raid_linux_driver_930_00276.zip named that says:

can you inform us of where you found this driver?...the problem we've had so far with the one provided by lenovo is that it is only licensed for use on lenovo linux systems, and nvme raid support was not enabled by the lenovo drivers on non-lenovo hardware(for example: i had no success with my Phantom Gaming 6 motherboard).

[MarkLTZ]

May be I have found a version of AMD drivers that support NVMe RAID. In this case, on next days I will create a PR for this repository, but please @Fuzzo give to us the link of driver you found because if it's newer than mine I will use it.

[Fuzzo]

The drivers can be downloaded at this page:
Direct download link is:

Even if they are present in the x370 page, the RAID subsystem should be the same also for other chipsets.

EDITED DUE TO WRONG FILE: i uploaded the right one in comments below, sorry

[MarkLTZ]

perfect! thanks!

[Fuzzo]

@Fuzzo do you have and X570 chipset, right?

if yes, can you run this command for me with and without nvme driver blacklisted?

lspci -nn | grep 1022

As shown in this article PCI 1022:b000 (AMD RAID Bottom Device) should be supported, but on my X470 I'm not able to see it.

00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
03:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
03:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
06:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
06:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
06:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
07:00.0 RAID bus controller [0104]: Advanced Micro Devices, Inc. [AMD] Device [1022:7916] (rev 51)
08:00.0 RAID bus controller [0104]: Advanced Micro Devices, Inc. [AMD] Device [1022:7916] (rev 51)
0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
0b:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
0b:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]

The output is same (by diff) with and without nvme, ahci and rcraid modules loaded.

Looks like the important lines are:

07:00.0 RAID bus controller [0104]: Advanced Micro Devices, Inc. [AMD] Device [1022:7916] (rev 51)
08:00.0 RAID bus controller [0104]: Advanced Micro Devices, Inc. [AMD] Device [1022:7916] (rev 51)

even if don't match 1022:b000 you're looking for.

I post also the RAIDXpert (for Windows) controllers view:

We can see that 7916 is the device_id of my controller (instead of b000).

Hope this help.

I wrote a message to AMD support....

Me too... and also to Asus.

Edit
These are screenshot of the devices from Windows (the storport driver name is rcraid.sys):

The bottom device is the AData disk...

[MarkLTZ]

@Fuzzo wow! What your chipeset is? X370? X470? X570?

In the BIOS do you have a settings called "NVMe RAID Mode" or something similar?

[Fuzzo]

My chipset Is X570. Raid is working only on Windows, from where i took the screenshots. Il gio 20 gen 2022, 20:20 Mark LTZ ***@***.***> ha scritto:

…

@Fuzzo <https://github.com/Fuzzo> wow! What your chipeset is? X370? X470? X570? Which driver do you used for having it working? — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPAGMNFEG3DVGVPDI4PN53UXBN6ZANCNFSM4NSAP37A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[MarkLTZ]

@Fuzzo

In the BIOS do you have a settings called "NVMe RAID" enabled?

[Fuzzo]

Yes Il gio 20 gen 2022, 21:14 Mark LTZ ***@***.***> ha scritto:

…

@Fuzzo <https://github.com/Fuzzo> In the BIOS do you have a settings called "NVMe RAID" enabled? — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPAGMILIJCCCNQH6ZDI6XLUXBUJ3ANCNFSM4NSAP37A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[MarkLTZ]

@Fuzzo ok grazie :) solo ora mi accorgo che siamo entrambi italiani :)

[Fuzzo]

Ciao @MarkLTZ, piacere! :)

Asus replied to my ticket: "Linux is not supported", still waiting for AMD...
I'm available for any other testing you may need...

[MarkLTZ]

@Fuzzo Could you please tell me the complete model name of your Asus motherboard?

[Fuzzo]

Asus TUF GAMING X570-PLUS

[MarkLTZ]

@Fuzzo

As reported from linux-hardware.org:

Are you really sure to have NVMe raid support turned on?

Check also if "SATA Express PCIE Storage RAID Support" is enabled.

If not, please enable both of them and check if on Linux you are able so see PCI device 1022:b000

You can use sudo hw-probe -all command for more details. Devices are collected into /root/HW_PROBE/LATEST/hw.info/devices file.

BTW: AMD replied to my service request:

As of now their is no support for RAID for X570 and X470 chipsets for Linux OS. Thank you for your valuable feedback,
However I will forward your feedback onto relevant team for further consideration.
Thanks for contacting AMD

[MarkLTZ]

[ducphuc]

The AMD RAID driver and RAIDXpert2 Utility for both RHEL and Ubuntu were posted on the AMD driver download page on 12/8/2021 for the WRX80 motherboard. The RHEL driver version is 9.3.0.00283 and the Ubuntu driver version is 9.3.0.00276. The motherboard UEFI driver version needs to be the same major and minor version. The following are listed as supported in Linux:

• 3rd Gen AMD Ryzen™ Threadripper™ Processors
• 2nd Gen AMD Ryzen™ Threadripper™ Processor
• AMD TRX40 Chipset
• AMD WRX80 Chipset

https://www.amd.com/en/support/chipsets/amd-socket-swrx8/wrx80.
There are documents on how to install, compile, and configure the drivers for each distro.

[MarkLTZ]

@ducphuc
We know it.
We are working to make it compatible with X470 / X570 chipset.

[MarkLTZ]

The AMD RAID driver and RAIDXpert2 Utility for both RHEL and Ubuntu were posted on the AMD driver download page on 12/8/2021 for the WRX80 motherboard. The RHEL driver version is 9.3.0.00283 and the Ubuntu driver version is 9.3.0.00276. The motherboard UEFI driver version needs to be the same major and minor version. The following are listed as supported in Linux:

• 3rd Gen AMD Ryzen™ Threadripper™ Processors
• 2nd Gen AMD Ryzen™ Threadripper™ Processor
• AMD TRX40 Chipset
• AMD WRX80 Chipset

https://www.amd.com/en/support/chipsets/amd-socket-swrx8/wrx80. There are documents on how to install, compile, and configure the drivers for each distro.

Did you tried it? Do you have a TRX40 or WRX80 Chipset and RAID on NVMe ?

[ducphuc]

OK. It didn't look like you were using the latest version of the RAIDXpert2 Utility. However, it looks like you might have a lower minor version of the UEFI driver based on the date of you BIOS. Yes, I have a Lenovo ThinkStation P620 which has a WRX80 chipset. I was running NVMe RAID but stopped when there were performance issues with the Samsung 980 Pro firmware. I plan to go back to using NVMe RAID but running Windows when my server arrives in the next weeks. You should seriously consider not running NVMe RAID with Linux. Linux can boot from software RAID. Windows cannot so it is required for that platform. Check the benchmarks people have done with this. Wendell from Level1Techs has a YouTube video on this.

[MarkLTZ]

OK. It didn't look like you were using the latest version of the RAIDXpert2 Utility. However, it looks like you might have a lower minor version of the UEFI driver based on the date of you BIOS. Yes, I have a Lenovo ThinkStation P620 which has a WRX80 chipset. I was running NVMe RAID but stopped when there were performance issues with the Samsung 980 Pro firmware. I plan to go back to using NVMe RAID but running Windows when my server arrives in the next weeks. You should seriously consider not running NVMe RAID with Linux. Linux can boot from software RAID. Windows cannot so it is required for that platform. Check the benchmarks people have done with this. Wendell from Level1Techs has a YouTube video on this.

I know there are performance issues and on Linux I can use software RAID, but I want make the driver to support also NVMe raid if it's configured on BIOS.

For example on my computer:

• Array 1 (NVMe Raid 1) where Windows 10 is installed
• Array 2 (SDD Raid 1) where Linux is installed

Now, the problems with this configuration due the lack of full raid support on the AMD driver are:

1. Dual boot menu from GRUB is not possible and I need each time to press F10 and select from which UEFI disk I want to boot (except for the default configured on BIOS).
2. Accessing on Windows partitions from Linux is not possible.

[ducphuc]

Booting is only supported from the first array. Does your UEFI HII for the AMD RAID show NVMe RAID controllers created and the arrays? It would be your UEFI firmware creating the 1022:b000 controllers. Make sure both the NVMe and AHCI drivers are blacklisted in Linux. I dual booted but grub and the Linux boot partition was on the first array with Windows. The dual boot setup was not reliable. It would misbehave with drives that were not in the array. It created extra AMD NVMe controllers.

[MarkLTZ]

Booting is only supported from the first array. Does your UEFI HII for the AMD RAID show NVMe RAID controllers created and the arrays? It would be your UEFI firmware creating the 1022:b000 controllers. Make sure both the NVMe and AHCI drivers are blacklisted in Linux. I dual booted but grub and the Linux boot partition was on the first array with Windows. The dual boot setup was not reliable. It would misbehave with drives that were not in the array. It created extra AMD NVMe controllers.

Boot works on both array, but on bios UEFI boot was set on Windows (array1), so if I want to boot on Linux I need to press F10 and select "Ubuntu (array2) each time.

On Windows everything works fine.

On Linux rcraid works almost fine (NVMe and AHCI drivers are blacklisted) except the os is not able to see array1 (NVMe)

As I said before, we are working to make it possible on motherboard having X470 or X570 chipset.

You are lucky to have WRX80 chipset where everything is working!

[ducphuc]

There seems to be a disconnect. Booting from both arrays does not look like it is working correctly. Your previous post indicates that the AMD RAID array1 is not being enumerated by the UEFI firmware and driver when you boot from array2. I suspect this is because you selected the second array,"array2", to boot. The statement that booting is only supported from the first array is from the AMD RAID documentation. The enumeration of both AMD RAID array1 and array2 works when you boot from array1.

I suggest you boot Linux from the first AMD RAID array, array1. I believe using a boot loader on the EFI partition on array1 to boot a Linux from array2 was sufficient. You can try chain loading GRUB from the Windows bootmgr EFI application or install grub to the EFI partition on arrary1. I used the rEFInd boot manager as well. If you boot into Linux from arrary1, enumeration of arrary1 and array2 would need to occur since you are booting from them.

Hopefully, your chipset will support this. You didn't answer my question if AMD RAID HII shows the AMD NVMe RAID controller(s). That would be a good indicator if your UEFI AMD RAID driver firmware is correctly enumerating the NVMe drives.

[MarkLTZ]

@ducphuc

From BIOS all arrays are OK

On boot pressing F10 key UEFI menu shows me both arrays and I am able to boot from array1 (windows) or array2 (linux)

On Linux the array 1 is visible but not managed from the current AMD driver that shows it as OFFLINE due missing disks (see pics on this thread)

Because I'm sure this is a 100% issue in the AMD driver, what I'm doing is to modify it for being compatible also on X470 / X570 chipsets,

I don't need workarounds or alternatives. I need only an help from some kernel developer to investigate and modify the original AMD driver for including NVMe RAID support also on these chipsets.

Regarding the UEFI firmware I wrote to MSI support asking to them to investigate and upgrade their bios.

[Fuzzo]

@Fuzzo

As reported from linux-hardware.org:

Are you really sure to have NVMe raid support turned on?

Check also if "SATA Express PCIE Storage RAID Support" is enabled.

If not, please enable both of them and check if on Linux you are able so see PCI device 1022:b000

You can use sudo hw-probe -all command for more details. Devices are collected into /root/HW_PROBE/LATEST/hw.info/devices file.

BTW: AMD replied to my service request:

As of now their is no support for RAID for X570 and X470 chipsets for Linux OS. Thank you for your valuable feedback,
However I will forward your feedback onto relevant team for further consideration.
Thanks for contacting AMD

My settings in BIOS for SATA:

For controller:

For array:

For physical disks:

I've not any SATA device connected to my system. Seems the device you're looking for, the b000, is a kind of RAID/NVME to SATA bridge... or am I wrong?

[MarkLTZ]

I've not any SATA device connected to my system. Seems the device you're looking for, the b000, is a kind of RAID/NVME to >SATA bridge... or am I wrong?

Yes. Anyway you have 9.30 UEFI firmware :(

Just last 2 checks:

• it the BIOS latest available?
• what is the output of lspci -nn with NVMe RAID Mode disabled?

[MarkLTZ]

Rediscovery information

Expected:

03:00.0 Non-Volatile memory controller [0108]: Advanced Micro Devices, Inc. [AMD] Device [1022:b000] (prog-if 02 [NVM Express])
Subsystem: Samsung Electronics Co Ltd Device [144d:a801]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 33
NUMA node: 0
Region 0: Memory at fca00000 (64-bit, non-prefetchable) [size=16K]
Capabilities:
Kernel driver in use: nvme
Kernel modules: nvme

Found:

01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804] (prog-if 02 [NVM Express])
Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a801]
Flags: bus master, fast devsel, latency 0, IRQ 255
Memory at f7a00000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [158] Power Budgeting <?>
Capabilities: [168] Secondary PCI Express
Capabilities: [188] Latency Tolerance Reporting
Capabilities: [190] L1 PM Substates
Kernel modules: nvme

[MarkLTZ]

Updates of this driver for NVMe raid support is not possible due missing function into rcblob.x86_64. NVMe support is present in 9.3.0 version of this file but:

1. it does not work at all on X470 / X570 chipsets:

[ 2.845683] Array 2 not allowed by current license level.
[ 2.845704] Array 2 not allowed by current license level.
[ 2.847149] Array 1 has gone offline.

2. it does not bring online LUNs:

[ 2.876110] <4>rc_msg_process_srb: Bus/Target/Lun 0/0/0 OFFLINE
[ 2.876116] <4>rc_msg_process_srb: Bus/Target/Lun 0/1/0 OFFLINE
[ 2.876758] <4>rc_msg_process_srb: Bus/Target/Lun 0/0/0 OFFLINE
[ 2.876859] <4>rc_msg_process_srb: Bus/Target/Lun 0/1/0 OFFLINE

I hope AMD will release an official release for solving our issue.

[kubuntugamer]

just out of curiosity, is it possible to spoof the hardware IDs needed by these drivers? or is that legally questionable?

[MarkLTZ]

@kubuntugamer

From original 9.3.0 AMD driver, file rc_init.c:

    {
            .vendor = RC_PD_VID_AMD,
            .device = AMD_NVME_DID,
            .subvendor = PCI_ANY_ID,
            .subdevice = PCI_ANY_ID,
            .class = (PCI_CLASS_NVME_STORAGE_EXPRESS),
            .class_mask = 0xffffff,
            .driver_data = (unsigned long)&rc_nvme_version
    },

Defined inside rc_pci_ids.h:

RC_PD_VID_AMD = 0x1022
AMD_NVME_DID = 0xB000

Anyway using vendor = 0x144d and device = 0xa804 the driver still does not work for NVMe because most of the code that make it works is inside the rcblob.x86_64 file that is not provided as source code from AMD but as ELF64, so it's not possible to update it.

Original AMD driver 9.3.0 for Linux supports:

• 3rd Gen AMD Ryzen™ Threadripper™ Processors
• 2nd Gen AMD Ryzen™ Threadripper™ Processors
• AMD TRX40 Chipset
• AMD WRX80 Chipset
• Supported RAID Levels RAID 0,1,10 for both NVMe and SATA RAID on the above listed AMD products.

[kubuntugamer]

@kubuntugamer

From original 9.3.0 AMD driver, file rc_init.c:

    {
            .vendor = RC_PD_VID_AMD,
            .device = AMD_NVME_DID,
            .subvendor = PCI_ANY_ID,
            .subdevice = PCI_ANY_ID,
            .class = (PCI_CLASS_NVME_STORAGE_EXPRESS),
            .class_mask = 0xffffff,
            .driver_data = (unsigned long)&rc_nvme_version
    },

Defined inside rc_pci_ids.h:

RC_PD_VID_AMD = 0x1022 AMD_NVME_DID = 0xB000

Anyway using vendor = 0x144d and device = 0xa804 the driver still does not work for NVMe because most of the code that make it works is inside the rcblob.x86_64 file that is not provided as source code from AMD but as ELF64, so it's not possible to update it.

Original AMD driver 9.3.0 for Linux supports:

• 3rd Gen AMD Ryzen™ Threadripper™ Processors
• 2nd Gen AMD Ryzen™ Threadripper™ Processors
• AMD TRX40 Chipset
• AMD WRX80 Chipset
• Supported RAID Levels RAID 0,1,10 for both NVMe and SATA RAID on the above listed AMD products.

gotcha...thanx for the reply and explanation.

[Fuzzo]

• what is the output of lspci -nn with NVMe RAID Mode disabled?

lspcinn_nvmeraiddisabled.txt

[developerPaul]

Sorry guys. Raid 1 is important to me for the theoretical uptime. But due to anything less than x570 posing a problem with how the pcie bus works (it falls out of sync), they won't release the driver for Linux. B450 (with b550 and x570 being better) works for windows with nvme raid with the cc drivers (load 1) bottom, 2)raid, then 3)config drivers at windows install - load drivers) but not Linux. It’s an attack vector issue. Tpm (irq 6) which is not spi tpm has problems with the bus management problem as well. Sata drives load after nvme. I think windows has encrypted memory (ram at bootup) with checksums to deal with preventing firmware upgrades through userspace, bus hacking through ram, and the ip theft problem. Most don't know the technology of nvme raid which causes unnecessary trouble. It's kinda like this: the bad guy installs malware on your computer which establishes a bad relay channel to where they know that law enforcement has gotten in or not. Then, the government once being aware breaks all your stuff without any remorse. You're left with the bill if you don't secure your devices and these devices are getting expensive. Asking for a warrant does no good and the grunts will keep saying that they don't care because they're jaded. Your tax dollars at work. Good for kids, bad for adults' pocket books. Adults fight, kids don't understand.

Your best bet is to get a pcie nvme raid card that has raid available. Your x570 motherboard should support bifurcation, check the support, not all do for pcie raid cards. A glotrends product (linux support by asmedia) lists it by showing if some motherboard manufacturer bios/uefi has the bifurcation option. SYBA is more expensive but their webpage claims linux support but limited by your motherboard. This should save you some weeks of time. And your witness as a Christian in my case, 😅

[justinkb]

chiming in that b650 doesn't work either, even for sata drives. I ported the driver to 6.9.3 but get hit by the licensing thing "not allowed by current license level" for a simple striping array