MEM_PCIE_RANGE_PCIE_START on rpi 4b with 4GB

[petemoore]

In #209 @dwhinham was having an issue with his Raspberry Pi 4B with 4GB RAM. There was an issue with the pcie interface of the bcm2711 "b0" stepping revision which was fixed in the "c0" revision which shipped in the Raspberry Pi 400. The issue reported by Dale may be related to the b0 stepping revision, since it didn't present for Rene on the c0 issue Raspberry Pi 400.

The Linux driver code states:

To further complicate matters, the [inbound] viewport must start on a pcie-address that is aligned on a multiple of its size.

Linux has a 1GB inbound viewport starting at pcie address 0xc0000000 (3GB). Currently circle also has a 1GB inbound viewport, but starting at pcie address 0xf8000000 (i.e. not at a multiple of its size).

I wonder if an alternative fix to #209 is to switch MEM_PCIE_RANGE_PCIE_START to 0xc0000000 to match Linux? Would this offer a path for using MSI again?

I performed a test on my Raspberry Pi 400, by applying the following patch to the develop branch:

diff --git a/include/circle/memorymap.h b/include/circle/memorymap.h
index e96f9208..74cacf70 100644
--- a/include/circle/memorymap.h
+++ b/include/circle/memorymap.h
@@ -81,7 +81,7 @@
 // PCIe memory range (outbound)
 #define MEM_PCIE_RANGE_START		0x600000000ULL
 #define MEM_PCIE_RANGE_SIZE		0x4000000ULL
-#define MEM_PCIE_RANGE_PCIE_START	0xF8000000ULL		// mapping on PCIe side
+#define MEM_PCIE_RANGE_PCIE_START	0xC0000000ULL		// mapping on PCIe side
 #define MEM_PCIE_RANGE_START_VIRTUAL	0xFA000000UL
 #define MEM_PCIE_RANGE_END_VIRTUAL	(MEM_PCIE_RANGE_START_VIRTUAL + 2*MEGABYTE - 1ULL)
 
diff --git a/include/circle/memorymap64.h b/include/circle/memorymap64.h
index 666f1282..35aa4bec 100644
--- a/include/circle/memorymap64.h
+++ b/include/circle/memorymap64.h
@@ -71,7 +71,7 @@
 // PCIe memory range (outbound)
 #define MEM_PCIE_RANGE_START		0x600000000UL
 #define MEM_PCIE_RANGE_SIZE		0x4000000UL
-#define MEM_PCIE_RANGE_PCIE_START	0xF8000000UL		// mapping on PCIe side
+#define MEM_PCIE_RANGE_PCIE_START	0xC0000000UL		// mapping on PCIe side
 #define MEM_PCIE_RANGE_START_VIRTUAL	MEM_PCIE_RANGE_START
 #define MEM_PCIE_RANGE_END_VIRTUAL	(MEM_PCIE_RANGE_START_VIRTUAL + MEM_PCIE_RANGE_SIZE - 1UL)
 

The USB keyboard continued to operate as before, suggesting switching the adjusted start address on the pcie side was ok. However, I don't have a Raspberry Pi 4B with 4GB and with the b0 stepping revision to test there too, and I haven't performed any other USB/PCIe tests.

I then applied the same patch to the parent of 3a3bec6 (i.e. prior to switching from MSI to INTA) and that also worked on my Raspberry Pi 400.

I looked at reverting 3a3bec6 (the fix for #209) from the develop branch and applying my patch instead, but the revert introduced a couple of merge conflicts that I didn't feel comfortable resolving. Rather than spend too long trying to create a patch and submit a PR, I thought it was best to just open a discussion here, and see what others think. Perhaps somebody with a Raspberry Pi 4B with the b0 stepping revision and 4GB RAM may be interested to try reverting to da386c7 (the revision before the fix to #209), confirming they can reproduce the issue described by Dale in #209, and then seeing if my patch above to adjust the pcie-side start address of the inbound viewport resolves the issue for them.

Thanks!

[petemoore]

... Note, if there are no objections to just changing the pcie start address to 0xc0000000, I can make a PR for that... (i.e. the patch above). The (optional) switch from INTA back to MSI could be independent of that patch, I guess. Just would be nice to see if that actually solves the issue. But even if it doesn't, it might make the configuration more aligned with Linux, which I believe uses 0xc0000000 on the PCIe side too...

... or maybe Linux gets these values from the dtb? I don't think it has a hardcoded MEM_PCIE_RANGE_PCIE_START like circle does ... And maybe the VC firmware is setting the value in the dtb dynamically before launching the Linux kernel... If so, that might be the better fix.

[petemoore]

If you like, I can take a stab at this Rene.

I see that the pcie_setup function in circle is based on the Linux brcm_pcie_setup function in Linux. Currently the circle version doesn't read values from the device tree but uses fixed values defined in constants. In contrast, Linux calls out to a separate brcm_pcie_get_rc_bar2_size_and_offset function that reads the device tree (dma_ranges) to determine the values.

I see examples in machineinfo.cpp for reading from the device tree (e.g. I see it reading from /axi/dma@10000) so I have a starting point to work from, if you are happy for me to have a go at this. I would create a similar function to brcm_pcie_get_rc_bar2_size_and_offset that would read the values from the device tree and adjust accordingly, to match the logic of Linux.

If you are happy for me to work on this, and are happy with the approach, let me know. Thanks Rene!

[rsta2]

Thanks Pete for investigating this! Unfortunately I'm not willing to integrate such modifications into Circle at the moment, because I cannot see a real benefit from this, while it would risk to produce problems on the great variety of Raspberry Pi 4 revisions and RAM sizes: https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#new-style-revision-codes-in-use I'm glad that these models all seem to work at the moment, I'm not aware of any problem reports, because I have to rely on this. I only have one RPi 4B 1.1 2 GB here, not more, and cannot test other models. BTW. MEM_PCIE_RANGE_PCIE_START is an outbound address, which is written into the BARs in the PCI configuration space of the PCIe device. It's not a inbound address. Circle must work without the DTB available on the RPi 4. Otherwise the chain boot feature would not work, which is used by some users and for the network bootloader.

[petemoore]

No worries Rene. I totally understand. I suspected this might be the case, so thought it best to ask before working on it. :-)

Note, the way I spotted this was by dumping the mmio read/writes under Linux and comparing with Circle. Although almost identical read/writes are performed, the rc bar2 offset was one of the notable differences. On my rpi400, the code in Linux here updates PCIE_MISC_RC_BAR2_CONFIG_LO and PCIE_MISC_RC_BAR2_CONFIG_HI, configuring rc bar2 as 0x00000000c0000000 where circle configures rc bar2 as 0x00000000fc000000. I believe if Linux were to use 0x00000000fc000000 for rc bar2 offset, then -EINVAL would get returned here.

It was perhaps confusing that I mentioned the inbound viewport, as indeed MEM_PCIE_RANGE_PCIE_START is a pcie address written to the PCI configuration space. Note, that the restriction is not about the ARM address of the inbound viewport being a multiple of the viewport size, it is a restriction about the PCIe address of the inbound viewport. In other words, the rc bar2 offset needs to be a multiple of the rc bar2 size. This would be consistent with the failure that Dale experienced under MSI using a rpi 4B with 4GB (if he had b0 stepping). It would also explain why this issue didn't present on the 1GB or 2GB rpi 4B models, nor on the 4GB rpi 400 with c0 stepping. But since Cirlce is not using MSI now, I agree it probably makes sense not to change it now. Maybe just something to be aware of, that it is a deviation from Linux configuration. I tested with 0xc0000000 and it worked for me, but that is a very limited test.

Anyway, yet again I've learned a ton from the circle library, thank you so much for your dedication to this project!

[rsta2]

Thanks Pete. Perhaps I can use your findings some day, when Circle has to support MSIs.