Use external SDRAM as RAM

[dgarfias]

Greetings,
My FPGA (DE10-Lite) has an 64M SDRAM chip. I was able to configure it using the Wishbone interface and have already done a memtest so it seems to be working, but I wanted to use it as RAM (instead of DMEM).

I tried modifying the linker script memory configuration section to ram (rwx) : ORIGIN = 0x90000000, LENGTH = 64*1024*1024. But it clearly does not look to be working and I really don't know what I am doing. I wanted to ask what would be the correct way to do it.

[stnolting]

Hey there!

At first I would recommend to check if memory accesses from the processor to the SDRAM work flawlessly. You could use the following setup:

• use the default memory configuration with processor-internal (BRAM-based) memories
• use the default linker script (no need to expose the SDRAM yet)
• run the sw/example/bus_explorer from internal memories; this program provides a simple UART console that allows to trigger arbitrary memory accesses to any address within the 32-bit address space; here is an example:

<<< NEORV32 Bus Explorer >>>

This program allows to read/write/dump memory space by hand.
Type 'help' to see the help menu.

BUS_EXPLORER:> help
Available commands:
 help   - show this text
 setup  - configure memory access width (byte,half,word)
 read   - read from address (byte,half,word)
 write  - write to address (byte,half,word)
 atomic - perform atomic LR/SC access (word-only)
 dump   - dump several bytes/halfs/words from base address
BUS_EXPLORER:> setup
Select data size (press 'x' to abort):
 'b' - byte, 8-bit, unsigned
 'h' - half-word, 16-bit, unsigned
 'w' - word, 32-bit, unsigned
selection: w
BUS_EXPLORER:> write
Enter address (8 hex chars): 0xf0000000
Enter data (8 hex chars): 0xababcdcd
[0xf0000000] <= 0xababcdcd
BUS_EXPLORER:> read
Enter address (8 hex chars): 0xf0000000
[0xf0000000] => 0xababcdcd
BUS_EXPLORER:>

The nice thing here is that you could "spy" on the actual SDRAM accesses by inserting an ILA. 😉

---

When the memory tests are ok, we can come back to making the SDRAM the primary DMEM. You linker script modifications look good so far.

Do you still use the internal DMEM in your setup? Then you will have an address space hole between the end of the internal DMEM and the start of the SDRAM, which the C runtime cannot handle. In this case I would recommend disabling the internal DMEM (MEM_INT_IMEM_EN => false,) and relocating your SDRAM to 0x80000000.

[stnolting]

The memory tests seem to work fine, I wrote to and read from many addresses in the SDRAM memory space without any problems using the bus_explorer, and also ran a memtest.

That sounds good!

The bootloader requires at least 512 bytes of RAM (📚 Data Sheet: Bootloader). The pre-installed bootloader image was compiled using the default linker script - so it requires the RAM to start at address 0x80000000. Relocate your SDRAM to that base address; that should make the bootloader work again.

By the way, what is the worst-case access time (in cycles) to your SDRAM?
The processor's external memory interface has a default "time out" (📚 MEM_EXT_TIMEOUT generic, default = 256 cycles). If the memory access is not acknowledge within this time window the transfer will be aborted and the CPU will raise an exception.

[dgarfias]

Thanks for replying.

So, if I understand correctly, if the internal DMEM is not implemented then accesses to 0x80000000 will be forwarded to the external bus, right? My SDRAM is at the start of the external bus so it should be working unless I am missing something or I am not doing the modifications correctly.

About the memory access, according to the datasheet of the IS42S16320F (https://www.issi.com/WW/pdf/42-45R-S_86400F-16320F.pdf) there should be no problem.

[stnolting]

So, if I understand correctly, if the internal DMEM is not implemented then accesses to 0x80000000 will be forwarded to the external bus, right?

That is correct - of course this requires that the external memory interface is enabled (MEM_EXT_EN = true,).
More information can be found here: Data Sheet: External Memories

About the memory access, according to the datasheet of the IS42S16320F (https://www.issi.com/WW/pdf/42-45R-S_86400F-16320F.pdf) there should be no problem.

What about the SDRAM controller itself? The controller will require several cycles for example to open a certain memory bank. Furthermore, refresh cycles will interrupt normal memory access requiring additional cycles to complete.

My SDRAM is at the start of the external bus so it should be working unless I am missing something or I am not doing the modifications correctly.

For a first setup you could try setting MEM_EXT_TIMEOUT to very high value (like 16384). Alternatively, you could try disabling the timeout function by setting the generic to zero.

[dgarfias]

Okay :) I understand. I tried disabling the timeout and now it boots and I get UART but the CPU raises an exception. [ERROR - Unexpected exception! mcause=0x00000001 mepc=0xffff0e3c mtval=0xffff0e40] trying to resume...

Edit: I measured the cpu cycles it takes for a memory read and the result was 48 (too high, I guess) so I modified max_proc_int_response_time_cmax_proc_int_response_time_c to 64 and now it's working perfectly. Using internal DMEM I got a coremark of 95 and now I get a coremark of 53, I guess this is the price to pay for having 64M of external DMEM :)

[stnolting]

I tried disabling the timeout and now it boots and I get UART

That's great! 👍

but the CPU raises an exception. [ERROR - Unexpected exception! mcause=0x00000001 mepc=0xffff0e3c mtval=0xffff0e40] trying to resume...

That's interesting....
mcause=0x00000001 means an illegal instruction - so the CPU tried to execute an invalid instruction. In this case this was fetched from mepc=0xffff0e3c, which is the .rodata section of the bootloader (containing constants like strings). I think there was a problem during SDRAM access and the CPU fetched an invalid return address from it and finally landed somewhere in the .rodata section. I would have thought that this is caused by a problem in the Wishbone<->SDRAM-controller handshake.

I measured the cpu cycles it takes for a memory read and the result was 48 (too high, I guess) so I modified max_proc_int_response_time_c to 64

That's even more interesting! 😉
The processor-internal bus timeout should be ignored when doing accesses via wishbone - MEM_EXT_TIMEOUT should be used in this case. Seems like I need to check this logic again...

Anyway, good to hear your system is working now!

now I get a coremark of 53, I guess this is the price to pay for having 64M of external DMEM

Memory transfers are the problem here. Having a data cache would hide most of the SDRAM's latency. I am working on that... 😉
However, you could use a "Wishbone" cache between the processor and the SDRAM controller as a workaround. I think I saw such buffers somewhere on GitHub.