configuration about Intel Optane NVDIMMs

[sunshineffmx]

Hi,
I’m curious about your experimental configuration that uses two DIMMs per socket and manages all eight DIMMs across four sockets with interleaved mode. Is this configuration beneficial to the data transmission bandwidth between GPU and PM? And why not deploy DIMMs in the same socket as much as possible?

[AKKamath]

Hi. Thanks for the interest.

We found the bandwidth across the PCIe between the GPU and PM scaled almost linearly with the number of channels used by NVM. That is, 8 DIMMs provided almost 8x as much bandwidth as a single DIMM on a single socket.

Our system was limited to 6 channels per socket, and so to achieve the maximum bandwidth, we were forced to distribute the DIMMs across multiple sockets.

[sunshineffmx]

Hi, Thanks for your reply.
I have understood your meaning, with more PM DIMMs interleaved, the data transmission bandwidth can be close to the maximum PCIe bandwidth. I have another question. If we have two configurations, 1) 2 DIMMs per socket and a total of 6 DIMMs interleaved across 3 sockets; 2) 6 DIMMs interleaved within one socket. Can they provide similar bandwidth?

[AKKamath]

I believe they should.

It's important to note that with 6 DIMMs in one socket, you can turn PM interleaving on in order to stripe the persistent file across the DIMMs.

For 2 DIMMs per socket, the persistent file is not automatically striped across DIMMs of all sockets.
Instead, you must use Linux's dm-stripe (https://pmem.io/blog/2018/05/using-persistent-memory-devices-with-the-linux-device-mapper/) to virtually stripe files across the DIMMs. This is what we used to stripe data in our configuration.
If this procedure isn't followed, the file is only kept across 2 DIMMs, and will only provide the bandwidth of 2 DIMMs.