gem5-users@gem5.org

The gem5 Users mailing list

View all threads

cache-coherent numa config with classic memory system

AK
Antoine Kaufmann
Mon, May 15, 2023 11:47 AM

Hi everyone,

I am currently trying to build gem5 config with the classic memory/cache system
that reasonbly resembles a typical multi-socket x86 numa server, but have so far
not been successful. (I know Ruby has much more flexiblity here, but I would
also like a config with classic components if at all possible).

Conceptually I was expecting this should be straight forward: I create a
coherent xbar as the main system bus. For each numa node I create a separate
xbar where I connect the node's memory controller and local PCI devices, both
PIO and DMA, along with the shared L3 cache.

The bit I have not been able to figure out yet, is how to connect the
per-numa-node xbars to the system xbar.

Initially I was planning on using a pair of bridges (one per direction) to
connect each numa xbar to the system bus. The problem here is that the bridges
seem to be able to only advertise statically configured address ranges, fine for
the memory controllers but bad for PCI bars, which are configured dynamically.

Directly connecting both cpu and mem-side ports of the busses cross-wise clearly
also does not work because of overlapping address ranges once updates propagate.
(i.e. once numa node 0 receives the ranges from numa node 1 it will advertise to
the system xbar as well and then there are two ports with overlapping ranges).
I've also experimented with using the default port on the per-numa-node xbars to
connect to the system bus, but also without success.

So far I have not found a way to address this with existing classic memory
system components (despite coherent_xbar.hh mentioning that it can be used to
model things like qpi or hypertransport). Am I missing something obvious here?

Alternatively I am thinking of implementing a custom  "bi-directional" bridge
component that will only forward address ranges in the direction where it sees
them arrive first.

I would also appreciate any other ideas.

Thanks,
Antoine

--
Antoine Kaufmann
antoinek@mpi-sws.org

Hi everyone, I am currently trying to build gem5 config with the classic memory/cache system that reasonbly resembles a typical multi-socket x86 numa server, but have so far not been successful. (I know Ruby has much more flexiblity here, but I would also like a config with classic components if at all possible). Conceptually I was expecting this should be straight forward: I create a coherent xbar as the main system bus. For each numa node I create a separate xbar where I connect the node's memory controller and local PCI devices, both PIO and DMA, along with the shared L3 cache. The bit I have not been able to figure out yet, is how to connect the per-numa-node xbars to the system xbar. Initially I was planning on using a pair of bridges (one per direction) to connect each numa xbar to the system bus. The problem here is that the bridges seem to be able to only advertise statically configured address ranges, fine for the memory controllers but bad for PCI bars, which are configured dynamically. Directly connecting both cpu and mem-side ports of the busses cross-wise clearly also does not work because of overlapping address ranges once updates propagate. (i.e. once numa node 0 receives the ranges from numa node 1 it will advertise to the system xbar as well and then there are two ports with overlapping ranges). I've also experimented with using the default port on the per-numa-node xbars to connect to the system bus, but also without success. So far I have not found a way to address this with existing classic memory system components (despite coherent_xbar.hh mentioning that it can be used to model things like qpi or hypertransport). Am I missing something obvious here? Alternatively I am thinking of implementing a custom "bi-directional" bridge component that will only forward address ranges in the direction where it sees them arrive first. I would also appreciate any other ideas. Thanks, Antoine -- Antoine Kaufmann <antoinek@mpi-sws.org>