Hi,

Does anyone know how to do a chiplet based simulation on the latest Gem5
version with garnet?

My basic requirements are: running distributed C++ applications (using
multi-threading) on the Gem5 O3CPYU type, having the ability to control the
number of cores on individual chiplet dies (even one core per die is
sufficient), control the D-to-D bandwidth, and getting various statistics
including power consumption per die and in the interposer, overall latency
of communication between dies.

On the other hand these are optional: configuring a package with different
topologies (mesh XY, butterfly, etc), possibly different packaging
techniques (like 2D, 2.5D, 3D), simulate heterogenous components like GPU
with CPU, and other NoC parameters like flit with, interposer width, etc.

I tried an older version of Gem5 from Dr. Tushar Krishna's group called
gem5_chips (https://github.com/GT-CHIPS/gem5_chips) but I had to use an
older version of Python (2.7) and GCC (8) to even compile Gem5 on an Ubuntu
20.04 system. Despite that I was not able to run their benchmark beyond a
certain instruction limit (using max-insts) since the simulator started
throwing assertion errors in the garnet2.0/OutVcState.cc code.

I would really appreciate it if someone can point me towards the right
direction here.

Thank you. Best regards,
Preet.

The latest version of garnet in the gem5 repo allows you to model multi-chiplet NoCs. [The gem5_chips repo is deprecated - and was actually forked off an earlier version of HeteroGarnet].
Here is its documentation on the gem5 wiki: https://www.gem5.org/documentation/general_docs/ruby/heterogarnet/

Best,
Tushar
On Feb 21, 2024 at 11:17 AM -0500, Preet Derasari via gem5-users gem5-users@gem5.org, wrote:
Hi,

Does anyone know how to do a chiplet based simulation on the latest Gem5 version with garnet?

My basic requirements are: running distributed C++ applications (using multi-threading) on the Gem5 O3CPYU type, having the ability to control the number of cores on individual chiplet dies (even one core per die is sufficient), control the D-to-D bandwidth, and getting various statistics including power consumption per die and in the interposer, overall latency of communication between dies.

On the other hand these are optional: configuring a package with different topologies (mesh XY, butterfly, etc), possibly different packaging techniques (like 2D, 2.5D, 3D), simulate heterogenous components like GPU with CPU, and other NoC parameters like flit with, interposer width, etc.

I tried an older version of Gem5 from Dr. Tushar Krishna's group called gem5_chips (https://github.com/GT-CHIPS/gem5_chips) but I had to use an older version of Python (2.7) and GCC (8) to even compile Gem5 on an Ubuntu 20.04 system. Despite that I was not able to run their benchmark beyond a certain instruction limit (using max-insts) since the simulator started throwing assertion errors in the garnet2.0/OutVcState.cc code.

I would really appreciate it if someone can point me towards the right direction here.

Thank you. Best regards,
Preet.

Hi Preet,

There's a missing parameter in configs/ruby/Ruby.py:259 (maybe different, search makeTopology), you should add a full_system param to this method.
The order of these parameters should be same as makeTopology in KiteLarge_EWMC.py.

Best,
Congwu Zhang

From: Preet Derasari via gem5-users
Date: 2024-02-23 04:05
To: Srikant Bharadwaj
CC: The gem5 Users mailing list; Krishna, Tushar; Preet Derasari
Subject: [gem5-users] Re: Chiplet Simulation with Gem5
Hi Srikant,

Thank you so much for answering my questions and providing the reference file.

Since I am new to NoC simulation and using Garnet 3.0 (HeteroGarnet), I feel a little lost regarding running a SE or FS mode simulation.

I checked the documentation on Gem5 posted (https://www.gem5.org/documentation/general_docs/ruby/heterogarnet/) and read the Kite paper (https://dl.acm.org/doi/abs/10.5555/3437539.3437683).

Nevertheless I could not figure out how to run a simulation (specifically in SE mode) where I can use the topology you provided.

I am using the deprecated se.py file in the following manner:
build/X86/gem5.opt configs/deprecated/example/se.py --num-cpus=4 --num-dirs=4 --cpu-type=DerivO3CPU --cpu-clock=2.5GHz --caches --l1d_size=64kB --l1d_assoc=8 --l1i_size=32kB --l1i_assoc=8 --mem-type=SimpleMemory --mem-size=8GB --ruby --network=garnet --topology=KiteLarge_EWMC --mesh-rows=2 --cmd=</path/to/binary> --options="<binary args>"

When I run this command, I get the following message:

---=====
gem5 Simulator System.  https://www.gem5.org
gem5 is copyrighted software; use the --copyright option for details.

gem5 version 23.1.0.0
gem5 compiled Feb 19 2024 14:33:34
gem5 started Feb 22 2024 14:48:41
command line: build/X86/gem5.opt configs/deprecated/example/se.py --num-cpus=4 --num-dirs=4 --cpu-type=DerivO3CPU --cpu-clock=2.5GHz --caches --l1d_size=64kB --l1d_assoc=8 --l1i_size=32kB --l1i_assoc=8 --mem-type=SimpleMemory --mem-size=8GB --ruby --network=garnet --topology=KiteLarge_EWMC --mesh-rows=2 --cmd=</path/to/binary> '--options=<binary args>'

warn: The se.py script is deprecated. It will be removed in future releases of  gem5.
TypeError: makeTopology() missing 1 required positional argument: 'Router'

At:
gem5/configs/ruby/Ruby.py(262): create_system
configs/deprecated/example/se.py(268): <module>
src/python/m5/main.py(669): main

---======

I understand that the se.py file may not be equipped to run a HeteroGarnet simulation, but I could not find any other simulation files that would help me.

I would really appreciate it if you could help me find a simulation file along with any modifications needed to Gem5 to support a simulation with the Kite Large topology.

Thank you again for your help.

Sincerely,
Preet.

On Wed, Feb 21, 2024 at 8:08 PM Srikant Bharadwaj srikantvv@gmail.com wrote:
Hi Preet,

1. I am attaching a config file for Kite Large topology that you can use as a reference for building chiplet-based topologies. More info on Kite Large can be found in this paper: https://dl.acm.org/doi/abs/10.5555/3437539.3437683
2. HeteroGarnet only handles the network traffic. So HeteroGarnet is can handle any thread pinning that happens on the core levels as long as the cores are connected to each other using a topology and Heterogarnet is enabled.
3. Yes, you can perform SE mode simulations as well as Garnet standalone simulations with HeteroGarnet. Let me know if you have any questions.

Thanks,
Srikant

On Wed, Feb 21, 2024 at 1:55 PM Preet Derasari via gem5-users gem5-users@gem5.org wrote:
Dear Dr. Krishna,

Thank you so much for your response.

I read the documentation you linked to in your email and have a few questions.

1. Are there any example config files that you are aware of that can help me start with chiplet simulation (with some basic routing techniques and topologies)? If so, could you point me to them?
2. How does HeteroGarnet assign processing and other nodes to an application? Can I use methods like CPUSET() in C++ to assign a simulated core (node) to a thread (created using thread)?
3. Can I perform SE mode simulations of my applications, or is HeteroGarnet only applicable to FS (or the new gem5 v22 "boards")?

I really appreciate your help.

Thank you. Sincerely,
Preet.

On Wed, Feb 21, 2024 at 11:24 AM Krishna, Tushar tushar@ece.gatech.edu wrote:
The latest version of garnet in the gem5 repo allows you to model multi-chiplet NoCs. [The gem5_chips repo is deprecated - and was actually forked off an earlier version of HeteroGarnet].
Here is its documentation on the gem5 wiki: https://www.gem5.org/documentation/general_docs/ruby/heterogarnet/

Best,
Tushar
On Feb 21, 2024 at 11:17 AM -0500, Preet Derasari via gem5-users gem5-users@gem5.org, wrote:
Hi,

Does anyone know how to do a chiplet based simulation on the latest Gem5 version with garnet?

My basic requirements are: running distributed C++ applications (using multi-threading) on the Gem5 O3CPYU type, having the ability to control the number of cores on individual chiplet dies (even one core per die is sufficient), control the D-to-D bandwidth, and getting various statistics including power consumption per die and in the interposer, overall latency of communication between dies.

On the other hand these are optional: configuring a package with different topologies (mesh XY, butterfly, etc), possibly different packaging techniques (like 2D, 2.5D, 3D), simulate heterogenous components like GPU with CPU, and other NoC parameters like flit with, interposer width, etc.

I tried an older version of Gem5 from Dr. Tushar Krishna's group called gem5_chips (https://github.com/GT-CHIPS/gem5_chips) but I had to use an older version of Python (2.7) and GCC (8) to even compile Gem5 on an Ubuntu 20.04 system. Despite that I was not able to run their benchmark beyond a certain instruction limit (using max-insts) since the simulator started throwing assertion errors in the garnet2.0/OutVcState.cc code.

I would really appreciate it if someone can point me towards the right direction here.

Thank you. Best regards,
Preet.

gem5-users mailing list -- gem5-users@gem5.org
To unsubscribe send an email to gem5-users-leave@gem5.org

Hi Preet,

I think it's ok to add these options to Options.py, such as defining a function like addChipletOptions.
If you want to add these options in other files, you can add ruby related options in configs/ruby/Ruby.py.
For convenience, you should add them in a file.

Best,
Congwu

zhangcongwu@ict.ac.cn

From: Preet Derasari via gem5-users
Date: 2024-02-24 00:18
To: The gem5 Users mailing list
CC: zhangcongwu@ict.ac.cn; Preet Derasari
Subject: [gem5-users] Re: Chiplet Simulation with Gem5
Hi Congwu Zhang,

I was able to fix that error with this information, but now there are some other errors related to missing options that the KiteLarge_EWMC.py script requires.
Some of these may be:
concentration_factor
num_chiplets
num_mem_ctrls
chiplet_width
chiplet_clock
top_vc
tsv_width
noi_width
tsv_clock
bottom_vc
mem_clock

This may not be an exhaustive list of all the options needed, but I figured this out from the KiteLarge_EWMC.py file. I just looked for options exclusively found in that file and nowhere else in Gem5.
As far as I know, if I want these options enabled, I may need to add them to the Options.py file manually. Is that correct?
Also, if possible, do you know how I can add those options otherwise?

Thank you for your time and help. I really appreciate it.

Sincerely,
Preet.

On Fri, Feb 23, 2024 at 2:46 AM zhangcongwu--- via gem5-users gem5-users@gem5.org wrote:
Hi Preet,

There's a missing parameter in configs/ruby/Ruby.py:259 (maybe different, search makeTopology), you should add a full_system param to this method.
The order of these parameters should be same as makeTopology in KiteLarge_EWMC.py.

Best,
Congwu Zhang

From: Preet Derasari via gem5-users
Date: 2024-02-23 04:05
To: Srikant Bharadwaj
CC: The gem5 Users mailing list; Krishna, Tushar; Preet Derasari
Subject: [gem5-users] Re: Chiplet Simulation with Gem5
Hi Srikant,

Thank you so much for answering my questions and providing the reference file.

Since I am new to NoC simulation and using Garnet 3.0 (HeteroGarnet), I feel a little lost regarding running a SE or FS mode simulation.

I checked the documentation on Gem5 posted (https://www.gem5.org/documentation/general_docs/ruby/heterogarnet/) and read the Kite paper (https://dl.acm.org/doi/abs/10.5555/3437539.3437683).

Nevertheless I could not figure out how to run a simulation (specifically in SE mode) where I can use the topology you provided.

I am using the deprecated se.py file in the following manner:
build/X86/gem5.opt configs/deprecated/example/se.py --num-cpus=4 --num-dirs=4 --cpu-type=DerivO3CPU --cpu-clock=2.5GHz --caches --l1d_size=64kB --l1d_assoc=8 --l1i_size=32kB --l1i_assoc=8 --mem-type=SimpleMemory --mem-size=8GB --ruby --network=garnet --topology=KiteLarge_EWMC --mesh-rows=2 --cmd=</path/to/binary> --options="<binary args>"

When I run this command, I get the following message:

---=====
gem5 Simulator System.  https://www.gem5.org
gem5 is copyrighted software; use the --copyright option for details.

gem5 version 23.1.0.0
gem5 compiled Feb 19 2024 14:33:34
gem5 started Feb 22 2024 14:48:41
command line: build/X86/gem5.opt configs/deprecated/example/se.py --num-cpus=4 --num-dirs=4 --cpu-type=DerivO3CPU --cpu-clock=2.5GHz --caches --l1d_size=64kB --l1d_assoc=8 --l1i_size=32kB --l1i_assoc=8 --mem-type=SimpleMemory --mem-size=8GB --ruby --network=garnet --topology=KiteLarge_EWMC --mesh-rows=2 --cmd=</path/to/binary> '--options=<binary args>'

warn: The se.py script is deprecated. It will be removed in future releases of  gem5.
TypeError: makeTopology() missing 1 required positional argument: 'Router'

At:
gem5/configs/ruby/Ruby.py(262): create_system
configs/deprecated/example/se.py(268): <module>
src/python/m5/main.py(669): main

---======

I understand that the se.py file may not be equipped to run a HeteroGarnet simulation, but I could not find any other simulation files that would help me.

I would really appreciate it if you could help me find a simulation file along with any modifications needed to Gem5 to support a simulation with the Kite Large topology.

Thank you again for your help.

Sincerely,
Preet.

On Wed, Feb 21, 2024 at 8:08 PM Srikant Bharadwaj srikantvv@gmail.com wrote:
Hi Preet,

1. I am attaching a config file for Kite Large topology that you can use as a reference for building chiplet-based topologies. More info on Kite Large can be found in this paper: https://dl.acm.org/doi/abs/10.5555/3437539.3437683
2. HeteroGarnet only handles the network traffic. So HeteroGarnet is can handle any thread pinning that happens on the core levels as long as the cores are connected to each other using a topology and Heterogarnet is enabled.
3. Yes, you can perform SE mode simulations as well as Garnet standalone simulations with HeteroGarnet. Let me know if you have any questions.

Thanks,
Srikant

On Wed, Feb 21, 2024 at 1:55 PM Preet Derasari via gem5-users gem5-users@gem5.org wrote:
Dear Dr. Krishna,

Thank you so much for your response.

I read the documentation you linked to in your email and have a few questions.

1. Are there any example config files that you are aware of that can help me start with chiplet simulation (with some basic routing techniques and topologies)? If so, could you point me to them?
2. How does HeteroGarnet assign processing and other nodes to an application? Can I use methods like CPUSET() in C++ to assign a simulated core (node) to a thread (created using thread)?
3. Can I perform SE mode simulations of my applications, or is HeteroGarnet only applicable to FS (or the new gem5 v22 "boards")?

I really appreciate your help.

Thank you. Sincerely,
Preet.

On Wed, Feb 21, 2024 at 11:24 AM Krishna, Tushar tushar@ece.gatech.edu wrote:
The latest version of garnet in the gem5 repo allows you to model multi-chiplet NoCs. [The gem5_chips repo is deprecated - and was actually forked off an earlier version of HeteroGarnet].
Here is its documentation on the gem5 wiki: https://www.gem5.org/documentation/general_docs/ruby/heterogarnet/

Best,
Tushar
On Feb 21, 2024 at 11:17 AM -0500, Preet Derasari via gem5-users gem5-users@gem5.org, wrote:
Hi,

Does anyone know how to do a chiplet based simulation on the latest Gem5 version with garnet?

My basic requirements are: running distributed C++ applications (using multi-threading) on the Gem5 O3CPYU type, having the ability to control the number of cores on individual chiplet dies (even one core per die is sufficient), control the D-to-D bandwidth, and getting various statistics including power consumption per die and in the interposer, overall latency of communication between dies.

On the other hand these are optional: configuring a package with different topologies (mesh XY, butterfly, etc), possibly different packaging techniques (like 2D, 2.5D, 3D), simulate heterogenous components like GPU with CPU, and other NoC parameters like flit with, interposer width, etc.

I tried an older version of Gem5 from Dr. Tushar Krishna's group called gem5_chips (https://github.com/GT-CHIPS/gem5_chips) but I had to use an older version of Python (2.7) and GCC (8) to even compile Gem5 on an Ubuntu 20.04 system. Despite that I was not able to run their benchmark beyond a certain instruction limit (using max-insts) since the simulator started throwing assertion errors in the garnet2.0/OutVcState.cc code.

I would really appreciate it if someone can point me towards the right direction here.

Thank you. Best regards,
Preet.

gem5-users mailing list -- gem5-users@gem5.org
To unsubscribe send an email to gem5-users-leave@gem5.org

gem5-users mailing list -- gem5-users@gem5.org
To unsubscribe send an email to gem5-users-leave@gem5.org