Hi Jae-Eon,

You are right. I have just read the protocol files and it’s definitely not
trivial to modify the protocol implementation. But actually I got blocked
when I worked with the classical memory model to implement my cache
connection design.

I want a 2-core shared L1 and a 8-L1 shared L2, which means it will have
16 cores in the system. I used a coherent bus to connect the 2 cores and L1
cache, and found it can work. But when I connect the 8 L1s (which means 16
including i and d caches) with the L2, I found the request number of
snooping and the bus traffic is not affordable. And the bus busy also leads
to cpu’s fatal error. So maybe I need a directory protocol between L1 and
L2. That’s why I want to have a try at the ruby system.

So can I get a cache system which has a snooping protocol between the
multiple-cpu and l1 and a directory protocol between L1 and L2 in ruby
system? What’s your comment to implement my design?

Thanks a lot for your response.

Chao

On Sep 18, 2014, at 9:48 AM, Jae-Eon Jo via gem5-users <
gem5-users(a)gem5.org> wrote:

Hi, Chao,

As far as I know, each protocol of Ruby is tightly coupled with the memory
hierarchy.
That is, you should modify the protocol implementation
(src/mem/protocol/MESI_Two_Level*) to change the memory hierarchy, which is
not trivial.

My recommendation is to use the classic memory system (the default memory
system), which models MOESI snooping protocol.
(I think for interconnection network, if you do not have a special
purpose, snooping is a better choice only for two cores.)
This system is quite flexible in terms of memory hierarchy, so you can
only modify the configuration scripts to achieve what you want.

You will mostly modify two files (Note that if you modify a file under src
directory, you need to recompile gem5):
configs/common/CacheConfig.py    (option parsing & shared L2)
src/cpu/BaseCPU.py                        (private L1's & port
connections)

The current implementation has private L1 caches and shared L2 cache. To
make L1 cache shared, you can refer to how L2 shared cache is configured.

Thanks,
Jae-Eon

2014-09-17 23:45 GMT+09:00 Chao Zhang via gem5-users <gem5-users(a)gem5.org>
:

gem5-users mailing list
gem5-users(a)gem5.org
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

Okay.

If I understood your plan exactly, I have an idea to implement this easily.
As the implementation of CPU is decoupled from that of the memory system,
you will basically build 16-core system, but with the modification of port
assignment.

I'll just give an illustrative example of how it can be implemented (it
will not work if you just follow it.)
(1) Modify example/fs.py
Before:
if options.ruby:
...
for (i, cpu) in enumerate(test_sys.cpu):
After:
if options.ruby:
...
for (i, cpu) in enumerate(test_sys.cpu):
if i % 2 == 1: continue
Then you only make L1 caches for odd cpus.

(2) Modify topologies/Crossbar.py (if you use Crossbar topology)
Before:
routers = [Router(router_id=i) for i in range(len(self.nodes)+1)]
After:
routers = [Router(router_id=i) for i in range(len(self.nodes)/2+1)]
So, you only have a half number of routers.

Before:
ext_links = [ExtLink(link_id=i, ext_node=n, int_node=routers[i])
for (i, n) in enumerate(self.nodes)]
After:
ext_links = [ExtLink(link_id=i, ext_node=n, int_node=routers[i/2])
for (i, n) in enumerate(self.nodes)]
A pair of (odd, even) CPU is assigned to the same external port.

You can debug port assignment by appending the code right
after makeTopology call in configs/ruby/Ruby.py
for (i, node) in enumerate(topology.nodes):
print "id:%d name:%s" %(i, node.get_name())

Thanks,
Jae-Eon

2014-09-18 11:42 GMT+09:00 Chao Zhang <zhang.chao(a)pku.edu.cn>:

I forgot to mention that you have to use MESI two level of Ruby, if you want to follow my suggestion.

Thanks,
Jae-Eon

2014-09-18 14:33 GMT+09:00 Jae-Eon Jo <mirujj99(a)gmail.com>:
Okay.

If I understood your plan exactly, I have an idea to implement this easily.
As the implementation of CPU is decoupled from that of the memory system, you will basically build 16-core system, but with the modification of port assignment.

I'll just give an illustrative example of how it can be implemented (it will not work if you just follow it.)
(1) Modify example/fs.py
Before:
if options.ruby:
...
for (i, cpu) in enumerate(test_sys.cpu):
After:
if options.ruby:
...
for (i, cpu) in enumerate(test_sys.cpu):
if i % 2 == 1: continue
Then you only make L1 caches for odd cpus.

(2) Modify topologies/Crossbar.py (if you use Crossbar topology)
Before:
routers = [Router(router_id=i) for i in range(len(self.nodes)+1)]
After:
routers = [Router(router_id=i) for i in range(len(self.nodes)/2+1)]
So, you only have a half number of routers.

Before:
ext_links = [ExtLink(link_id=i, ext_node=n, int_node=routers[i])
for (i, n) in enumerate(self.nodes)]
After:
ext_links = [ExtLink(link_id=i, ext_node=n, int_node=routers[i/2])
for (i, n) in enumerate(self.nodes)]
A pair of (odd, even) CPU is assigned to the same external port.

You can debug port assignment by appending the code right after makeTopology call in configs/ruby/Ruby.py
for (i, node) in enumerate(topology.nodes):
print "id:%d name:%s" %(i, node.get_name())

Thanks,
Jae-Eon

2014-09-18 11:42 GMT+09:00 Chao Zhang <zhang.chao(a)pku.edu.cn>:
Hi Jae-Eon,

You are right. I have just read the protocol files and it’s definitely not trivial to modify the protocol implementation. But actually I got blocked when I worked with the classical memory model to implement my cache connection design.

I want a 2-core shared L1 and a 8-L1 shared L2, which means it will have 16 cores in the system. I used a coherent bus to connect the 2 cores and L1 cache, and found it can work. But when I connect the 8 L1s (which means 16 including i and d caches) with the L2, I found the request number of snooping and the bus traffic is not affordable. And the bus busy also leads to cpu’s fatal error. So maybe I need a directory protocol between L1 and L2. That’s why I want to have a try at the ruby system.

So can I get a cache system which has a snooping protocol between the multiple-cpu and l1 and a directory protocol between L1 and L2 in ruby system? What’s your comment to implement my design?

Thanks a lot for your response.

Chao

On Sep 18, 2014, at 9:48 AM, Jae-Eon Jo via gem5-users <gem5-users(a)gem5.org> wrote:

gem5-users mailing list
gem5-users(a)gem5.org
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

Okay.

If I understood your plan exactly, I have an idea to implement this easily.
As the implementation of CPU is decoupled from that of the memory system, you will basically build 16-core system, but with the modification of port assignment.

I'll just give an illustrative example of how it can be implemented (it will not work if you just follow it.)
(1) Modify example/fs.py
Before:
if options.ruby:
...
for (i, cpu) in enumerate(test_sys.cpu):
After:
if options.ruby:
...
for (i, cpu) in enumerate(test_sys.cpu):
if i % 2 == 1: continue
Then you only make L1 caches for odd cpus.

(2) Modify topologies/Crossbar.py (if you use Crossbar topology)
Before:
routers = [Router(router_id=i) for i in range(len(self.nodes)+1)]
After:
routers = [Router(router_id=i) for i in range(len(self.nodes)/2+1)]
So, you only have a half number of routers.

Before:
ext_links = [ExtLink(link_id=i, ext_node=n, int_node=routers[i])
for (i, n) in enumerate(self.nodes)]
After:
ext_links = [ExtLink(link_id=i, ext_node=n, int_node=routers[i/2])
for (i, n) in enumerate(self.nodes)]
A pair of (odd, even) CPU is assigned to the same external port.

You can debug port assignment by appending the code right after makeTopology call in configs/ruby/Ruby.py
for (i, node) in enumerate(topology.nodes):
print "id:%d name:%s" %(i, node.get_name())

Thanks,
Jae-Eon

2014-09-18 11:42 GMT+09:00 Chao Zhang <zhang.chao(a)pku.edu.cn>:
Hi Jae-Eon,

You are right. I have just read the protocol files and it’s definitely not trivial to modify the protocol implementation. But actually I got blocked when I worked with the classical memory model to implement my cache connection design.

I want a 2-core shared L1 and a 8-L1 shared L2, which means it will have 16 cores in the system. I used a coherent bus to connect the 2 cores and L1 cache, and found it can work. But when I connect the 8 L1s (which means 16 including i and d caches) with the L2, I found the request number of snooping and the bus traffic is not affordable. And the bus busy also leads to cpu’s fatal error. So maybe I need a directory protocol between L1 and L2. That’s why I want to have a try at the ruby system.

So can I get a cache system which has a snooping protocol between the multiple-cpu and l1 and a directory protocol between L1 and L2 in ruby system? What’s your comment to implement my design?

Thanks a lot for your response.

Chao

On Sep 18, 2014, at 9:48 AM, Jae-Eon Jo via gem5-users <gem5-users(a)gem5.org> wrote:

Hi Jae-Eon,

It begins to run now, after your suggestion.
But it goes to a CPU0 deadlock, which looks like following:

panic: Possible Deadlock detected. Aborting!

And according to its error information, there seams  be some memory
request staved, and the deadlock checker is activated.
Have you ever been through this kind of error? I will keep digging into it.

Regards,
Chao

On Sep 18, 2014, at 1:33 PM, Jae-Eon Jo <mirujj99(a)gmail.com
javascript:_e(%7B%7D,'cvml','mirujj99(a)gmail.com');> wrote:

Okay.

If I understood your plan exactly, I have an idea to implement this easily.
As the implementation of CPU is decoupled from that of the memory system,
you will basically build 16-core system, but with the modification of port
assignment.

I'll just give an illustrative example of how it can be implemented (it
will not work if you just follow it.)
(1) Modify example/fs.py
Before:
if options.ruby:
...
for (i, cpu) in enumerate(test_sys.cpu):
After:
if options.ruby:
...
for (i, cpu) in enumerate(test_sys.cpu):
if i % 2 == 1: continue
Then you only make L1 caches for odd cpus.

(2) Modify topologies/Crossbar.py (if you use Crossbar topology)
Before:
routers = [Router(router_id=i) for i in range(len(self.nodes)+1)]
After:
routers = [Router(router_id=i) for i in range(len(self.nodes)/2+1)]
So, you only have a half number of routers.

Before:
ext_links = [ExtLink(link_id=i, ext_node=n, int_node=routers[i])
for (i, n) in enumerate(self.nodes)]
After:
ext_links = [ExtLink(link_id=i, ext_node=n, int_node=routers[i/2])
for (i, n) in enumerate(self.nodes)]
A pair of (odd, even) CPU is assigned to the same external port.

You can debug port assignment by appending the code right
after makeTopology call in configs/ruby/Ruby.py
for (i, node) in enumerate(topology.nodes):
print "id:%d name:%s" %(i, node.get_name())

Thanks,
Jae-Eon

2014-09-18 11:42 GMT+09:00 Chao Zhang <zhang.chao(a)pku.edu.cn
javascript:_e(%7B%7D,'cvml','zhang.chao(a)pku.edu.cn');>:

Hi Chao,

Yes, I had exactly the same error, when I was trying to switch CPU. You can find the details from the link below.

http://www.mail-archive.com/gem5-users(a)gem5.org/msg10431.html

Unfortunately, currently I'm not digging into it, so do not know about this problem.

Thanks,
Jae-Eon

2014년 9월 18일 목요일, Chao Zhang<zhang.chao(a)pku.edu.cn>님이 작성한 메시지:
Hi Jae-Eon,

It begins to run now, after your suggestion.
But it goes to a CPU0 deadlock, which looks like following:

panic: Possible Deadlock detected. Aborting!

And according to its error information, there seams  be some memory request staved, and the deadlock checker is activated.
Have you ever been through this kind of error? I will keep digging into it.

Regards,
Chao

On Sep 18, 2014, at 1:33 PM, Jae-Eon Jo <mirujj99(a)gmail.com> wrote: