Hi All,
I'm trying to add a custom thread barrier to the array_add function by using __sync_fetch_and_add. The system I'm using is MSI using Ruby in SE mode. When I run this, I get the following error: build/X86/mem/ruby/system/RubyPort.cc:434: fatal: Ruby functional read failed for address 0x15b918.
Till now I have observed that there is an open issue which mentions that Ruby+multithreading+SE is not supported. The exact fault occurs in Futex syscall. (Ref. https://gem5.atlassian.net/browse/GEM5-676). However, I have tested the same for MI_Example protocol (using simple_ruby.py) and MESI_Two_Level (using se.py) upto 30+ threads and it works. This makes me wonder whether this is an MSI protocol bug. Any suggestions on how I should go on about this issue would be appreciated! Otherwise, is Full System mode the recommended option for Ruby+custom coherence protocol+multithreading (possibly using atomics)?
Thanks and Regards,
Gautam Pathak
Hi Gautam,
Functional accesses (i.e., fake/debug accesses that bypass all timing) and
Ruby do not play well together. Fundamentally, it is hard to know what is
the most up to date value or which value(s) you have to update when a cache
block is in an intermediate state. With MI_example and MESI, these
protocols probably have more support for functional reads than MSI, but
you're also just getting lucky with those. At some point, if you're doing
functional accesses, it's going to fail.
In SE mode, to implement the fake system calls which execute in the
simulator instead of via the guest kernel executing on the simulated CPU,
we must use the fake/unrealistic/magic functional accesses. So, this is why
SE mode + multithreading + Ruby will almost always run into a problem at
some point. FS mode will not have this issue because it does not use
functional accesses (almost none at all) and nothing is "faked." The
simulated CPU executes all of the kernel system calls.
Hopefully this answers your question. Let me know if I can provide more
info :)
Cheers,
Jason
On Thu, Sep 1, 2022 at 3:58 PM Gautam Pathak gautam.pathak@uwaterloo.ca
wrote:
Hi All,
I'm trying to add a custom thread barrier to the array_add function by
using __sync_fetch_and_add. The system I'm using is MSI using Ruby in SE
mode. When I run this, I get the following error:
build/X86/mem/ruby/system/RubyPort.cc:434: fatal: Ruby functional read
failed for address 0x15b918.
Till now I have observed that there is an open issue which mentions that
Ruby+multithreading+SE is not supported. The exact fault occurs in Futex
syscall. (Ref. https://gem5.atlassian.net/browse/GEM5-676). However, I
have tested the same for MI_Example protocol (using simple_ruby.py) and
MESI_Two_Level (using se.py) upto 30+ threads and it works. This makes me
wonder whether this is an MSI protocol bug. Any suggestions on how I should
go on about this issue would be appreciated! Otherwise, is Full System mode
the recommended option for Ruby+custom coherence protocol+multithreading
(possibly using atomics)?
Thanks and Regards,
Gautam Pathak
gem5-users mailing list -- gem5-users@gem5.org
To unsubscribe send an email to gem5-users-leave@gem5.org
Hi Prof. Jason,
Thank you for the clarification. I further wanted to ask whether there is any other way to do multithreaded application tests correctly in SE+ruby mode? If not, then we'll be forced to use FS mode which afaik does not work currently with MSI (our modified protocol will have to be changed again because FS mode requires (?) DMA support). Do you have any advice for things to watch out for while testing our multithreaded applications in Full System mode?
Thanks and Regards,
Gautam Pathak
From: Jason Lowe-Power jason@lowepower.com
Sent: Friday, September 2, 2022 10:51 AM
To: The gem5 Users mailing list gem5-users@gem5.org
Subject: [gem5-users] Re: Trying to add barrier to threads example
Hi Gautam,
Functional accesses (i.e., fake/debug accesses that bypass all timing) and Ruby do not play well together. Fundamentally, it is hard to know what is the most up to date value or which value(s) you have to update when a cache block is in an intermediate state. With MI_example and MESI, these protocols probably have more support for functional reads than MSI, but you're also just getting lucky with those. At some point, if you're doing functional accesses, it's going to fail.
In SE mode, to implement the fake system calls which execute in the simulator instead of via the guest kernel executing on the simulated CPU, we must use the fake/unrealistic/magic functional accesses. So, this is why SE mode + multithreading + Ruby will almost always run into a problem at some point. FS mode will not have this issue because it does not use functional accesses (almost none at all) and nothing is "faked." The simulated CPU executes all of the kernel system calls.
Hopefully this answers your question. Let me know if I can provide more info :)
Cheers,
Jason
On Thu, Sep 1, 2022 at 3:58 PM Gautam Pathak <gautam.pathak@uwaterloo.camailto:gautam.pathak@uwaterloo.ca> wrote:
Hi All,
I'm trying to add a custom thread barrier to the array_add function by using __sync_fetch_and_add. The system I'm using is MSI using Ruby in SE mode. When I run this, I get the following error: build/X86/mem/ruby/system/RubyPort.cc:434: fatal: Ruby functional read failed for address 0x15b918.
Till now I have observed that there is an open issue which mentions that Ruby+multithreading+SE is not supported. The exact fault occurs in Futex syscall. (Ref. https://gem5.atlassian.net/browse/GEM5-676). However, I have tested the same for MI_Example protocol (using simple_ruby.py) and MESI_Two_Level (using se.py) upto 30+ threads and it works. This makes me wonder whether this is an MSI protocol bug. Any suggestions on how I should go on about this issue would be appreciated! Otherwise, is Full System mode the recommended option for Ruby+custom coherence protocol+multithreading (possibly using atomics)?
Thanks and Regards,
Gautam Pathak
gem5-users mailing list -- gem5-users@gem5.orgmailto:gem5-users@gem5.org
To unsubscribe send an email to gem5-users-leave@gem5.orgmailto:gem5-users-leave@gem5.org
Hi Gautam,
There are a number of limitations of SE mode including with respect to
multithreading as I described above. You may be able to get around the
functional access issue by extending the protocol to support functional
accesses better (e.g., improving the implementation of functionalRead
and
functionalWrite
in the cache controller). Otherwise, you can probably use
the MI_example or MESI protocols as a good example of how to implement
DMAs. Other than DMA, things should work in FS mode. As always, using the
Ruby random tester is faster than waiting for linux boot to run into an
error.
Cheers,
Jason
On Sat, Sep 3, 2022 at 1:10 PM Gautam Pathak gautam.pathak@uwaterloo.ca
wrote:
Hi Prof. Jason,
Thank you for the clarification. I further wanted to ask whether there is
any other way to do multithreaded application tests correctly in SE+ruby
mode? If not, then we'll be forced to use FS mode which afaik does not work
currently with MSI (our modified protocol will have to be changed again
because FS mode requires (?) DMA support). Do you have any advice for
things to watch out for while testing our multithreaded applications in
Full System mode?
From: Jason Lowe-Power jason@lowepower.com
Sent: Friday, September 2, 2022 10:51 AM
To: The gem5 Users mailing list gem5-users@gem5.org
Subject: [gem5-users] Re: Trying to add barrier to threads example
Hi Gautam,
Functional accesses (i.e., fake/debug accesses that bypass all timing) and
Ruby do not play well together. Fundamentally, it is hard to know what is
the most up to date value or which value(s) you have to update when a cache
block is in an intermediate state. With MI_example and MESI, these
protocols probably have more support for functional reads than MSI, but
you're also just getting lucky with those. At some point, if you're doing
functional accesses, it's going to fail.
In SE mode, to implement the fake system calls which execute in the
simulator instead of via the guest kernel executing on the simulated CPU,
we must use the fake/unrealistic/magic functional accesses. So, this is why
SE mode + multithreading + Ruby will almost always run into a problem at
some point. FS mode will not have this issue because it does not use
functional accesses (almost none at all) and nothing is "faked." The
simulated CPU executes all of the kernel system calls.
Hopefully this answers your question. Let me know if I can provide more
info :)
Cheers,
Jason
On Thu, Sep 1, 2022 at 3:58 PM Gautam Pathak gautam.pathak@uwaterloo.ca
wrote:
Hi All,
I'm trying to add a custom thread barrier to the array_add function by
using __sync_fetch_and_add. The system I'm using is MSI using Ruby in SE
mode. When I run this, I get the following error:
build/X86/mem/ruby/system/RubyPort.cc:434: fatal: Ruby functional read
failed for address 0x15b918.
Till now I have observed that there is an open issue which mentions that
Ruby+multithreading+SE is not supported. The exact fault occurs in Futex
syscall. (Ref. https://gem5.atlassian.net/browse/GEM5-676). However, I
have tested the same for MI_Example protocol (using simple_ruby.py) and
MESI_Two_Level (using se.py) upto 30+ threads and it works. This makes me
wonder whether this is an MSI protocol bug. Any suggestions on how I should
go on about this issue would be appreciated! Otherwise, is Full System mode
the recommended option for Ruby+custom coherence protocol+multithreading
(possibly using atomics)?
Thanks and Regards,
Gautam Pathak
gem5-users mailing list -- gem5-users@gem5.org
To unsubscribe send an email to gem5-users-leave@gem5.org
gem5-users mailing list -- gem5-users@gem5.org
To unsubscribe send an email to gem5-users-leave@gem5.org