gem5-users@gem5.org

The gem5 Users mailing list

View all threads

Trying to add barrier to threads example

GP
Gautam Pathak
Thu, Sep 1, 2022 10:57 PM

Hi All,

I'm trying to add a custom thread barrier to the array_add function by using __sync_fetch_and_add. The system I'm using is MSI using Ruby in SE mode. When I run this, I get the following error: build/X86/mem/ruby/system/RubyPort.cc:434: fatal: Ruby functional read failed for address 0x15b918.
Till now I have observed that there is an open issue which mentions that Ruby+multithreading+SE is not supported. The exact fault occurs in Futex syscall. (Ref. https://gem5.atlassian.net/browse/GEM5-676). However, I have tested the same for MI_Example protocol (using simple_ruby.py) and MESI_Two_Level (using se.py) upto 30+ threads and it works. This makes me wonder whether this is an MSI protocol bug. Any suggestions on how I should go on about this issue would be appreciated! Otherwise, is Full System mode the recommended option for Ruby+custom coherence protocol+multithreading (possibly using atomics)?

Thanks and Regards,
Gautam Pathak

Hi All, I'm trying to add a custom thread barrier to the array_add function by using __sync_fetch_and_add. The system I'm using is MSI using Ruby in SE mode. When I run this, I get the following error: build/X86/mem/ruby/system/RubyPort.cc:434: fatal: Ruby functional read failed for address 0x15b918. Till now I have observed that there is an open issue which mentions that Ruby+multithreading+SE is not supported. The exact fault occurs in Futex syscall. (Ref. https://gem5.atlassian.net/browse/GEM5-676). However, I have tested the same for MI_Example protocol (using simple_ruby.py) and MESI_Two_Level (using se.py) upto 30+ threads and it works. This makes me wonder whether this is an MSI protocol bug. Any suggestions on how I should go on about this issue would be appreciated! Otherwise, is Full System mode the recommended option for Ruby+custom coherence protocol+multithreading (possibly using atomics)? Thanks and Regards, Gautam Pathak
JL
Jason Lowe-Power
Fri, Sep 2, 2022 2:51 PM

Hi Gautam,

Functional accesses (i.e., fake/debug accesses that bypass all timing) and
Ruby do not play well together. Fundamentally, it is hard to know what is
the most up to date value or which value(s) you have to update when a cache
block is in an intermediate state. With MI_example and MESI, these
protocols probably have more support for functional reads than MSI, but
you're also just getting lucky with those. At some point, if you're doing
functional accesses, it's going to fail.

In SE mode, to implement the fake system calls which execute in the
simulator
instead of via the guest kernel executing on the simulated CPU,
we must use the fake/unrealistic/magic functional accesses. So, this is why
SE mode + multithreading + Ruby will almost always run into a problem at
some point. FS mode will not have this issue because it does not use
functional accesses (almost none at all) and nothing is "faked." The
simulated CPU executes all of the kernel system calls.

Hopefully this answers your question. Let me know if I can provide more
info :)

Cheers,
Jason

On Thu, Sep 1, 2022 at 3:58 PM Gautam Pathak gautam.pathak@uwaterloo.ca
wrote:

Hi All,

I'm trying to add a custom thread barrier to the array_add function by
using __sync_fetch_and_add. The system I'm using is MSI using Ruby in SE
mode. When I run this, I get the following error:
build/X86/mem/ruby/system/RubyPort.cc:434: fatal: Ruby functional read
failed for address 0x15b918.
Till now I have observed that there is an open issue which mentions that
Ruby+multithreading+SE is not supported. The exact fault occurs in Futex
syscall. (Ref. https://gem5.atlassian.net/browse/GEM5-676). However, I
have tested the same for MI_Example protocol (using simple_ruby.py) and
MESI_Two_Level (using se.py) upto 30+ threads and it works. This makes me
wonder whether this is an MSI protocol bug. Any suggestions on how I should
go on about this issue would be appreciated! Otherwise, is Full System mode
the recommended option for Ruby+custom coherence protocol+multithreading
(possibly using atomics)?

Thanks and Regards,
Gautam Pathak


gem5-users mailing list -- gem5-users@gem5.org
To unsubscribe send an email to gem5-users-leave@gem5.org

Hi Gautam, Functional accesses (i.e., fake/debug accesses that bypass all timing) and Ruby do not play well together. Fundamentally, it is hard to know what is the most up to date value or which value(s) you have to update when a cache block is in an intermediate state. With MI_example and MESI, these protocols probably have more support for functional reads than MSI, but you're also just getting lucky with those. At some point, if you're doing functional accesses, it's going to fail. In SE mode, to implement the fake system calls which execute *in the simulator* instead of via the guest kernel executing on the simulated CPU, we must use the fake/unrealistic/magic functional accesses. So, this is why SE mode + multithreading + Ruby will almost always run into a problem at some point. FS mode will not have this issue because it does not use functional accesses (almost none at all) and nothing is "faked." The simulated CPU executes all of the kernel system calls. Hopefully this answers your question. Let me know if I can provide more info :) Cheers, Jason On Thu, Sep 1, 2022 at 3:58 PM Gautam Pathak <gautam.pathak@uwaterloo.ca> wrote: > Hi All, > > I'm trying to add a custom thread barrier to the array_add function by > using __sync_fetch_and_add. The system I'm using is MSI using Ruby in SE > mode. When I run this, I get the following error: > build/X86/mem/ruby/system/RubyPort.cc:434: fatal: Ruby functional read > failed for address 0x15b918. > Till now I have observed that there is an open issue which mentions that > Ruby+multithreading+SE is not supported. The exact fault occurs in Futex > syscall. (Ref. https://gem5.atlassian.net/browse/GEM5-676). However, I > have tested the same for MI_Example protocol (using simple_ruby.py) and > MESI_Two_Level (using se.py) upto 30+ threads and it works. This makes me > wonder whether this is an MSI protocol bug. Any suggestions on how I should > go on about this issue would be appreciated! Otherwise, is Full System mode > the recommended option for Ruby+custom coherence protocol+multithreading > (possibly using atomics)? > > Thanks and Regards, > Gautam Pathak > _______________________________________________ > gem5-users mailing list -- gem5-users@gem5.org > To unsubscribe send an email to gem5-users-leave@gem5.org >
GP
Gautam Pathak
Sat, Sep 3, 2022 8:09 PM

Hi Prof. Jason,

Thank you for the clarification. I further wanted to ask whether there is any other way to do multithreaded application tests correctly in SE+ruby mode? If not, then we'll be forced to use FS mode which afaik does not work currently with MSI (our modified protocol will have to be changed again because FS mode requires (?) DMA support). Do you have any advice for things to watch out for while testing our multithreaded applications in Full System mode?

Thanks and Regards,
Gautam Pathak


From: Jason Lowe-Power jason@lowepower.com
Sent: Friday, September 2, 2022 10:51 AM
To: The gem5 Users mailing list gem5-users@gem5.org
Subject: [gem5-users] Re: Trying to add barrier to threads example

Hi Gautam,

Functional accesses (i.e., fake/debug accesses that bypass all timing) and Ruby do not play well together. Fundamentally, it is hard to know what is the most up to date value or which value(s) you have to update when a cache block is in an intermediate state. With MI_example and MESI, these protocols probably have more support for functional reads than MSI, but you're also just getting lucky with those. At some point, if you're doing functional accesses, it's going to fail.

In SE mode, to implement the fake system calls which execute in the simulator instead of via the guest kernel executing on the simulated CPU, we must use the fake/unrealistic/magic functional accesses. So, this is why SE mode + multithreading + Ruby will almost always run into a problem at some point. FS mode will not have this issue because it does not use functional accesses (almost none at all) and nothing is "faked." The simulated CPU executes all of the kernel system calls.

Hopefully this answers your question. Let me know if I can provide more info :)

Cheers,
Jason

On Thu, Sep 1, 2022 at 3:58 PM Gautam Pathak <gautam.pathak@uwaterloo.camailto:gautam.pathak@uwaterloo.ca> wrote:
Hi All,

I'm trying to add a custom thread barrier to the array_add function by using __sync_fetch_and_add. The system I'm using is MSI using Ruby in SE mode. When I run this, I get the following error: build/X86/mem/ruby/system/RubyPort.cc:434: fatal: Ruby functional read failed for address 0x15b918.
Till now I have observed that there is an open issue which mentions that Ruby+multithreading+SE is not supported. The exact fault occurs in Futex syscall. (Ref. https://gem5.atlassian.net/browse/GEM5-676). However, I have tested the same for MI_Example protocol (using simple_ruby.py) and MESI_Two_Level (using se.py) upto 30+ threads and it works. This makes me wonder whether this is an MSI protocol bug. Any suggestions on how I should go on about this issue would be appreciated! Otherwise, is Full System mode the recommended option for Ruby+custom coherence protocol+multithreading (possibly using atomics)?

Thanks and Regards,
Gautam Pathak


gem5-users mailing list -- gem5-users@gem5.orgmailto:gem5-users@gem5.org
To unsubscribe send an email to gem5-users-leave@gem5.orgmailto:gem5-users-leave@gem5.org

Hi Prof. Jason, Thank you for the clarification. I further wanted to ask whether there is any other way to do multithreaded application tests correctly in SE+ruby mode? If not, then we'll be forced to use FS mode which afaik does not work currently with MSI (our modified protocol will have to be changed again because FS mode requires (?) DMA support). Do you have any advice for things to watch out for while testing our multithreaded applications in Full System mode? Thanks and Regards, Gautam Pathak ________________________________ From: Jason Lowe-Power <jason@lowepower.com> Sent: Friday, September 2, 2022 10:51 AM To: The gem5 Users mailing list <gem5-users@gem5.org> Subject: [gem5-users] Re: Trying to add barrier to threads example Hi Gautam, Functional accesses (i.e., fake/debug accesses that bypass all timing) and Ruby do not play well together. Fundamentally, it is hard to know what is the most up to date value or which value(s) you have to update when a cache block is in an intermediate state. With MI_example and MESI, these protocols probably have more support for functional reads than MSI, but you're also just getting lucky with those. At some point, if you're doing functional accesses, it's going to fail. In SE mode, to implement the fake system calls which execute *in the simulator* instead of via the guest kernel executing on the simulated CPU, we must use the fake/unrealistic/magic functional accesses. So, this is why SE mode + multithreading + Ruby will almost always run into a problem at some point. FS mode will not have this issue because it does not use functional accesses (almost none at all) and nothing is "faked." The simulated CPU executes all of the kernel system calls. Hopefully this answers your question. Let me know if I can provide more info :) Cheers, Jason On Thu, Sep 1, 2022 at 3:58 PM Gautam Pathak <gautam.pathak@uwaterloo.ca<mailto:gautam.pathak@uwaterloo.ca>> wrote: Hi All, I'm trying to add a custom thread barrier to the array_add function by using __sync_fetch_and_add. The system I'm using is MSI using Ruby in SE mode. When I run this, I get the following error: build/X86/mem/ruby/system/RubyPort.cc:434: fatal: Ruby functional read failed for address 0x15b918. Till now I have observed that there is an open issue which mentions that Ruby+multithreading+SE is not supported. The exact fault occurs in Futex syscall. (Ref. https://gem5.atlassian.net/browse/GEM5-676). However, I have tested the same for MI_Example protocol (using simple_ruby.py) and MESI_Two_Level (using se.py) upto 30+ threads and it works. This makes me wonder whether this is an MSI protocol bug. Any suggestions on how I should go on about this issue would be appreciated! Otherwise, is Full System mode the recommended option for Ruby+custom coherence protocol+multithreading (possibly using atomics)? Thanks and Regards, Gautam Pathak _______________________________________________ gem5-users mailing list -- gem5-users@gem5.org<mailto:gem5-users@gem5.org> To unsubscribe send an email to gem5-users-leave@gem5.org<mailto:gem5-users-leave@gem5.org>
JL
Jason Lowe-Power
Tue, Sep 6, 2022 5:58 PM

Hi Gautam,

There are a number of limitations of SE mode including with respect to
multithreading as I described above. You may be able to get around the
functional access issue by extending the protocol to support functional
accesses better (e.g., improving the implementation of functionalRead and
functionalWrite in the cache controller). Otherwise, you can probably use
the MI_example or MESI protocols as a good example of how to implement
DMAs. Other than DMA, things should work in FS mode. As always, using the
Ruby random tester is faster than waiting for linux boot to run into an
error.

Cheers,
Jason

On Sat, Sep 3, 2022 at 1:10 PM Gautam Pathak gautam.pathak@uwaterloo.ca
wrote:

Hi Prof. Jason,

Thank you for the clarification. I further wanted to ask whether there is
any other way to do multithreaded application tests correctly in SE+ruby
mode? If not, then we'll be forced to use FS mode which afaik does not work
currently with MSI (our modified protocol will have to be changed again
because FS mode requires (?) DMA support). Do you have any advice for
things to watch out for while testing our multithreaded applications in
Full System mode?

Thanks and Regards,
Gautam Pathak

From: Jason Lowe-Power jason@lowepower.com
Sent: Friday, September 2, 2022 10:51 AM
To: The gem5 Users mailing list gem5-users@gem5.org
Subject: [gem5-users] Re: Trying to add barrier to threads example

Hi Gautam,

Functional accesses (i.e., fake/debug accesses that bypass all timing) and
Ruby do not play well together. Fundamentally, it is hard to know what is
the most up to date value or which value(s) you have to update when a cache
block is in an intermediate state. With MI_example and MESI, these
protocols probably have more support for functional reads than MSI, but
you're also just getting lucky with those. At some point, if you're doing
functional accesses, it's going to fail.

In SE mode, to implement the fake system calls which execute in the
simulator
instead of via the guest kernel executing on the simulated CPU,
we must use the fake/unrealistic/magic functional accesses. So, this is why
SE mode + multithreading + Ruby will almost always run into a problem at
some point. FS mode will not have this issue because it does not use
functional accesses (almost none at all) and nothing is "faked." The
simulated CPU executes all of the kernel system calls.

Hopefully this answers your question. Let me know if I can provide more
info :)

Cheers,
Jason

On Thu, Sep 1, 2022 at 3:58 PM Gautam Pathak gautam.pathak@uwaterloo.ca
wrote:

Hi All,

I'm trying to add a custom thread barrier to the array_add function by
using __sync_fetch_and_add. The system I'm using is MSI using Ruby in SE
mode. When I run this, I get the following error:
build/X86/mem/ruby/system/RubyPort.cc:434: fatal: Ruby functional read
failed for address 0x15b918.
Till now I have observed that there is an open issue which mentions that
Ruby+multithreading+SE is not supported. The exact fault occurs in Futex
syscall. (Ref. https://gem5.atlassian.net/browse/GEM5-676). However, I
have tested the same for MI_Example protocol (using simple_ruby.py) and
MESI_Two_Level (using se.py) upto 30+ threads and it works. This makes me
wonder whether this is an MSI protocol bug. Any suggestions on how I should
go on about this issue would be appreciated! Otherwise, is Full System mode
the recommended option for Ruby+custom coherence protocol+multithreading
(possibly using atomics)?

Thanks and Regards,
Gautam Pathak


gem5-users mailing list -- gem5-users@gem5.org
To unsubscribe send an email to gem5-users-leave@gem5.org


gem5-users mailing list -- gem5-users@gem5.org
To unsubscribe send an email to gem5-users-leave@gem5.org

Hi Gautam, There are a number of limitations of SE mode including with respect to multithreading as I described above. You may be able to get around the functional access issue by extending the protocol to support functional accesses better (e.g., improving the implementation of `functionalRead` and `functionalWrite` in the cache controller). Otherwise, you can probably use the MI_example or MESI protocols as a good example of how to implement DMAs. Other than DMA, things *should* work in FS mode. As always, using the Ruby random tester is faster than waiting for linux boot to run into an error. Cheers, Jason On Sat, Sep 3, 2022 at 1:10 PM Gautam Pathak <gautam.pathak@uwaterloo.ca> wrote: > Hi Prof. Jason, > > Thank you for the clarification. I further wanted to ask whether there is > any other way to do multithreaded application tests correctly in SE+ruby > mode? If not, then we'll be forced to use FS mode which afaik does not work > currently with MSI (our modified protocol will have to be changed again > because FS mode requires (?) DMA support). Do you have any advice for > things to watch out for while testing our multithreaded applications in > Full System mode? > > Thanks and Regards, > Gautam Pathak > ------------------------------ > *From:* Jason Lowe-Power <jason@lowepower.com> > *Sent:* Friday, September 2, 2022 10:51 AM > *To:* The gem5 Users mailing list <gem5-users@gem5.org> > *Subject:* [gem5-users] Re: Trying to add barrier to threads example > > Hi Gautam, > > Functional accesses (i.e., fake/debug accesses that bypass all timing) and > Ruby do not play well together. Fundamentally, it is hard to know what is > the most up to date value or which value(s) you have to update when a cache > block is in an intermediate state. With MI_example and MESI, these > protocols probably have more support for functional reads than MSI, but > you're also just getting lucky with those. At some point, if you're doing > functional accesses, it's going to fail. > > In SE mode, to implement the fake system calls which execute *in the > simulator* instead of via the guest kernel executing on the simulated CPU, > we must use the fake/unrealistic/magic functional accesses. So, this is why > SE mode + multithreading + Ruby will almost always run into a problem at > some point. FS mode will not have this issue because it does not use > functional accesses (almost none at all) and nothing is "faked." The > simulated CPU executes all of the kernel system calls. > > Hopefully this answers your question. Let me know if I can provide more > info :) > > Cheers, > Jason > > On Thu, Sep 1, 2022 at 3:58 PM Gautam Pathak <gautam.pathak@uwaterloo.ca> > wrote: > > Hi All, > > I'm trying to add a custom thread barrier to the array_add function by > using __sync_fetch_and_add. The system I'm using is MSI using Ruby in SE > mode. When I run this, I get the following error: > build/X86/mem/ruby/system/RubyPort.cc:434: fatal: Ruby functional read > failed for address 0x15b918. > Till now I have observed that there is an open issue which mentions that > Ruby+multithreading+SE is not supported. The exact fault occurs in Futex > syscall. (Ref. https://gem5.atlassian.net/browse/GEM5-676). However, I > have tested the same for MI_Example protocol (using simple_ruby.py) and > MESI_Two_Level (using se.py) upto 30+ threads and it works. This makes me > wonder whether this is an MSI protocol bug. Any suggestions on how I should > go on about this issue would be appreciated! Otherwise, is Full System mode > the recommended option for Ruby+custom coherence protocol+multithreading > (possibly using atomics)? > > Thanks and Regards, > Gautam Pathak > _______________________________________________ > gem5-users mailing list -- gem5-users@gem5.org > To unsubscribe send an email to gem5-users-leave@gem5.org > > _______________________________________________ > gem5-users mailing list -- gem5-users@gem5.org > To unsubscribe send an email to gem5-users-leave@gem5.org >