gem5-users@gem5.org

The gem5 Users mailing list

View all threads

Packet VALID_ADDR being cleared when try to resend it !

AA
Abdlerhman Abotaleb
Wed, Jul 13, 2022 3:52 PM

P {margin-top:0;margin-bottom:0;}  Hi all

What does it make a Packet's "VALID_ADDR" flag being cleared , even it was set before?

  1. I'm sending some packets from the cache to unblocking memory object that I created
  2. Inside this unblocking memory object, I crated a queue to store packets that fail to be sent
  3. After a while , this memory object receives a packet that failed to be sent , so this packet is blocked.
  4. Then, I'm inserting the packet to the queue of blocked requests.
  5. Inside the recvReqRetry I'm calling sendTimingReq(pkt).
  6. Eventually this packet is sent to  CoherentXBar::recvTimingReq it fails there.

The following assertion fails:

gem5.debug: build/RISCV/mem/packet.hh:832: bool gem5::Packet::isSecure() const: Assertion `flags.isSet(VALID_ADDR)' failed.

Even If I tried to enable any packet debugging method (like calling pkt->print()) a segmentation fault arises!

I can find only that Packet's flags.clear() happens :

a. happens when constructing a new Packet from request, but even with that the "VALID_ADDR" is being set if it has a PAddr.

(There are two constructors)

b. Another case , is for a third constructor that takes another packet as an argument, but this constructor set the "VALID_ADDR" and "VALID_SIZE" to be the same as the packet being passed.

I can't find anywhere that flags.clear(VALID_ADDR) happens.

So, what can be the problem cause?

Thank you.

regards,

Abdelrhman

AA
Abdlerhman Abotaleb
Wed, Jul 13, 2022 9:14 PM

P {margin-top:0;margin-bottom:0;}  After through debugging

It appears that the problem happens at the following call:

Inside "  CoherentXBar::recvTimingReq":

success = memSidePorts[mem_side_port_id]->sendTimingReq(pkt);
This "sendTimingReq" appears to cause Pkt modifications.

This function actually will end up calling recvTimingReq at the memory controller.

I found an intersteing comment few lines before calling this function:

// store the original address as an address mapper could possibly

// modify the address upon a sendTimingRequest

const Addr addr(pkt->getAddr());

I still have no clue what is happening , but this converges the problem search if one can help.

Still, if I tried to "print" the packet (i.e. pkt->print()) anywhere on the path (or at least enable CoherentXBar debug flag) , it will cause a segmentation fault!

Thank you.

regards,

Abdelrhman


From: Abdlerhman Abotaleb <abotalea@mcmaster.ca>
Sent: Wednesday, July 13, 2022 11:52 AM
To: Balazs Gerofi via gem5-users <gem5-users@gem5.org>
Subject: [gem5-users] Packet VALID_ADDR being cleared when try to resend it !

<!-- p {margin-top:0; margin-bottom:0} -->

Hi all

What does it make a Packet's "VALID_ADDR" flag being cleared , even it was set before?

  1. I'm sending some packets from the cache to unblocking memory object that I created
  2. Inside this unblocking memory object, I crated a queue to store packets that fail to be sent
  3. After a while , this memory object receives a packet that failed to be sent , so this packet is blocked.
  4. Then, I'm inserting the packet to the queue of blocked requests.
  5. Inside the recvReqRetry I'm calling sendTimingReq(pkt).
  6. Eventually this packet is sent to  CoherentXBar::recvTimingReq it fails there.

The following assertion fails:

gem5.debug: build/RISCV/mem/packet.hh:832: bool gem5::Packet::isSecure() const: Assertion `flags.isSet(VALID_ADDR)' failed.

Even If I tried to enable any packet debugging method (like calling pkt->print()) a segmentation fault arises!

I can find only that Packet's flags.clear() happens :

a. happens when constructing a new Packet from request, but even with that the "VALID_ADDR" is being set if it has a PAddr.

(There are two constructors)

b. Another case , is for a third constructor that takes another packet as an argument, but this constructor set the "VALID_ADDR" and "VALID_SIZE" to be the same as the packet being passed.

I can't find anywhere that flags.clear(VALID_ADDR) happens.

So, what can be the problem cause?

Thank you.

regards,

Abdelrhman

EM
Eliot Moss
Wed, Jul 13, 2022 10:27 PM

On 7/13/2022 5:14 PM, Abdlerhman Abotaleb wrote:

After through debugging
It appears that the problem happens at the following call:
Inside "CoherentXBar::recvTimingReq":
success = memSidePorts[mem_side_port_id]->sendTimingReq(pkt);
This "sendTimingReq" appears to cause Pkt modifications.
This function actually will end up calling *recvTimingReq *at the memory controller.

I found an intersteing comment few lines before calling this function:
// store the original address as an address mapper could possibly
// modify the address upon a sendTimingRequest
const Addr addr(pkt->getAddr());

I still have no clue what is happening , but this converges the problem search if one can help.
Still, if I tried to "print" the packet (i.e. pkt->print()) anywhere on the path (or at least enable
CoherentXBar debug flag) , it will cause a segmentation fault!

I looked at that code, and while it indicates that the address can be changed
via mapping, I don't think that would lead to a problem, though of course I
could be wrong.  What occurred to me was that, since you are queueing these
things for later processing, maybe the place that allocated them expected a
shorter lifetime, so part of the memory is getting freed and reused, which
could cause all kinds of oddities.  Among other things, this could happen if
the Packet object is local to some function, rather than being allocated
explicitly with new and later disposed of with an explicit delete.

This thought also could be wrong, but might suggest a place to look.

While it can be tricky to trigger on just the right thing in gdb, if the
packet's address in memory, or the address being accessed, are repeatable, you
can set a conditional breakpoint at the place where the packet is created, or
other places along the path of its processing, and possibly find where things
go off the rails.

Regards - Eliot Moss


From: Abdlerhman Abotaleb abotalea@mcmaster.ca
Sent: Wednesday, July 13, 2022 11:52 AM
To: Balazs Gerofi via gem5-users gem5-users@gem5.org
Subject: [gem5-users] Packet VALID_ADDR being cleared when try to resend it !
Hi all

What does it make a Packet's "VALID_ADDR" flag being cleared , even it was set before?

  1. I'm sending some packets from the cache to unblocking memory object that I created
  2. Inside this unblocking memory object, I crated a queue to store packets that fail to be sent
  3. After a while , this memory object receives a packet that failed to be sent , so this packet is
    blocked.
  4. Then, I'm inserting the packet to the queue of blocked requests.
  5. Inside the recvReqRetry I'm calling  sendTimingReq(pkt).
  6. Eventually this packet is sent to CoherentXBar::recvTimingReq it fails there.

The following assertion fails:
gem5.debug: build/RISCV/mem/packet.hh:832: bool gem5::Packet::isSecure() const:Assertion
`flags.isSet(VALID_ADDR)' failed.

Even If I tried to enable any packet debugging method (like calling pkt->print()) a segmentation
fault arises!

I can find only that Packet's flags.clear() happens :
      a. happens when constructing a new Packet from request, but even with that the "VALID_ADDR" is
being set if it has a PAddr.
(There are two constructors)
      b. Another case , is for a third constructor that takes another packet as an argument, but
this constructor set the "VALID_ADDR" and "VALID_SIZE" to be the same as the packet being passed.
      
I can't find anywhere that flags.clear(VALID_ADDR) happens.
So, what can be the problem cause?

Thank you.

regards,
Abdelrhman


gem5-users mailing list -- gem5-users@gem5.org
To unsubscribe send an email to gem5-users-leave@gem5.org

On 7/13/2022 5:14 PM, Abdlerhman Abotaleb wrote: > After through debugging > It appears that the problem happens at the following call: > Inside "CoherentXBar::recvTimingReq": > success = memSidePorts[mem_side_port_id]->*sendTimingReq*(pkt); > This "*sendTimingReq*" appears to cause Pkt modifications. > This function actually will end up calling *recvTimingReq *at the memory controller. > > I found an intersteing comment few lines before calling this function: > // store the original address as an address mapper could possibly > // modify the address upon a sendTimingRequest > const Addr addr(pkt->getAddr()); > > I still have no clue what is happening , but this converges the problem search if one can help. > Still, if I tried to "print" the packet (i.e. pkt->print()) anywhere on the path (or at least enable > CoherentXBar debug flag) , it will cause a segmentation fault! I looked at that code, and while it indicates that the address can be changed via mapping, I don't think that would lead to a problem, though of course I could be wrong. What occurred to me was that, since you are queueing these things for later processing, maybe the place that allocated them expected a shorter lifetime, so part of the memory is getting freed and reused, which could cause all kinds of oddities. Among other things, this could happen if the Packet object is local to some function, rather than being allocated explicitly with new and later disposed of with an explicit delete. This thought also could be wrong, but might suggest a place to look. While it can be tricky to trigger on just the right thing in gdb, if the packet's address in memory, or the address being accessed, are repeatable, you can set a conditional breakpoint at the place where the packet is created, or other places along the path of its processing, and possibly find where things go off the rails. Regards - Eliot Moss > ---------------------------------------------------------------------------------------------------- > *From:* Abdlerhman Abotaleb <abotalea@mcmaster.ca> > *Sent:* Wednesday, July 13, 2022 11:52 AM > *To:* Balazs Gerofi via gem5-users <gem5-users@gem5.org> > *Subject:* [gem5-users] Packet VALID_ADDR being cleared when try to resend it ! > Hi all > > What does it make a Packet's "VALID_ADDR" flag being cleared , even it was set before? > > 1. I'm sending some packets from the cache to unblocking memory object that I created > 2. Inside this unblocking memory object, I crated a queue to store packets that fail to be sent > 3. After a while , this memory object receives a packet that failed to be sent , so this packet is > blocked. > 4. Then, I'm inserting the packet to the queue of blocked requests. > 5. Inside the recvReqRetry I'm calling sendTimingReq(pkt). > 6. Eventually this packet is sent to CoherentXBar::recvTimingReq it fails there. > > The following assertion fails: > gem5.debug: build/RISCV/mem/packet.hh:832: bool gem5::Packet::isSecure() const:*Assertion > `flags.isSet(VALID_ADDR)' failed.* > Even If I tried to enable any packet debugging method (like calling pkt->print()) a segmentation > fault arises! > > I can find only that Packet's flags.clear() happens : >       a. happens when constructing a new Packet from request, but even with that the "VALID_ADDR" is > being set if it has a PAddr. > (There are two constructors) >       b. Another case , is for a third constructor that takes another packet as an argument, but > this constructor set the "VALID_ADDR" and "VALID_SIZE" to be the same as the packet being passed. >        > I can't find anywhere that flags.clear(VALID_ADDR) happens. > So, what can be the problem cause? > > Thank you. > > regards, > Abdelrhman > > > > > > > > > > _______________________________________________ > gem5-users mailing list -- gem5-users@gem5.org > To unsubscribe send an email to gem5-users-leave@gem5.org >
AA
Abdlerhman Abotaleb
Thu, Jul 14, 2022 6:46 PM

P {margin-top:0;margin-bottom:0;}  Thank you, Eliot, for your reply. 🙏

I solved it, but still need to understand the cause.

This is the full story:

I defined an STL queue of pointers to Packets to store the packets for further resending.

std::queue<PacketPtr> pktQueue;

It is defined as a member of the memory object class.

And I'm debugging before and after this call

Inside "  CoherentXBar::recvTimingReq":

success = memSidePorts[mem_side_port_id]->sendTimingReq(pkt);

Before the call, it appears that the Packet contains all correct fields (i.e. the address and flags etc.)

After  the call it fails to assert the "VALID_ADDR".

I solved this problem as the following:

Inside the new memory simulation object, when inserting a new blocked packet:

Before:

blockedPacketQueue.push(pkt)

After:

PacketPtr pPkt= new Packet(pkt,false,true);

pktQueue.push(pPkt);

I still can't figure why the first way doesn't work !


From: Eliot Moss <moss@cs.umass.edu>
Sent: Wednesday, July 13, 2022 6:27 PM
To: The gem5 Users mailing list <gem5-users@gem5.org>
Subject: [gem5-users] Re: Packet VALID_ADDR being cleared when try to resend it !

On 7/13/2022 5:14 PM, Abdlerhman Abotaleb wrote:
> After through debugging
> It appears that the problem happens at the following call:
> Inside "CoherentXBar::recvTimingReq":
> success = memSidePorts[mem_side_port_id]->sendTimingReq(pkt);
> This "sendTimingReq" appears to cause Pkt modifications.
> This function actually will end up calling *recvTimingReq *at the memory controller.
>
> I found an intersteing comment few lines before calling this function:
> // store the original address as an address mapper could possibly
> // modify the address upon a sendTimingRequest
> const Addr addr(pkt->getAddr());
>
> I still have no clue what is happening , but this converges the problem search if one can help.
> Still, if I tried to "print" the packet (i.e. pkt->print()) anywhere on the path (or at least enable
> CoherentXBar debug flag) , it will cause a segmentation fault!

I looked at that code, and while it indicates that the address can be changed
via mapping, I don't think that would lead to a problem, though of course I
could be wrong. What occurred to me was that, since you are queueing these
things for later processing, maybe the place that allocated them expected a
shorter lifetime, so part of the memory is getting freed and reused, which
could cause all kinds of oddities. Among other things, this could happen if
the Packet object is local to some function, rather than being allocated
explicitly with new and later disposed of with an explicit delete.

This thought also could be wrong, but might suggest a place to look.

While it can be tricky to trigger on just the right thing in gdb, if the
packet's address in memory, or the address being accessed, are repeatable, you
can set a conditional breakpoint at the place where the packet is created, or
other places along the path of its processing, and possibly find where things
go off the rails.

Regards - Eliot Moss

> ----------------------------------------------------------------------------------------------------
> From: Abdlerhman Abotaleb <abotalea@mcmaster.ca>
> Sent: Wednesday, July 13, 2022 11:52 AM
> To: Balazs Gerofi via gem5-users <gem5-users@gem5.org>
> Subject: [gem5-users] Packet VALID_ADDR being cleared when try to resend it !
> Hi all
>
> What does it make a Packet's "VALID_ADDR" flag being cleared , even it was set before?
>
> 1. I'm sending some packets from the cache to unblocking memory object that I created
> 2. Inside this unblocking memory object, I crated a queue to store packets that fail to be sent
> 3. After a while , this memory object receives a packet that failed to be sent , so this packet is
> blocked.
> 4. Then, I'm inserting the packet to the queue of blocked requests.
> 5. Inside the recvReqRetry I'm calling sendTimingReq(pkt).
> 6. Eventually this packet is sent to CoherentXBar::recvTimingReq it fails there.
>
> The following assertion fails:
> gem5.debug: build/RISCV/mem/packet.hh:832: bool gem5::Packet::isSecure() const:Assertion
> `flags.isSet(VALID_ADDR)' failed.

> Even If I tried to enable any packet debugging method (like calling pkt->print()) a segmentation
> fault arises!
>
> I can find only that Packet's flags.clear() happens :
> a. happens when constructing a new Packet from request, but even with that the "VALID_ADDR" is
> being set if it has a PAddr.
> (There are two constructors)
> b. Another case , is for a third constructor that takes another packet as an argument, but
> this constructor set the "VALID_ADDR" and "VALID_SIZE" to be the same as the packet being passed.
>
> I can't find anywhere that flags.clear(VALID_ADDR) happens.
> So, what can be the problem cause?
>
> Thank you.
>
> regards,
> Abdelrhman
>
>
>
>
>
>
>
>
>
> _______________________________________________
> gem5-users mailing list -- gem5-users@gem5.org
> To unsubscribe send an email to gem5-users-leave@gem5.org
>


gem5-users mailing list -- gem5-users@gem5.org
To unsubscribe send an email to gem5-users-leave@gem5.org

EM
Eliot Moss
Thu, Jul 14, 2022 8:21 PM

On 7/14/2022 2:46 PM, Abdlerhman Abotaleb wrote:

Thank you, Eliot, for your reply. 🙏
I solved it, but still need to understand the cause. *
*
*This is the full story: *
I defined an STL queue of pointers to Packets to store the packets for further resending.
std::queue<PacketPtr> pktQueue;
It is defined as a member of the memory object class.

And I'm debugging before and after this call
Inside "CoherentXBar::recvTimingReq":
success = memSidePorts[mem_side_port_id]->sendTimingReq(pkt);
Before the call, it appears that the Packet contains all correct fields (i.e. the address and
flags etc.)
*After * the call it fails to assert the "VALID_ADDR".

I *solved *this problem as the following:
Inside the newmemory simulation object, when inserting a new blocked packet:
Before:
blockedPacketQueue.push(pkt)
After:
PacketPtr pPkt= new Packet(pkt,false,true);
          pktQueue.push(pPkt);

I still can't figure why the first way doesn't work !

Well, I still wonder if the packet is getting freed
(or something) down in the sendTimingReq actions.
Among other things, I wonder if you are trying to save
it and it is getting responded to, which would turn
it into a response, etc.

Copying it like that is fixing the symptom, but I am
not sure it is fixing the real problem.

Regards - Eliot

On 7/14/2022 2:46 PM, Abdlerhman Abotaleb wrote: > Thank you, Eliot, for your reply. 🙏 > I solved it, but still need to understand the cause. * > * > *This is the full story: * > I defined an STL queue of pointers to Packets to store the packets for further resending. > std::queue<PacketPtr> pktQueue; > It is defined as a member of the memory object class. > > And I'm debugging before and after this call > Inside "CoherentXBar::recvTimingReq": > success = memSidePorts[mem_side_port_id]->*sendTimingReq*(pkt); > *Before* the call, it appears that the Packet contains all correct fields (i.e. the address and > flags etc.) > *After * the call it fails to assert the "VALID_ADDR". > > I *solved *this problem as the following: > Inside the newmemory simulation object, when inserting a new blocked packet: > Before: > blockedPacketQueue.push(pkt) > After: > PacketPtr pPkt= new Packet(pkt,false,true); >           pktQueue.push(pPkt); > > I still can't figure why the first way doesn't work ! Well, I still wonder if the packet is getting freed (or something) down in the sendTimingReq actions. Among other things, I wonder if you are trying to save it *and* it is getting responded to, which would turn it into a response, etc. Copying it like that is fixing the symptom, but I am not sure it is fixing the real problem. Regards - Eliot
AA
Abdlerhman Abotaleb
Fri, Jul 15, 2022 12:07 AM

P {margin-top:0;margin-bottom:0;}  Thanks Eliot for the follow up.

The reposne is created independently using the memory controller.

and the problem arises inside recvTimingReq, so it happens even before complete sending teh request.

I don't free up the pkt pointer "pkt", I pop the queue element, after the re-try manages to send the packet.

and pop doesn't do any allocation free-up.


From: Eliot Moss <moss@cs.umass.edu>
Sent: Thursday, July 14, 2022 4:21 PM
To: The gem5 Users mailing list <gem5-users@gem5.org>
Subject: [gem5-users] Re: Packet VALID_ADDR being cleared when try to resend it !

On 7/14/2022 2:46 PM, Abdlerhman Abotaleb wrote:
> Thank you, Eliot, for your reply. 🙏
> I solved it, but still need to understand the cause. *
> *
> *This is the full story: *
> I defined an STL queue of pointers to Packets to store the packets for further resending.
> std::queue<PacketPtr> pktQueue;
> It is defined as a member of the memory object class.
>
> And I'm debugging before and after this call
> Inside "CoherentXBar::recvTimingReq":
> success = memSidePorts[mem_side_port_id]->sendTimingReq(pkt);
> Before the call, it appears that the Packet contains all correct fields (i.e. the address and
> flags etc.)
> *After * the call it fails to assert the "VALID_ADDR".
>
> I *solved *this problem as the following:
> Inside the newmemory simulation object, when inserting a new blocked packet:
> Before:
> blockedPacketQueue.push(pkt)
> After:
> PacketPtr pPkt= new Packet(pkt,false,true);
> pktQueue.push(pPkt);
>
> I still can't figure why the first way doesn't work !

Well, I still wonder if the packet is getting freed
(or something) down in the sendTimingReq actions.
Among other things, I wonder if you are trying to save
it and it is getting responded to, which would turn
it into a response, etc.

Copying it like that is fixing the symptom, but I am
not sure it is fixing the real problem.

Regards - Eliot


gem5-users mailing list -- gem5-users@gem5.org
To unsubscribe send an email to gem5-users-leave@gem5.org

EM
Eliot Moss
Fri, Jul 15, 2022 12:48 AM

On 7/14/2022 8:07 PM, Abdlerhman Abotaleb wrote:

Thanks Eliot for the follow up.
The reposne is created independently using the memory controller.
and the problem arises inside recvTimingReq, so it happens even before complete sending teh request.

I don't free up the pkt pointer "pkt", I pop the queue element, after the re-try manages to send the
packet.
and pop doesn't do any allocation free-up.

I think we're talking a little at cross purposes.

I was suggesting that some of the existing gem5
code was somehow freeing the packet, while you
had retained a pointer to it.  So there's nothing
exactly wrong with your code in that scenario
except maybe having a wrong expectation about
the packet's lifetime.

EM

On 7/14/2022 8:07 PM, Abdlerhman Abotaleb wrote: > Thanks Eliot for the follow up. > The reposne is created independently using the memory controller. > and the problem arises inside recvTimingReq, so it happens even before complete sending teh request. > > I don't free up the pkt pointer "pkt", I pop the queue element, after the re-try manages to send the > packet. > and pop doesn't do any allocation free-up. I think we're talking a little at cross purposes. I was suggesting that some of the existing gem5 code was somehow freeing the packet, while you had retained a pointer to it. So there's nothing exactly wrong with your code in that scenario except maybe having a wrong expectation about the packet's lifetime. EM
AA
Abdlerhman Abotaleb
Fri, Jul 15, 2022 11:54 AM

P {margin-top:0;margin-bottom:0;} But in such case, GEM5 code will fail without my memory object being added and this is not the case, it was working perfectly.

Because, omitting the added simulation object, the packet pointer "pkt" is being sent to the crossbar then the memory controller.


From: Eliot Moss <moss@cs.umass.edu>
Sent: Thursday, July 14, 2022 8:48 PM
To: The gem5 Users mailing list <gem5-users@gem5.org>
Subject: [gem5-users] Re: Packet VALID_ADDR being cleared when try to resend it !

On 7/14/2022 8:07 PM, Abdlerhman Abotaleb wrote:
> Thanks Eliot for the follow up.
> The reposne is created independently using the memory controller.
> and the problem arises inside recvTimingReq, so it happens even before complete sending teh request.
>
> I don't free up the pkt pointer "pkt", I pop the queue element, after the re-try manages to send the
> packet.
> and pop doesn't do any allocation free-up.

I think we're talking a little at cross purposes.

I was suggesting that some of the existing gem5
code was somehow freeing the packet, while you
had retained a pointer to it. So there's nothing
exactly wrong with your code in that scenario
except maybe having a wrong expectation about
the packet's lifetime.

EM


gem5-users mailing list -- gem5-users@gem5.org
To unsubscribe send an email to gem5-users-leave@gem5.org

EM
Eliot Moss
Fri, Jul 15, 2022 12:47 PM

On 7/15/2022 7:54 AM, Abdlerhman Abotaleb wrote:

But in such case, GEM5 code will fail without my memory object being added and this is not the case,
it was working perfectly.
Because, omitting the added simulation object, the packet pointer "pkt" is being sent to the
crossbar then the memory controller.

Let me try again to explain the scenario I was thinking of.

The timing packet originates from a component such as the cpu.  In stages, as
timing delay time are reached, it is forwarded, component to component, to the
memory, then it is turn into a response, and sent back through the components,
generally in several steps, according to delays.  Then it is destroyed.

Your change put a pointer to the packet into some private queue - but the
packet was also sent along its regular path.  By the time you pull the
packet out of your queue and look at it, it has flowed through the whole
regular process and been deleted.

By making a copy, you of course avoid that - but the packet may be being
processed twice.

As I've said, I could be wrong about what is going on, but this is the
possible scenario I was trying to explain.

Regards - Eliot

On 7/15/2022 7:54 AM, Abdlerhman Abotaleb wrote: > But in such case, GEM5 code will fail without my memory object being added and this is not the case, > it was working perfectly. > Because, omitting the added simulation object, the packet pointer "pkt" is being sent to the > crossbar then the memory controller. Let me try again to explain the scenario I was thinking of. The timing packet originates from a component such as the cpu. In stages, as timing delay time are reached, it is forwarded, component to component, to the memory, then it is turn into a response, and sent back through the components, generally in several steps, according to delays. Then it is destroyed. Your change put a pointer to the packet into some private queue - but the packet was *also* sent along its regular path. By the time you pull the packet out of your queue and look at it, it has flowed through the whole regular process and been deleted. By making a copy, you of course avoid that - but the packet may be being processed twice. As I've said, I could be wrong about what is going on, but this is the possible scenario I was trying to explain. Regards - Eliot