On 11.07.2012 17:15, Anthony Gutierrez wrote:

having a problem when switching between two arm_detailed model CPUs. I
am getting the following assertion failure:

build/ARM/dev/dma_device.cc:90: virtual bool
DmaPort::recvTimingResp(Packet*): Assertion `pendingCount >= 0' failed.

following trace, the switch_cpu is attempting to send a read request for
0xfffa7c4, shortly after it says it's "Done". A switchout occurs and the
cpu attempts to send a read request for addr 0x7ffc, shortly after it
says "Failed". It does a retry then says "Done". Later it attempts a
read response for both of these addresses, however the pendingCount is
only 1, thus leading to the assertion failure.

drain functionality to only drain when transmitList.size() == 0, both
inside drain() and when processing the drainEvent, this had no effect.
I'm not too familiar with how the DMA code should work so any ideas
would be appreciated. One thing that I noticed is that inside of the
do-while loops inside both sendDma() and recvRetry() when the
transmitList is popped, the pendingCount isn't decremented for Timing
mode. Should it be? Also, another thing that is strange is the the
drainEvent is only processed if state == Enums::atomic.

system.switch_cpus.itb.walker.dma: Starting DMA for addr: 0xfffa7c4
size: 4 sched: 0

pendingCount should be incremented here and sholud be

for addr: 0xfffa7c4 size: 4

system.switch_cpus.itb.walker.dma: Attempting to send ReadReq addr
0xfffa7c4

5403270500: system.cpu.itb.walker.dma: Starting DMA for addr: 0x7ffc
size: 4 sched: 0

for addr: 0x7ffc size: 4

Attempting to send ReadReq addr 0x7ffc

system.cpu.itb.walker.dma: -- Failed: queued

system.cpu.itb.walker.dma: Retry on ReadReq addr 0x7ffc

system.cpu.itb.walker.dma: -- Done

system.cpu.itb.walker.dma: TransmitList: 0, backoffTime: 0 inRetry: 0
es: 0

ReadResp addr 0xfffa7c4 size 0x4 pending cnt 1

It doesn't get
decremented until here, so the switch should have never been allowed to
happen.
5403285000: system.c

some debugging around the src/pythom/m5/simulatie.py:drain(root) to see
when drain is ca
. It should have been called and the code should have
never let the drain happen. Are you sure you're draining all objects in
the system?

Ali

I've added some more DPRINTF's to the dma port to track what is happening
with the pendingCount:

5403270500: system.switch_cpus.itb.walker.dma: Starting DMA for addr:
0xfffa7c4 size: 4 sched: 0 pending count: 0
5403270500: system.switch_cpus.itb.walker.dma: incrementing pendingCount:
0->1
5403270500: system.switch_cpus.itb.walker.dma: --Queuing DMA for addr:
0xfffa7c4 size: 4 pendingCout: 1
5403270500: system.switch_cpus.itb.walker.dma: Attempting to send ReadReq
addr 0xfffa7c4
5403270500: system.switch_cpus.itb.walker.dma: -- Done
5403270500: system.cpu.itb.walker.dma: Starting DMA for addr: 0x7ffc size:
4 sched: 0 pending count: 0
5403270500: system.cpu.itb.walker.dma: incrementing pendingCount: 0->1
5403270500: system.cpu.itb.walker.dma: --Queuing DMA for addr: 0x7ffc size:
4 pendingCout: 1
5403270500: system.cpu.itb.walker.dma: Attempting to send ReadReq addr
0x7ffc
5403270500: system.cpu.itb.walker.dma: -- Failed: queued
5403272000: system.cpu.itb.walker.dma: Retry on ReadReq addr 0x7ffc
5403272000: system.cpu.itb.walker.dma: -- Done
5403272000: system.cpu.itb.walker.dma: TransmitList: 0, backoffTime: 0
inRetry: 0 es: 0
5403282500: system.cpu.itb.walker.dma: Received response ReadResp addr
0xfffa7c4 size 0x4 pending cnt 1
5403282500: system.cpu.itb.walker.dma: decrementing pendingCount: 1->0
5403285000: system.cpu.itb.walker.dma: Received response ReadResp addr
0x7ffc size 0x4 pending cnt 0
5403285000: system.cpu.itb.walker.dma: decrementing pendingCount: 0->-1

though it's not being decremented.

Every object seems to be draining, assuming their drain functionality is
reasonable correct. Checking which objects haven't drained inside
simulate.py:drain()

info: Entering event queue @ 5303263000.  Starting simulation...
switch_cpus not drained and contributes 1 objects that need to be drained
walker not drained and contributes 2 objects that need to be drained
l2 not drained and contributes 1 objects that need to be drained
re-entering simulation inside drain() with 4 unready objects
info: Entering event queue @ 5403263000.  Starting simulation...
system.switch_cpus processing drain event
system.switch_cpus.itb.walker.dma processing drain event: recvTimingResp()
system.l2.cpu_side-CpuSidePort processing drain event
system.switch_cpus.itb.walker processing drain event
exiting simulation in drain()

switching cpus
info: Entering event queue @ 5403270500.  Starting simulation...
m5.opt: build/ARM/dev/dma_device.cc:86: virtual bool
DmaPort::recvTimingResp(Packet*): Assertion `pendingCount >= 0' failed.
Program aborted at cycle 5403285000
Aborted

On Wed, Jul 11, 2012 at 5:46 PM, Ali Saidi <saidi(a)umich.edu> wrote:

I've determined this is because drainEvent()->process() is being called
when the port has properly drained. However, once it signals that it has
drained, it gets another request. After this the l2 cpu-side port and and
itb walker itself signal drained and drain() exits. Seems there is a race
condition where the dma port drains before the other devices and it gets a
request in the meantime. I suppose I need to make sure the dma port does
not accept any more requests once it's signified that it's drained.
However, none of the dma code seems to rely on the State state; of the
underlying SimObject.

5403270000: system.switch_cpus.itb.walker.dma: Received response ReadResp
addr 0x7ffc size 0x4 pending cnt 1
5403270000: system.switch_cpus.itb.walker.dma: decrementing pendingCount:
1->0
5403270000: system.switch_cpus.itb.walker.dma: processing drain event
pending count: 0
5403270500: system.switch_cpus.itb.walker.dma: Starting DMA for addr:
0xfffa7c4 size: 4 sched: 0 pending count: 0
5403270500: system.switch_cpus.itb.walker.dma: incrementing pendingCount:
0->1
5403270500: system.switch_cpus.itb.walker.dma: --Queuing DMA for addr:
0xfffa7c4 size: 4 pendingCout: 1
5403270500: system.switch_cpus.itb.walker.dma: Attempting to send ReadReq
addr 0xfffa7c4
5403270500: system.switch_cpus.itb.walker.dma: -- Done
5403270500: system.cpu.itb.walker.dma: Starting DMA for addr: 0x7ffc size:
4 sched: 0 pending count: 0
5403270500: system.cpu.itb.walker.dma: incrementing pendingCount: 0->1

On Thu, Jul 12, 2012 at 2:21 AM, Anthony Gutierrez <atgutier(a)umich.edu>wrote:

On 12.07.2012 02:45, Anthony Gutierrez wrote:

this is because drainEvent()->process() is being called when the port
has properly drained. However, once it signals that it has drained, it
gets another request.

So the issue here isn't the tlb, but it's another
component in the system that either issued a request while it said it
was draining or improperly signaled it was drained.

l2 cpu-side port and and itb walker itself signal drained and drain()
exits. Seems there is a race condition where the dma port drains before
the other devices and it gets a request in the meantime. I suppose I
need to make sure the dma port does not accept any more requests once
it's signified that it's drained.

That won't quite work, because a
component on the other side of teh port might need to complete a request
before it can say it's drained. Really that component needs to handle
just itself. The reasoning si that no one component can ever say it's
done as there might be a request flowing through the system somewhere.
That is why drain is called multiple times. It's perfectly acceptable
for the component that was holding up the drain() to be done and signal
it's read to drain, but the only reason it's ready is that it's handed
responsibility for a request off to someone else. That is why we call
drain() successively until it comes back with 0 immediately.

Ali

However, none of t

So I have been following this thread as well since I too have an
interest in getting this working. From the trace, it seems that the
problematic request in this case is to address: 0x7ffc

The ITLB Walker is unable to queue the read request, and so it needs
to be retried

But it seems to retry again in the same cycle, is that what should
normally happen, or is it only retrying in the same cycle because its
waiting to drain?

And also, shouldn't the pendingCount be increased, even if it failed
on the first try (aka added to retryList?).

Would adding a nonzero backoff time alleviate the race condition, or
would doing that just avoid running into the scenario?

Malek

On Thu, Jul 12, 2012 at 9:14 AM, Ali Saidi <saidi(a)umich.edu> wrote:

You are right. The port is being "undrained" by the table walker. Using
doDrain() instead of drain() gets past this problem.

-Tony

On Thu, Jul 12, 2012 at 9:14 AM, Ali Saidi <saidi(a)umich.edu> wrote: