Enabling bus mastering/BMDMA
Greetings,
I am building a custom full system configuration (largely influenced by ARM in configs/FSConfig.py) and am running into an issue where the kernel hangs on boot, being unable to access the disk image to mount the root filesystem. Looking at the kernel boot logs and gem5 call trace, I strongly suspect the culprit has to do with this:
[ 0.000005] ata_piix 0000:00:01.0: BMDMA: BAR4 is zero, falling back to PIO
In other words, while a standard ARM FS mode simulation will enable bus mastering on the simulated gem5 (PIIX) IDE controller and use DMA to access the disk, my simulation reverts to PIO instead. The PCI node of my device tree is the one from ARM’s vexpress (aarch64) dtb. My PCI host/IDE controller setup is also similar to ARM’s setup:
self.cf0 = CowIdeDisk(driveID='master')
self.cf0.childImage(mdesc.disk())
self.pci_host = GenericPciHost(conf_base=0x30000000, conf_size=’256MB’, conf_device_bits=12, pci_pio_base=0x2f000000)
self.pci_ide = IdeController(disks=[self.cf0], pci_dev=1, pci_bus=0)
pci_devices.append(self.pci_ide)
…
self.pci_host.pio = self.iobus.master
for dev_id, dev in enumerate(pci_devices):
dev.host = self.pci_host
dev.pio = self.iobus.master
dev.dma = self.iobus.slave
The gem5 call trace includes the following (no methods from CowDiskImage are called):
IdeController::Channel::accessCommand: offset = 6, size = 1, read = 0
IdeDisk::updateState: action = 4.
IdeDisk::writeCommand: Write to disk at offset: 0x6 data 0xa0
IdeController::dispatchAccess: Write from offset: 0x2f000016 size: 0x1 data: 0xa0
IdeDisk::readControl: Read to disk at offset: 0x2 data 0x40
IdeController::dispatchAccess: Read from offset: 0x2f000022 size: 0x1 data: 0x40
IdeDisk::writeControl: Write to disk at offset: 0x2 data 0xa
IdeController::dispatchAccess: Write from offset: 0x2f000022 size: 0x1 data: 0xa
I have already verified that my kernel includes drivers for the PIIX, legacy ATA controllers, and BMDMA. I have also attempted to change the BAR4 value of the IDE controller directly in the system configuration file and update the device tree with a dma-ranges field for the PCI node. None of these solutions worked.
TL;DR: How do I ensure that my full system configuration supports and enables bus mastering/BMDMA for the IDE controller? Is there anything special I need to add to the device tree binary?
Best regards,
Joshua Klein
Greetings all,
I wanted to post a follow-up and ask another question in case anyone has any insight. So I learned that the third bit of the Command field in the PCI config can be set to explicitly say bus mastering is available. The layout and meaning of the Command register can be found here: https://wiki.osdev.org/PCI
Even though I can set the Command value now, the same aforementioned error is present. Linux assigns BAR4 to zero, and therefore BMDMA is still not utilized.
If I keep the default BARX values of the IdeController (BARX=0x1), the kernel will report during boot:
[ 0.000002] pci 0000:00:02.0: [8086:7111] type 00 class 0x010185
[ 0.000002] pci 0000:00:02.0: reg 0x10: [io 0x0000-0x0007]
[ 0.000002] pci 0000:00:02.0: reg 0x14: [io 0x0000-0x0003]
[ 0.000002] pci 0000:00:02.0: reg 0x18: [io 0x0000-0x0007]
[ 0.000002] pci 0000:00:02.0: reg 0x1c: [io 0x0000-0x0003]
[ 0.000002] pci 0000:00:02.0: reg 0x20: [io 0x0000-0x000f]
[ 0.000002] pci 0000:00:02.0: reg 0x30: [mem 0x00000000-0x000007ff pref]
[ 0.000002] pci 0000:00:01.0: BAR 0: assigned [mem 0x40000000-0x4001ffff]
[ 0.000002] pci 0000:00:01.0: BAR 6: assigned [mem 0x40020000-0x400207ff pref]
[ 0.000002] pci 0000:00:02.0: BAR 6: assigned [mem 0x40020800-0x40020fff pref]
[ 0.000002] pci 0000:00:02.0: BAR 4: assigned [io 0x0000-0x000f]
[ 0.000002] pci 0000:00:02.0: BAR 0: assigned [io 0x0010-0x0017]
[ 0.000002] pci 0000:00:02.0: BAR 2: assigned [io 0x0018-0x001f]
[ 0.000002] pci 0000:00:02.0: BAR 1: assigned [io 0x0020-0x0023]
[ 0.000002] pci 0000:00:02.0: BAR 3: assigned [io 0x0024-0x0027]
However, if I set the following in the IdeController (all values +1 to preserve IO assignment):
BAR0=0x1011
BAR1=0x1021
BAR2=0x1019
BAR3=0x1025
BAR4=0x1001
The kernel will report this during boot:
[ 0.000002] pci 0000:00:02.0: [8086:7111] type 00 class 0x010185
[ 0.000002] pci 0000:00:02.0: reg 0x10: [io 0x1010-0x1017]
[ 0.000002] pci 0000:00:02.0: reg 0x14: [io 0x1020-0x1023]
[ 0.000002] pci 0000:00:02.0: reg 0x18: [io 0x1018-0x101f]
[ 0.000002] pci 0000:00:02.0: reg 0x1c: [io 0x1024-0x1027]
[ 0.000002] pci 0000:00:02.0: reg 0x20: [io 0x1000-0x100f]
[ 0.000002] pci 0000:00:02.0: reg 0x30: [mem 0x00000000-0x000007ff pref]
[ 0.000003] pci 0000:00:01.0: BAR 0: assigned [mem 0x40000000-0x4001ffff]
[ 0.000003] pci 0000:00:01.0: BAR 6: assigned [mem 0x40020000-0x400207ff pref]
[ 0.000003] pci 0000:00:02.0: BAR 6: assigned [mem 0x40020800-0x40020fff pref]
[ 0.000003] pci 0000:00:02.0: BAR 4: assigned [io 0x0000-0x000f]
[ 0.000003] pci 0000:00:02.0: BAR 0: assigned [io 0x0010-0x0017]
[ 0.000003] pci 0000:00:02.0: BAR 2: assigned [io 0x0018-0x001f]
[ 0.000003] pci 0000:00:02.0: BAR 1: assigned [io 0x0020-0x0023]
[ 0.000003] pci 0000:00:02.0: BAR 3: assigned [io 0x0024-0x0027]
You can see that even though the registers of the PCI configuration are read to have changed, the Linux BAR assignment remains unchanged. So now my question is, how can I convince Linux to change the BAR assignment?
Best regards,
Joshua Klein
From: gem5-users <gem5-users-bounces(a)gem5.org> on behalf of Klein Joshua Alexander Harrison <joshua.klein(a)epfl.ch>
Sent: Wednesday, November 13, 2019 2:09:18 PM
To: gem5-users(a)gem5.org
Subject: [gem5-users] Enabling bus mastering/BMDMA
Greetings,
I am building a custom full system configuration (largely influenced by ARM in configs/FSConfig.py) and am running into an issue where the kernel hangs on boot, being unable to access the disk image to mount the root filesystem. Looking at the kernel boot logs and gem5 call trace, I strongly suspect the culprit has to do with this:
[ 0.000005] ata_piix 0000:00:01.0: BMDMA: BAR4 is zero, falling back to PIO
In other words, while a standard ARM FS mode simulation will enable bus mastering on the simulated gem5 (PIIX) IDE controller and use DMA to access the disk, my simulation reverts to PIO instead. The PCI node of my device tree is the one from ARM’s vexpress (aarch64) dtb. My PCI host/IDE controller setup is also similar to ARM’s setup:
self.cf0 = CowIdeDisk(driveID='master')
self.cf0.childImage(mdesc.disk())
self.pci_host = GenericPciHost(conf_base=0x30000000, conf_size=’256MB’, conf_device_bits=12, pci_pio_base=0x2f000000)
self.pci_ide = IdeController(disks=[self.cf0], pci_dev=1, pci_bus=0)
pci_devices.append(self.pci_ide)
…
self.pci_host.pio = self.iobus.master
for dev_id, dev in enumerate(pci_devices):
dev.host = self.pci_host
dev.pio = self.iobus.master
dev.dma = self.iobus.slave
The gem5 call trace includes the following (no methods from CowDiskImage are called):
IdeController::Channel::accessCommand: offset = 6, size = 1, read = 0
IdeDisk::updateState: action = 4.
IdeDisk::writeCommand: Write to disk at offset: 0x6 data 0xa0
IdeController::dispatchAccess: Write from offset: 0x2f000016 size: 0x1 data: 0xa0
IdeDisk::readControl: Read to disk at offset: 0x2 data 0x40
IdeController::dispatchAccess: Read from offset: 0x2f000022 size: 0x1 data: 0x40
IdeDisk::writeControl: Write to disk at offset: 0x2 data 0xa
IdeController::dispatchAccess: Write from offset: 0x2f000022 size: 0x1 data: 0xa
I have already verified that my kernel includes drivers for the PIIX, legacy ATA controllers, and BMDMA. I have also attempted to change the BAR4 value of the IDE controller directly in the system configuration file and update the device tree with a dma-ranges field for the PCI node. None of these solutions worked.
TL;DR: How do I ensure that my full system configuration supports and enables bus mastering/BMDMA for the IDE controller? Is there anything special I need to add to the device tree binary?
Best regards,
Joshua Klein