Linux Topics (4.2) - opROM Bar attribute confuse of PCIe device

目录

1. Confuse

2. PCIe Config Space

2.1 BAR Introduction

2.2 BAR Attributes

3. BAR Initialization in Linux

3.1 Initialization Flow

3.2 pci_read_bases(）

---

1. Confuse

原因是查看linux kernel boot log的时候， 发现一个SSD 设备的opROM bar 属性被标志为pref,

 pci 0001:01:00.0: [144d:a80a] type 00 class 0x010802
 pci 0001:01:00.0: reg 0x10: [mem 0x740004110000-0x740004113fff 64bit]
 pci 0001:01:00.0: reg 0x30: [mem 0x740004100000-0x74000410ffff pref]

但是我记得该设备的BAR不是prefetchable， 到了OS 下， lspci 看了下，也没有标志prefetchable

#lspci -vvs 0001:01:00.0

0001:01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO (prog-if 02 [NVM Express])
        Region 0: Memory at 740004010000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at 740004000000 [disabled] [size=64K]

Dump 一下配置空间，也证明了 opROM BAR prefetch bit 没有置位：

00: 4d 14 0a a8 06 04 10 00 00 02 08 01 10 00 00 00
10: 04 00 01 04 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 4d 14 12 a8
30: 00 00 10 04 40 00 00 00 00 00 00 00 ff 01 00 00

但是查看了 /sys/devices 下面的设备资源信息，却是跟boot log 一致，标识opROM BAR为prefetchable。

#define IORESOURCE_PREFETCH    0x00002000    /* No side effects */
cat /sys/devices/pci0001\:00/0001\:00\:01.0/0001\:01\:00.0/resource
0x0000740004010000 0x0000740004013fff 0x0000000000140204
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000740004000000 0x000074000400ffff 0x0000000000046200  //ROM BAR, Pref Bit= 0x2000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000

那opROM BAR 如何初始化的呢？ 为何要这样标识呢？首先要查查代码

---

2. PCIe Config Space

2.1 BAR Introduction

Config space 的BARs  如果下图

• Type0 PCIe device的 config space, 共有 6 个BAR(0x10 – 0x24)， 一个ROM BAR(0x30).
• Type1 PCIe bridge 的 config space, 共有 2个BAR(0x10 – 0x14)， 一个ROM BAR(0x38).

ROM Bar 主要用来在BIOS 初始化的时候， 先在Pcie enumeration的时候分配合适的地址，

用来关联该device的 opROM driver(UEFI 时候， 包的就是UEFI driver）

对于BIOS 来说， 

• BIOS 把某个PCIE device的 opROM driver 直接包在代码里面，可以override 设备自带的较久的opROM driver, 这种设备是设计的时候已经设计在主板上了。
• 对于外置add-in card, 一般都是BIOS 初始化的是时候， 获取card 本身自带的opROM driver 来初始化。

2.2 BAR Attributes

重点看一下 bit 3, 用来标识是否 Prefetchable.

3. BAR Initialization in Linux

3.1 Initialization Flow

acpi_pci_root_add()

        pci_acpi_scan_root()

                root_ops->release_info = pci_acpi_generic_release_info;

                root_ops->prepare_resources = pci_acpi_root_prepare_resources;

                acpi_pci_root_create()

                      |-->ops->prepare_resources(info);

                      |      |-->pci_acpi_root_prepare_resources;     //Get resource from ACPI(_CRS..)

                      |--> pci_acpi_root_add_resources(info);

                      |-->pci_create_root_bus()

                      |      |-->pci_register_host_bridge()

                      |-->pci_scan_child_bus(bus);

                      |           pci_scan_child_bus_extend(bus, 0);

                      |                      pci_scan_slot(bus, devfn);

                      |                            pci_scan_single_device()

                      |                                    |-->pci_scan_device(bus, devfn); //扫描slot，并创建pci dev

                      |                                    |           |-->pci_setup_device()

                      |                                    |                   |-->pci_read_bases()

3.2 pci_read_bases(）

static void pci_read_bases(struct pci_dev *dev, unsigned int howmany, int rom)

{

    unsigned int pos, reg;

    if (dev->non_compliant_bars)         return;

    /* Per PCIe r4.0, sec 9.3.4.1.11, the VF BARs are all RO Zero */

    if (dev->is_virtfn)       return;

    for (pos = 0; pos < howmany; pos++) {

       struct resource *res = &dev->resource[pos];

       reg = PCI_BASE_ADDRESS_0 + (pos << 2);

       pos += __pci_read_base(dev, pci_bar_unknown, res, reg);

    }

    if (rom) {

       struct resource *res = &dev->resource[PCI_ROM_RESOURCE];

       dev->rom_base_reg = rom;

       res->flags = IORESOURCE_MEM | IORESOURCE_PREFETCH |  //why opROM BAR masked as prefetch?

              IORESOURCE_READONLY | IORESOURCE_SIZEALIGN;

       __pci_read_base(dev, pci_bar_mem32, res, rom);

    }

}

代码里面已经很明显了，如果是opROM bar, 那么就要标志为 IORESOURCE_PREFETCH,

虽然 prefetchable memory(MMIO)有它的好处，

Prefetchable memory is memory which the CPU can request in advance as an optimization, before actual code operates on it. This happens because the CPU guesses it may be needed in the future or because the code contains explicit prefetch instructions. Then it prefetches those memory locations while processing some other instructions or doing some other work.

但是一般来说，都是根据BAR 自己的属性定义来配置才对。

查下了 Linux kernel git repo, 在 Linus 开始 2.6 版本的时候，就已经是这样了， 虽然之后由于side-effect, 取消了该设定，但是后来又revert了。

至于最初肯定是出于性能目的这样设置，但是暂时找不到最初的记录，真是好奇害死猫，留个TBD吧。

commit bb44609361fe87c5e136c2b8dfde59bcbdbabf61

Author: Gary Hade <garyhade@us.ibm.com>

Date:   Tue Dec 11 17:09:13 2007 -0800

    PCI: Restore PCI expansion ROM P2P prefetch window creation

    Restore PCI expansion ROM P2P prefetch window creation.

    This patch reverts previous "Avoid creating P2P prefetch

    window for expansion ROMs" change due to regressions that

    were spotted on some systems.

Signed-off-by: Gary Hade <garyhade@us.ibm.com>

Signed-off-by: Greg Kroah-Hartman gregkh@suse.de

commit fd64cb4606cbdd592b7119e82341d4ae5b56f2cc

Author: Gary Hade <garyhade@us.ibm.com>

Date:   Wed Oct 3 15:56:30 2007 -0700

    PCI: avoid P2P prefetch window for expansion ROMs

    Avoid creating P2P prefetch window for expansion ROMs

    Because of the future possibility that P2P prefetch windows will contain

    address ranges above 4GB some BIOSes are providing space in the P2P

    non-prefetch windows for expansion ROMs.  This is due to expansion ROM

    BAR 32-bit limitation.  When expansion ROM BARs without BIOS assigned

    address(es) are currently found behind a P2P bridge, the kernel attempts

    to create a P2P prefetch window for them even though space for them has

    already been provided in the non-prefetch window.  _CRS on some systems

    with certain resource conservation conscious BIOSes may not provide the

    extra 1MB or more memory resource needed for the expansion ROM motivated

    prefetch window causing resource allocation errors.

    This change corrects the problem by removing IORESOURCE_PREFETCH from

    the expansion ROM flags initialization.  It also removes

    IORESOURCE_CACHEABLE which seems inappropriate if only non-cacheable

    memory is available.

    Signed-off-by: Gary Hade <gary.hade@us.ibm.com>

    Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>