PCIe configuration access mechanisms, topology discovery, and ECAM memory mapping
Configuration Space is a standardized memory region in every PCIe function that contains device identification, control registers, capabilities, and extended capabilities. It enables software discovery and configuration of devices.
Configuration Space (4KB per function):
┌──────────────────────────────────────────────────────────────┐
│ Offset 000h-03Fh: PCI-Compatible Header (64 bytes) │
│ ├── Type 0: Endpoints │
│ └── Type 1: Bridges/Switches │
├──────────────────────────────────────────────────────────────┤
│ Offset 040h-0FFh: PCI Capabilities (192 bytes) │
│ ├── Power Management │
│ ├── MSI/MSI-X │
│ ├── PCIe Capability │
│ └── Others... │
├──────────────────────────────────────────────────────────────┤
│ Offset 100h-FFFh: PCIe Extended Capabilities (3840 bytes) │
│ ├── AER (Advanced Error Reporting) │
│ ├── SR-IOV │
│ ├── ACS │
│ └── Many more... │
└──────────────────────────────────────────────────────────────┘
| Method | Offsets | Mechanism |
|---|---|---|
| PCI Compatible (Legacy) | 00h-FFh (256 bytes) | I/O ports CF8h/CFCh |
| ECAM (Enhanced) | 00h-FFFh (4KB) | Memory-mapped |
Configuration Address Port (CF8h):
┌─────────────────────────────────────────────────────────────────┐
│ 31 │ 30:24 │ 23:16 │ 15:11 │ 10:8 │ 7:2 │ 1:0 │
│Enable│ Rsvd │ Bus │ Device │ Function │ Reg │ 00 │
└─────────────────────────────────────────────────────────────────┘
Access:
1. Write address to CF8h
2. Read/Write data at CFCh
Limitation: Only accesses first 256 bytes (offsets 00h-FFh)
ECAM memory-maps the entire configuration space (4KB per function) into system address space, enabling efficient access to all configuration registers including extended capabilities.
ECAM Memory Layout:
ECAM_Base + (Bus × 2^20) + (Device × 2^15) + (Function × 2^12) + Offset
Or equivalently:
ECAM_Base + (Bus << 20) + (Device << 15) + (Function << 12) + Offset
Example:
ECAM_Base = 0xE000_0000
Bus = 5, Device = 0, Function = 2, Offset = 0x100
Address = 0xE000_0000 + (5 × 0x100000) + (0 × 0x8000) + (2 × 0x1000) + 0x100
= 0xE000_0000 + 0x500000 + 0 + 0x2000 + 0x100
= 0xE050_2100
| Buses | ECAM Size | Formula |
|---|---|---|
| 1 | 1 MB | 1 × 32 × 8 × 4KB = 1MB |
| 256 | 256 MB | 256 × 32 × 8 × 4KB = 256MB |
enumerate_bus(bus_num):
for device in 0..31:
for function in 0..7:
if read_vendor_id(bus, device, function) != 0xFFFF:
configure_device(bus, device, function)
if is_bridge(bus, device, function):
secondary_bus = assign_bus_number()
set_secondary_bus(bus, device, function, secondary_bus)
enumerate_bus(secondary_bus)
if function == 0 and not is_multifunction():
break # Skip remaining functions
Type 1 Header Bus Number Registers:
┌──────────────────────────────────────────────────────────────┐
│ Primary Bus Number (Offset 18h) │
│ Bus number of upstream port │
├──────────────────────────────────────────────────────────────┤
│ Secondary Bus Number (Offset 19h) │
│ Bus number immediately downstream │
├──────────────────────────────────────────────────────────────┤
│ Subordinate Bus Number (Offset 1Ah) │
│ Highest bus number behind this bridge │
└──────────────────────────────────────────────────────────────┘
Example:
Root Complex (Bus 0)
│
┌─────┴─────┐
│ Switch │ Pri=0, Sec=1, Sub=5
│ (Bus 1) │
└─────┬─────┘
┌─────────┼─────────┐
│ │ │
DSP (2-2) DSP (3-3) DSP (4-5)
│ │ │
GPU NVMe ┌───┴───┐
│ Bridge│ Pri=4, Sec=5, Sub=5
└───┬───┘
│
Device
| Type | Usage | Addressing |
|---|---|---|
| Type 0 (CfgRd0/CfgWr0) | Target is directly connected | Device, Function, Register |
| Type 1 (CfgRd1/CfgWr1) | Target is behind bridge | Bus, Device, Function, Register |
Bridges convert Type 1 to Type 0 when target bus equals secondary bus:
Incoming Type 1 CfgRd (Bus=5, Dev=0, Func=0)
│
▼
Bridge (Pri=4, Sec=5, Sub=5)
Target Bus (5) == Secondary Bus (5)?
YES → Convert to Type 0 and send downstream
Root Complex integrated endpoints (RCiEPs) may use RCRB for extended capabilities:
The MCFG table in ACPI provides ECAM base addresses:
MCFG Table Entry:
┌──────────────────────────────────────────────────────────────┐
│ Base Address (64-bit): ECAM region start │
│ Segment Group Number (16-bit): PCI segment │
│ Start Bus Number (8-bit): First bus in range │
│ End Bus Number (8-bit): Last bus in range │
└──────────────────────────────────────────────────────────────┘
Linux: /sys/firmware/acpi/tables/MCFG
Device may return CRS (Completion Status = 010) if not ready: