Virtualization Overview
PCIe provides extensive virtualization support for cloud and enterprise environments:
Key Virtualization Technologies
- SR-IOV: Single Root I/O Virtualization
- MR-IOV: Multi-Root I/O Virtualization
- ATS: Address Translation Services
- ARI: Alternative Routing-ID Interpretation
Single Root I/O Virtualization (SR-IOV)
SR-IOV allows a single physical device to present multiple virtual instances to the system:
Key Concepts
- Physical Function (PF): Full PCIe function with SR-IOV capability
- Virtual Function (VF): Lightweight function sharing resources with PF
- VFs have their own BARs, MSI-X, and configuration space
- Up to 256 VFs per PF
| Feature | Physical Function | Virtual Function |
|---|---|---|
| Configuration | Full Type 0 | Limited |
| BARs | 6 BARs | Up to 6 VF BARs |
| Interrupts | INTx, MSI, MSI-X | MSI-X only |
| Control | Full device control | Data path only |
| Direct VM Assignment | Hypervisor | Yes |
Multi-Root I/O Virtualization (MR-IOV)
MR-IOV extends SR-IOV for multi-host sharing (e.g., blade servers):
- Single device shared across multiple root complexes
- Requires MR-IOV capable switch
- Each host sees independent virtual hierarchies
Alternative Routing-ID Interpretation (ARI)
ARI extends function numbering beyond 8 functions per device:
- Standard: 8-bit Bus + 5-bit Device + 3-bit Function = 8 functions max
- ARI: 8-bit Bus + 8-bit Function = 256 functions per bus
- Essential for SR-IOV with many VFs
Address Translation Services (ATS)
ATS enables devices to cache address translations from IOMMU:
ATS Flow
- Device sends Translation Request to Root Complex
- Root Complex queries IOMMU for translation
- Translation Completion returned to device
- Device caches translation in ATC (Address Translation Cache)
- Future DMAs use translated addresses directly
Benefits
- Reduced latency for repeated DMA to same pages
- Offloads IOMMU translation overhead
- Essential for high-performance virtualized I/O
Page Request Interface (PRI)
PRI allows devices to request pages that are not present in memory:
- Works with demand-paged virtual memory
- Device sends Page Request when translation fails
- OS faults in the page and notifies device
- Enables true shared virtual memory (SVM)
Process Address Space ID (PASID)
PASID enables devices to access multiple process address spaces:
- 20-bit PASID in TLP prefix
- Each PASID maps to different address space
- Enables direct device access from user-space applications
- Foundation for heterogeneous computing (GPU, accelerators)