PASID (Process Address Space ID)

Deep-Dive: Per-Process Address Space Isolation for Virtualization and Shared Virtual Memory

1. PASID Overview

Process Address Space ID (PASID) is a mechanism that allows PCIe devices to target specific process address spaces within a system. This is critical for Shared Virtual Memory (SVM), where devices can directly access process-specific virtual addresses.

Why PASID?

  • Process Isolation: Device can access multiple process address spaces without conflicts
  • Shared Virtual Memory: Device uses same virtual addresses as CPU process
  • GPU/Accelerator Efficiency: Direct access to process memory without copying
  • SR-IOV with SVM: VFs can target guest process address spaces
  • Scalable I/O Virtualization (SIOV): Fine-grained device sharing

PASID in the PCIe/IOMMU Ecosystem

┌─────────────────────────────────────────────────────────────────────────────┐ │ System with PASID │ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Process A │ │ Process B │ │ Process C │ (User Space) │ │ │ PASID = 0x10 │ │ PASID = 0x20 │ │ PASID = 0x30 │ │ │ │ VA Space │ │ VA Space │ │ VA Space │ │ │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ │ │ │ │ │ │ └─────────┬─────────┴─────────┬─────────┘ │ │ │ │ │ │ ┌─────────▼─────────┐ │ │ │ │ IOMMU │◄────────┘ (PASID → Page Table lookup) │ │ │ (with PASID TLB) │ │ │ └─────────┬─────────┘ │ │ │ │ │ ┌─────────▼─────────┐ │ │ │ Root Complex │ │ │ └─────────┬─────────┘ │ │ │ │ │ ┌─────────────┼─────────────┐ │ │ │ │ │ │ │ ┌──▼──┐ ┌──▼──┐ ┌──▼──┐ │ │ │GPU │ │NVMe │ │NIC │ (Devices sending TLPs with PASID) │ │ │ │ │ │ │ │ │ │ └─────┘ └─────┘ └─────┘ │ │ │ │ GPU issues MWr with PASID=0x10 → IOMMU looks up Process A's page table │ │ NIC issues MRd with PASID=0x20 → IOMMU looks up Process B's page table │ └─────────────────────────────────────────────────────────────────────────────┘

2. PASID TLP Format

2.1 Non-Flit Mode: PASID TLP Prefix

PASID TLP Prefix Format (Non-Flit Mode): In Non-Flit Mode, PASID is carried as an End-End TLP Prefix. TLP Prefix Header (4 bytes): ┌─────────────────────────────────────────────────────────────────────────┐ │ Bit │ 31:29 │ 28:25 │ 24 │ 23 │ 22:20 │ 19:0 │ ├─────┼─────────┼───────┼──────┼──────┼───────┼───────────────────────────┤ │ │ Fmt │ Type │ (R) │ PMR │ ER │ PASID │ │ │ 001 │ 0100 │ 0 │ │ │ (20-bit value) │ └─────────────────────────────────────────────────────────────────────────┘ Fmt[2:0] = 001b (TLP Prefix) Type[3:0] = 0100b (End-End TLP Prefix, PASID) Field Definitions: PASID [19:0]: Process Address Space Identifier (20 bits = 1M spaces) PMR (bit 23): Privileged Mode Requested ER (bits 22:20): Execute Requested (3 bits for future use) TLP Structure with PASID Prefix: ┌─────────────────────────────────────┐ │ PASID TLP Prefix (4B) │ ← End-End Prefix ├─────────────────────────────────────┤ │ TLP Header (3-4 DW) │ ← Normal TLP header ├─────────────────────────────────────┤ │ Data Payload (optional) │ ← TLP data ├─────────────────────────────────────┤ │ ECRC (optional, 4B) │ ← Covers prefix + TLP └─────────────────────────────────────┘

2.2 Flit Mode: PASID Field

PASID in Flit Mode TLP Headers: In Flit Mode, PASID is integrated directly into the TLP header format. Flit Mode Memory Request Header with PASID: ┌─────────────────────────────────────────────────────────────────────────┐ │ Byte │ 7 │ 6 │ 5 │ 4 │ 3 │ 2 │ 1 │ 0 │ ├──────┼─────────┴─────────┴─────────┴─────────┼─────────┴─────┴─────┴────┤ │ 0-7 │ Address [63:0] │ Req ID + Tag │ ├──────┼───────────────────────────────────────┼──────────────────────────┤ │ 8-15 │ PASID[19:0] │ PMR │ ER │ Attr │ etc │ Length + Type │ └──────┴───────────────────────────────────────┴──────────────────────────┘ PASID Valid Bit: In Flit Mode, a "PASID Valid" bit indicates whether the PASID field contains a valid value for this transaction. PASID Field Encoding: Bits [19:0]: PASID value (0x00000 to 0xFFFFF) Bit [23]: Privileged Mode Requested (PMR) 0 = User mode access 1 = Privileged/supervisor mode access Bits [22:20]: Execute Requested (ER) Used for instruction fetch indication

2.3 PASID Usage in Different TLP Types

TLP Type PASID Allowed? Usage
Memory Read (MRd) Yes Device reading from process VA
Memory Write (MWr) Yes Device writing to process VA
AtomicOp Yes Atomic to process VA
Completion No (echoed) PASID from request
Configuration No Not applicable
Message Limited Some message types only

3. Shared Virtual Memory (SVM)

3.1 SVM Architecture with PASID

SVM Page Fault Flow: 1. Device issues MRd with PASID=0x10, VA=0x7FFF_0000 2. IOMMU looks up PASID-specific page table 3. Page not present → IOMMU signals page fault 4. OS handles fault: - Allocates physical page - Updates process page table - Updates IOMMU page table for PASID=0x10 - Signals IOMMU to retry 5. IOMMU completes translation 6. Device receives completion with data ┌─────────────────────────────────────────────────────────────────────────────┐ │ SVM Page Fault Sequence │ │ │ │ Device IOMMU Kernel Process │ │ │ │ │ │ │ │ │ MRd VA=0x7FFF │ │ │ │ │ │ PASID=0x10 │ │ │ │ │ │──────────────────►│ │ │ │ │ │ │ │ │ │ │ │ │ Page Fault IRQ │ │ │ │ │ │──────────────────►│ │ │ │ │ │ │ │ │ │ │ │ │ Alloc page │ │ │ │ │ │──────────────────►│ │ │ │ │ │ │ │ │ │ │ │ Update PT │ │ │ │ │◄──────────────────│ │ │ │ │ │ (PASID=0x10 PT) │ │ │ │ │ │ │ │ │ │ │ │ Translation OK │ │ │ │ │ │──────────────────►│ │ │ │ │ │ │ │ │ │ │ Completion │ │ │ │ │ │◄──────────────────│ │ │ │ │ │ │ │ │ │ └─────────────────────────────────────────────────────────────────────────────┘

3.2 IOMMU PASID Table Structure

Intel VT-d PASID Table (Scalable Mode): PASID Directory (First Level): ┌─────────────────────────────────────────────────────────────────┐ │ Entry 0 │ → PASID Table 0 (PASID 0x00000-0x001FF) │ │ Entry 1 │ → PASID Table 1 (PASID 0x00200-0x003FF) │ │ Entry 2 │ → PASID Table 2 (PASID 0x00400-0x005FF) │ │ ... │ │ │ Entry N │ → PASID Table N (PASID 0xFFE00-0xFFFFF) │ └─────────────────────────────────────────────────────────────────┘ PASID Table Entry (64 bytes): ┌─────────────────────────────────────────────────────────────────┐ │ Offset │ Field │ ├────────┼─────────────────────────────────────────────────────────┤ │ 0x00 │ Present, Flags, Domain ID │ │ 0x08 │ First-Level Page Table Pointer (process page table) │ │ 0x10 │ Second-Level Page Table Pointer (guest physical) │ │ 0x18 │ PASID-specific flags (Execute-Enable, etc.) │ │ 0x20 │ Reserved │ │ 0x28 │ Reserved │ │ 0x30 │ Reserved │ │ 0x38 │ Reserved │ └─────────────────────────────────────────────────────────────────┘ AMD IOMMU PASID Table: Similar structure with GCR3 tables pointing to guest page tables per-PASID. Uses PASID-indexed table for translation context.

3.3 PASID with SR-IOV

SR-IOV VF with PASID Support: ┌───────────────────────────────────────┐ │ Physical Function │ │ PASID Extended Capability │ │ ┌─────────────────────────────────┐ │ │ │ Max PASID Width: 20 bits │ │ │ │ Execute Permission Supported │ │ │ │ Privileged Mode Supported │ │ │ └─────────────────────────────────┘ │ └───────────────────┬───────────────────┘ │ ┌───────────────────────────┼───────────────────────────┐ │ │ │ ┌─────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐ │ VF 0 │ │ VF 1 │ │ VF 2 │ │ │ │ │ │ │ │ Assigned │ │ Assigned │ │ Assigned │ │ to VM1 │ │ to VM2 │ │ to VM3 │ │ │ │ │ │ │ │ Can use │ │ Can use │ │ Can use │ │ PASID │ │ PASID │ │ PASID │ │ 0x100-1FF │ │ 0x200-2FF │ │ 0x300-3FF │ └───────────┘ └───────────┘ └───────────┘ Each VF can target different guest process address spaces using distinct PASID values, enabling per-process virtualization.

4. PASID Extended Capability Registers

4.1 Capability Structure

PASID Extended Capability (ID = 0x001B): Offset Size Register Name ────── ──── ───────────────────────────── 00h 4 PASID Extended Capability Header 04h 2 PASID Capability Register 06h 2 PASID Control Register PASID Capability Register (Offset 04h): ┌───────────────────────────────────────────────────────────────────┐ │ Bit │ Field │ Description │ ├─────┼───────────────────────────┼────────────────────────────────┤ │ 0 │ Reserved │ │ │ 1 │ Execute Permission Supp │ Device supports ER field │ │ 2 │ Privileged Mode Supported │ Device supports PMR field │ │ 12:8│ Max PASID Width │ Log2 of supported PASID values │ │ │ │ (e.g., 20 = 2^20 = 1M PASIDs) │ │15:13│ Reserved │ │ └───────────────────────────────────────────────────────────────────┘ PASID Control Register (Offset 06h): ┌───────────────────────────────────────────────────────────────────┐ │ Bit │ Field │ Description │ ├─────┼───────────────────────────┼────────────────────────────────┤ │ 0 │ PASID Enable │ 1 = Device may use PASID │ │ 1 │ Execute Permission Enable │ 1 = ER field usage enabled │ │ 2 │ Privileged Mode Enable │ 1 = PMR field usage enabled │ │15:3 │ Reserved │ │ └───────────────────────────────────────────────────────────────────┘

4.2 Software Configuration

Enabling PASID on a Device: enable_pasid(device): // Step 1: Find PASID Extended Capability pasid_cap_offset = find_ext_capability(device, 0x001B) if pasid_cap_offset == 0: return ERROR_NOT_SUPPORTED // Step 2: Read capabilities pasid_cap = config_read_word(device, pasid_cap_offset + 0x04) max_pasid_width = (pasid_cap >> 8) & 0x1F supports_execute = (pasid_cap >> 1) & 0x01 supports_priv = (pasid_cap >> 2) & 0x01 // Step 3: Configure IOMMU for PASID granular translation iommu_enable_pasid(device.bdf, max_pasid_width) // Step 4: Enable PASID on device pasid_ctrl = 0x0001 // PASID Enable if supports_execute AND need_execute_permission: pasid_ctrl |= 0x0002 if supports_priv AND need_priv_mode: pasid_ctrl |= 0x0004 config_write_word(device, pasid_cap_offset + 0x06, pasid_ctrl) return SUCCESS

5. Normative Rules

PASID Implementation Rules (from PCIe Spec)

  1. R1: PASID MUST be 20 bits, supporting up to 2^20 address spaces.
  2. R2: PASID TLP Prefix MUST be an End-End TLP Prefix (not modified by switches).
  3. R3: PMR bit MUST only be set if Privileged Mode is enabled in Control register.
  4. R4: ER field MUST only be used if Execute Permission is enabled.
  5. R5: Completions MUST NOT include PASID TLP Prefix (PASID context from request).
  6. R6: IOMMU MUST use PASID + Requester ID for translation lookup.
  7. R7: Devices MUST NOT use PASID until PASID Enable bit is set.
  8. R8: Root Complex MUST route PASID-tagged TLPs to IOMMU for translation.
  9. R9: Page faults on PASID-tagged requests MUST identify the specific PASID.
  10. R10: Software MUST ensure PASID values are unique per-process within a device context.