UIO (Unordered I/O) Complete Deep-Dive

1. What is UIO (Unordered I/O)?

Technical Definition

UIO provides a mechanism where:

• UIO Memory Writes (UIOMWr) are Non-Posted operations that receive Completions
• UIO Memory Reads (UIOMRd) work similarly to traditional reads but can complete out-of-order
• The Requester explicitly handles ordering requirements, removing ordering burden from the fabric
• UIO requires Flit Mode operation (PCIe 6.0+ at 64 GT/s or 128 GT/s)
• UIO requires 14-bit Tags for proper transaction tracking

Key Characteristics

2. Why was UIO Introduced?

Problems with Traditional PCIe Ordering

1. Producer-Consumer Model Complexity

Traditional PCIe enforces strict ordering to support the producer-consumer programming model:

• Memory Writes are Posted (no completion, fire-and-forget)
• Memory Reads return data in order relative to writes
• This ordering creates dependencies that the fabric must track
• Switches and Root Complexes require complex ordering logic

2. Performance Limitations

• Ordering rules can create head-of-line blocking
• Writes cannot be acknowledged until they reach destination
• Complex reordering buffers needed in switches
• Multi-Virtual Channel (VC) configurations become complex

3. Modern Workload Requirements

• Many workloads (especially CXL memory) don't need producer-consumer ordering
• Device drivers can manage ordering explicitly when needed
• High-bandwidth memory operations benefit from relaxed ordering

UIO Benefits

3. When to Use UIO?

Ideal Use Cases

1. CXL Memory Accesses

• CXL.mem devices accessing host memory
• High-bandwidth memory pooling operations
• Memory tiering systems
• Coherent memory accesses where ordering is managed by coherency protocol

2. GPU and Accelerator Traffic

• Bulk data transfers where ordering isn't required
• Scatter-gather operations
• Independent memory region accesses

3. High-Performance Storage

• NVMe command completions to different queues
• Independent I/O operations
• Storage virtualization

When NOT to Use UIO

4. Where Does UIO Apply?

System Architecture Requirements

┌─────────────────────────────────────────────────────────────────────┐
│                         ROOT COMPLEX                                │
│  ┌──────────────┐                                                   │
│  │  Root Port   │◄── UIO Capability Required                        │
│  │  (UIO-aware) │                                                   │
│  └──────┬───────┘                                                   │
│         │ Flit Mode Link (64/128 GT/s)                              │
└─────────┼───────────────────────────────────────────────────────────┘
          │
          ▼
   ┌──────────────┐
   │    Switch    │◄── UIO forwarding capability required
   │  (UIO-aware) │    (all ports in the path)
   └──────┬───────┘
          │ Flit Mode Link (64/128 GT/s)
          ▼
   ┌──────────────┐
   │   Endpoint   │◄── UIO Requester or Completer capability
   │  (UIO-aware) │    14-bit Tag support required
   └──────────────┘

   ═══════════════════════════════════════════════════════════════════
   KEY: All components in UIO transaction path MUST support UIO
        All links MUST operate in Flit Mode
        All components MUST support 14-bit Tags for UIO

Topology Constraints

End-to-End Requirement

UIO requires that every component in the transaction path supports UIO:

• Requester: Must have UIO Requester capability
• Switches: Must forward UIO transactions (all ports in path)
• Root Complex: Must support UIO at Root Port
• Completer: Must support UIO Completer capability

Virtual Channel Mapping

UIO uses dedicated Virtual Channels for traffic segregation:

• UIO traffic is mapped to specific UIO VCs
• UIO VCs have separate flow control from non-UIO VCs
• This prevents UIO traffic from affecting traditional ordered traffic

5. How Does UIO Work?

UIO Memory Write Flow

   Requester                    Completer
      │                            │
      │ 1. UIOMWr Request          │
      │ ──────────────────────────►│
      │   (14-bit Tag, Address,    │
      │    Length, Data)           │
      │                            │
      │                            │ 2. Write data to memory
      │                            │
      │ 3. UIOWrCpl Completion     │
      │ ◄──────────────────────────│
      │   (Tag, Completion Status, │
      │    Length)                 │
      │                            │
      │ 4. Requester matches Tag   │
      │    Mark transaction done   │
      │                            │

   NOTES:
   - Unlike traditional MWr, UIOMWr receives a completion
   - Completion indicates write has reached destination
   - Multiple UIOMWr with same Tag allowed (coalesced completion)
   - Length in completion = DWs completed (for accounting)

UIO Memory Read Flow

   Requester                    Completer
      │                            │
      │ 1. UIOMRd Request          │
      │ ──────────────────────────►│
      │   (14-bit Tag, Address,    │
      │    Length)                 │
      │                            │
      │                            │ 2. Read data from memory
      │                            │
      │ 3. UIORdCplD Completion    │
      │ ◄──────────────────────────│
      │   (Tag, Lower Address,     │
      │    Length, Data)           │
      │                            │
      │ 4. Match Tag, extract data │
      │                            │

   SPLIT COMPLETION EXAMPLE:
   ┌────────────────────────────────────────┐
   │ Original UIOMRd: 1024 bytes            │
   │ Completion 1: 256 bytes (LA=0x000)     │
   │ Completion 3: 512 bytes (LA=0x200)     │ ← Out of order!
   │ Completion 2: 256 bytes (LA=0x100)     │
   │ Total: 1024 bytes (transaction done)   │
   └────────────────────────────────────────┘

Transaction Identification

UIO transactions are identified by a Transaction ID consisting of:

• Requester ID (16 bits): Bus/Device/Function of Requester
• Tag (14 bits): Unique identifier for transaction (mandatory for UIO)

6. UIO TLP Types

UIO defines specific TLP Types that are only valid in Flit Mode:

Type Field Encoding

In Flit Mode, the Type[7:0] field fully identifies the TLP type:

• Bits [7:5]: Format (presence of header DWs, data payload)
• Bits [4:0]: Specific transaction type
• UIO types are distinguished from non-UIO by specific bit patterns

7. UIO Header Formats

UIOMWr Header (UIO Memory Write Request)

UIO Completion Headers

UIOWrCpl and UIORdCpl (Without Data)

UIORdCplD (Read Completion with Data)

Includes all fields above plus:

• Lower Address[11:2]: Starting byte address for returned data
• Data Payload: Requested data (Length DWs)

8. UIO Completion Rules

General Completion Rules

Read Completion Boundary (RCB) Rules

• UIO Read Completions follow the same RCB rules as non-UIO reads
• Split completions must align to RCB boundaries (64 or 128 bytes)
• Lower Address must correctly indicate starting byte of data

Write Completion Rules

• UIO Write Completions CAN be coalesced for the same Transaction ID
• UIO Write Completions CAN be split for the same Transaction ID
• All DW Completion accounting must remain accurate

Completion Status Handling

When UIO Completions with different statuses are coalesced:

Transaction Completion Determination

A UIO transaction is considered complete when:

9. UIO Ordering Model

No Ordering Enforcement

Unlike traditional PCIe where the fabric enforces producer-consumer ordering, UIO explicitly removes this requirement:

Requester Responsibilities

When using UIO, the Requester takes on ordering responsibilities:

• Explicit Fencing: Use completion waiting to create ordering points
• Tag Management: Track which transactions are complete via Tags
• Application Ordering: Software must manage any needed ordering

Ordering with Non-UIO Traffic

10. UIO Flow Control

UIO Virtual Channels

UIO uses dedicated Virtual Channels with their own flow control credits:

• UIO VC: Dedicated VC for UIO traffic
• Separate Credit Pools: UIO has its own P, NP, and Cpl credits
• No Interference: UIO credits don't affect non-UIO flow control

Credit Types for UIO

Flit Mode Credit Blocks

In Flit Mode, flow control uses credit blocks instead of individual credits:

• Credit Block = 64 bytes (regardless of actual TLP size)
• UIO VC has dedicated/shared credit block allocation
• See Flit Mode Flow Control deep-dive for details

11. UIO and 14-Bit Tags

Mandatory 14-Bit Tags

UIO requires 14-bit Tags for several reasons:

• Large Outstanding Count: UIO may have many concurrent transactions
• Tag Reuse for Writes: Multiple UIOMWr can share Tag (with DW accounting)
• Flit Mode Alignment: 14-bit Tags are part of Flit Mode header format

Tag Field Location

12. System-Level Applications

CXL Memory Pooling

Primary use case for UIO is CXL memory systems:

• CXL.mem protocol uses UIO for memory accesses
• High-bandwidth memory pooling across multiple hosts
• Memory disaggregation architectures
• Coherent memory access without PCIe ordering overhead

High-Performance Computing

• RDMA-style operations where ordering is managed by protocol
• GPU direct memory access
• Accelerator-to-accelerator communication

Storage Systems

• NVMe controller implementations
• High-IOPS storage where command independence is guaranteed
• Storage virtualization systems

Implementation Considerations

13. Normative Rules Summary

14. UIO vs Traditional Posted/Non-Posted