Flow Control - Advanced Deep Dive

Credit Algorithms, Initialization, Deadlock Prevention

Home → Transaction Layer → Flow Control Advanced

Credit-Based Flow Control Fundamentals

PCIe flow control ensures a transmitter never sends more data than the receiver can buffer. Unlike XON/XOFF or request-grant schemes, credit-based flow control allows continuous transmission as long as credits are available, maximizing link utilization.

The Credit Invariant

At any point in time, the following must hold:

CREDITS_CONSUMED ≤ CREDIT_LIMIT

Where:

  • CREDITS_CONSUMED: Total credits used by transmitter (modulo counter)
  • CREDIT_LIMIT: Maximum credits advertised by receiver via UpdateFC

Credit Counter Mechanics

Both transmitter and receiver maintain 8-bit modulo counters for each credit type. The modulo arithmetic handles wraparound correctly.

// Transmitter Credit Check Algorithm function can_transmit_tlp(tlp): hdr_credits = 1 // All TLPs consume 1 header credit data_credits = ceil(tlp.payload_size / 16) // Check based on TLP type if tlp.type == POSTED: hdr_available = (PH_LIMIT - PH_CONSUMED) mod 256 data_available = (PD_LIMIT - PD_CONSUMED) mod 4096 elif tlp.type == NON_POSTED: hdr_available = (NPH_LIMIT - NPH_CONSUMED) mod 256 data_available = (NPD_LIMIT - NPD_CONSUMED) mod 4096 elif tlp.type == COMPLETION: hdr_available = (CplH_LIMIT - CplH_CONSUMED) mod 256 data_available = (CplD_LIMIT - CplD_CONSUMED) mod 4096 return (hdr_credits <= hdr_available) and (data_credits <= data_available) // After successful transmission function consume_credits(tlp): hdr_credits = 1 data_credits = ceil(tlp.payload_size / 16) if tlp.type == POSTED: PH_CONSUMED = (PH_CONSUMED + hdr_credits) mod 256 PD_CONSUMED = (PD_CONSUMED + data_credits) mod 4096 // ... similar for NP and Cpl

Flow Control Initialization Protocol

Before any TLPs can be transmitted, both sides must exchange initial credit values through a two-phase DLLP handshake.

FC Initialization Sequence Port A Port B InitFC1 (PH, PD, NPH, NPD, CplH, CplD) InitFC1 (PH, PD, NPH, NPD, CplH, CplD) InitFC2 (confirms receipt) InitFC2 (confirms receipt) FC_INIT2 FC_INIT2

InitFC DLLP Format

ByteFieldDescription
0TypeInitFC1-P, InitFC1-NP, InitFC1-Cpl, InitFC2-P, etc.
1HdrFC[7:0]Header credits (lower 8 bits)
2DataFC[3:0], HdrFC[11:8]Data credits lower + Header upper
3DataFC[11:4]Data credits (upper 8 bits)

Infinite Credits

A receiver advertises "infinite" credits by setting the credit value to 0 during initialization. This means the transmitter never needs to check credits for that type.

When to Use Infinite Credits

  • Completion Credits: Completers typically advertise infinite CplH/CplD because completions come from internal processing, not buffer space
  • Root Complex: May advertise infinite credits to memory since system memory is "unlimited"
  • Non-Posted Data: Often infinite since NP requests rarely carry data

UpdateFC DLLP Scheduling

After initialization, receivers must periodically send UpdateFC DLLPs to replenish transmitter credits.

// Receiver UpdateFC Scheduling Algorithm define UPDATE_THRESHOLD = advertised_credits / 2 // Or implementation-specific function check_update_needed(): for each credit_type in [PH, PD, NPH, NPD, CplH, CplD]: credits_freed = CREDITS_ALLOCATED - LAST_ADVERTISED if credits_freed >= UPDATE_THRESHOLD: send_updatefc_dllp(credit_type, CREDITS_ALLOCATED) LAST_ADVERTISED = CREDITS_ALLOCATED // UpdateFC must also be sent periodically to handle potential losses // Typical interval: 30μs to 200μs (implementation dependent)

Deadlock Prevention

PCIe's flow control design inherently prevents deadlock through careful separation of transaction types.

Deadlock Prevention Rules

  1. Completion Independence: Completions have dedicated credit pool. A Completer can always send completions regardless of request credit state.
  2. Posted vs Non-Posted: Posted requests cannot be blocked waiting for Non-Posted credits, and vice versa.
  3. Switch Guarantee: Switches must always be able to forward completions, even if request buffers are full.
  4. UpdateFC Always Sent: DLLPs (including UpdateFC) are not flow-controlled - they can always be transmitted.

Potential Deadlock Scenario (Prevented by Design)

Scenario: Device A sends MRd to Device B. Device B's buffer is full of requests to Device A. Device A's completion buffer is full.

Why it doesn't deadlock: Completion credits are separate. Even if Device A's request buffers are full, its completion buffer space is independently tracked. Device B can send the completion as long as CplH/CplD credits are available.

Flow Control in Flit Mode

PCIe 6.0+ Flit Mode changes flow control significantly:

Flit Mode Credits = Number of 256-byte Flit buffers available
Next: Ordering Rules →