Flow Control Complete Deep-Dive

1. Flow Control Fundamentals

What is Flow Control?

Flow control is a mechanism that prevents buffer overflow by ensuring a transmitter does not send more TLPs than the receiver can accept. PCIe uses a credit-based flow control system where:

Receiver advertises available buffer space (credits)
Transmitter consumes credits when sending TLPs
Receiver returns credits after processing TLPs
Transmitter cannot send if insufficient credits

Why Credit-Based Flow Control?

No retransmission: Unlike ACK/NAK, flow control prevents overflow entirely
Decoupled from latency: High latency links work efficiently
Per-VC isolation: Each Virtual Channel has independent credits
Hardware efficient: Simple counter-based implementation

Credit Types

Credit Type	Abbreviation	Description	TLP Types
Posted Header	PH	Credits for Posted request headers	MWr, Msg, MsgD
Posted Data	PD	Credits for Posted request data (4 DW units)	MWr data, MsgD data
Non-Posted Header	NPH	Credits for Non-Posted request headers	MRd, CfgRd, CfgWr, IORd, IOWr
Non-Posted Data	NPD	Credits for Non-Posted request data	CfgWr data, IOWr data
Completion Header	CplH	Credits for Completion headers	Cpl, CplD
Completion Data	CplD	Credits for Completion data	CplD data

2. Credit Accounting Equations

Credit Counter Variables

Transmitter Side Counters


CREDITS_CONSUMED[type] = Total credits used by transmitted TLPs

CREDIT_LIMIT[type] = Last received credit limit from receiver

CREDITS_AVAILABLE[type] = CREDIT_LIMIT[type] - CREDITS_CONSUMED[type]

Receiver Side Counters


CREDITS_ALLOCATED[type] = Total credits provided to transmitter

CREDITS_RECEIVED[type] = Credits for TLPs received and processed

CREDITS_TO_RETURN = CREDITS_RECEIVED[type] - CREDITS_ADVERTISED[type]

Credit Consumption Rules

Header Credit Consumption


For each TLP transmitted:
  Header_Credits_Consumed = 1 (one header credit per TLP)

Data Credit Consumption (Non-Flit Mode)


Data_Credits_Consumed = CEIL(Payload_Size_Bytes / 16)

Where:
  - Each data credit = 4 DW = 16 bytes
  - CEIL rounds up to nearest integer
  - Example: 100 byte payload = CEIL(100/16) = 7 credits

Transmission Gate Check

Can Transmit Check Algorithm


CanTransmit(TLP) = 
  (CREDITS_AVAILABLE[header_type] >= 1) AND
  (CREDITS_AVAILABLE[data_type] >= Data_Credits_Required(TLP))

Where header_type and data_type depend on TLP type:
  - Posted Request: PH, PD
  - Non-Posted Request: NPH, NPD
  - Completion: CplH, CplD

Modular Arithmetic for Credit Comparison

Credit counters use modular arithmetic with specific bit widths:

Credit Type	Counter Width	Modulus
Header Credits (PH, NPH, CplH)	8 bits	256
Data Credits (PD, NPD, CplD)	12 bits	4096

Credit Comparison (Modular)


Credits_Available = (Credit_Limit - Credits_Consumed) MOD 2^N

Where N = 8 for headers, 12 for data

Transmit allowed if Credits_Available > 0
(accounting for wraparound)

3. Flow Control Initialization Protocol

Two-Phase Initialization

Flow control initialization uses a two-phase handshake:

   Port A                                      Port B
     │                                           │
     │──────── InitFC1 (P) ──────────────────────►
     │◄─────── InitFC1 (P) ─────────────────────│
     │──────── InitFC1 (NP) ─────────────────────►
     │◄─────── InitFC1 (NP) ────────────────────│
     │──────── InitFC1 (Cpl) ────────────────────►
     │◄─────── InitFC1 (Cpl) ───────────────────│
     │                                           │
     │         ═══ Phase 1 Complete ═══          │
     │                                           │
     │──────── InitFC2 (P) ──────────────────────►
     │◄─────── InitFC2 (P) ─────────────────────│
     │──────── InitFC2 (NP) ─────────────────────►
     │◄─────── InitFC2 (NP) ────────────────────│
     │──────── InitFC2 (Cpl) ────────────────────►
     │◄─────── InitFC2 (Cpl) ───────────────────│
     │                                           │
     │         ═══ Phase 2 Complete ═══          │
     │                                           │
     │        Flow Control Initialized           │

InitFC DLLP Format

Field	Bits	Description
Type	[7:0]	DLLP Type (InitFC1 or InitFC2)
VC ID	[2:0]	Virtual Channel identifier
HdrFC	[7:0]	Header credits (0-255)
DataFC	[11:0]	Data credits (0-4095)

Initialization Requirements

Flow Control Initialization Rules

VC0 MUST complete initialization before any other VC
Each VC MUST complete P, NP, and Cpl initialization
InitFC1 MUST be sent for each credit type/VC combination
InitFC2 MUST be sent after receiving all InitFC1 for that VC
TLP transmission is blocked until initialization completes

4. UpdateFC DLLPs

Credit Return Mechanism

After initialization, UpdateFC DLLPs return credits to transmitters:

   Transmitter                              Receiver
        │                                      │
        │────── TLP (consumes credits) ───────►│
        │                                      │ Process TLP
        │                                      │ Free buffer space
        │◄──────── UpdateFC ──────────────────│
        │      (returns credits)               │
        │                                      │

UpdateFC Timing

UpdateFC must be sent before 30μs timeout to avoid errors
UpdateFC should be sent when significant credits are available
UpdateFC can be sent proactively or reactively
At least one UpdateFC per credit type every 30μs if link active

UpdateFC Scheduling

UpdateFC Trigger Conditions


Send UpdateFC when ANY of:
  1. Credits_To_Return >= Threshold (implementation specific)
  2. Timer expires (< 30μs since last UpdateFC)
  3. Credit limit has changed
  4. Recovery from low-credit situation

5. Infinite Credits

Infinite Credit Advertisement

A receiver can advertise "infinite" credits to indicate unlimited buffer space:

Credit Type	Infinite Value	Meaning
Header (HdrFC)	0x00	Unlimited header credits
Data (DataFC)	0x000	Unlimited data credits

Infinite Credit Rules

Infinite credits are advertised during InitFC only
Once infinite is advertised, UpdateFC for that type is not needed
Transmitter does not track consumed credits for infinite type
Root Complex completion credits are often infinite

6. Flit Mode Flow Control

Credit Block Model

Flit Mode uses a different credit model based on 64-byte credit blocks:

Flit Mode Credit Unit


Credit_Block_Size = 64 bytes (fixed)

Credits_Consumed = CEIL(TLP_Size / 64)

Where TLP_Size includes header + data + any OHC

Shared vs Dedicated Credits

Flit Mode supports both shared and dedicated credit pools:

Credit Type	Description	Use Case
Shared Credits	Single pool for all TLP types	Default, simpler management
Dedicated Credits	Separate pools per type	Guaranteed bandwidth

Flit Mode Local TLP Prefix

The Flit Mode Local TLP Prefix indicates dedicated credit usage:

Bit 0: TLP Uses Dedicated Credits
When Set: TLP must use dedicated credits
When Clear: TLP uses shared credits

Flit Mode Credit Equations

Credit Calculation for Flit Mode


TLP_Blocks = CEIL((Header_Size + Payload_Size + OHC_Size) / 64)

For Shared Credits:
  Shared_Credits_Consumed += TLP_Blocks

For Dedicated Credits:
  Dedicated_Credits_Consumed[type] += TLP_Blocks
  Where type = P, NP, or Cpl

7. Deadlock Prevention

Why Deadlock Can Occur

PCIe can deadlock if completion credits are exhausted while non-posted requests are pending:

  DEADLOCK SCENARIO:
  
  Device A                          Device B
     │                                 │
     │──── NP Request ────────────────►│  B receives request
     │                                 │  B needs to send Cpl
     │                                 │  B has no Cpl credits to A!
     │                                 │
     │◄─── (cannot send Cpl) ──────────│
     │                                 │
     │  A waiting for Cpl              │  B waiting for Cpl credits
     │  A cannot free Cpl credits      │  
     │         === DEADLOCK ===        │

Prevention Mechanisms

Deadlock Prevention Rules

Reserved Completion Credits: Always reserve minimum completion credits
Infinite Completion Credits: Root Complexes typically offer infinite
Separate VC for Completions: Isolate completion traffic
Credit Starvation Timeout: Detect and recover from low credits

Minimum Credit Requirements

Minimum Credits to Prevent Deadlock


Min_CplH_Credits >= Max_Outstanding_NP_Requests
Min_CplD_Credits >= Max_NP_Request_Size * Max_Outstanding_NP_Requests

Where Max_NP_Request_Size depends on Max_Read_Request_Size

8. Flow Control State Machine

Per-VC Flow Control States

               ┌─────────────┐
               │    IDLE     │
               │  (Reset)    │
               └──────┬──────┘
                      │ DL_Up
                      ▼
               ┌─────────────┐
               │   INIT_FC1  │◄──────────┐
               │  (Phase 1)  │           │
               └──────┬──────┘           │
                      │ All InitFC1      │ Timeout/Error
                      │ Received         │
                      ▼                  │
               ┌─────────────┐           │
               │   INIT_FC2  │───────────┘
               │  (Phase 2)  │
               └──────┬──────┘
                      │ All InitFC2
                      │ Received
                      ▼
               ┌─────────────┐
               │  FC_ACTIVE  │◄──────────┐
               │  (Normal)   │           │
               └──────┬──────┴───────────┘
                      │                  │
                      │ UpdateFC         │ Link
                      │ Exchange         │ Active
                      └──────────────────┘

9. Credit Calculation Examples

Example 1: Memory Write TLP

256-byte MWr Credit Consumption


TLP: MWr with 256 bytes data payload

Non-Flit Mode:
  PH Credits = 1 (one header)
  PD Credits = CEIL(256 / 16) = 16 credits

Flit Mode (Shared):
  Header = 16 bytes (4 DW)
  Payload = 256 bytes
  Total = 272 bytes
  Credit Blocks = CEIL(272 / 64) = 5 blocks

Example 2: Completion with Data

512-byte CplD Credit Consumption


TLP: CplD with 512 bytes data

Non-Flit Mode:
  CplH Credits = 1
  CplD Credits = CEIL(512 / 16) = 32 credits

Flit Mode:
  Header = 12 bytes (3 DW)
  Payload = 512 bytes
  Total = 524 bytes
  Credit Blocks = CEIL(524 / 64) = 9 blocks