Physical Layer

// PCIe Protocol Stack ┌─────────────────────────────────────────────────────────────────┐ │ SOFTWARE/CONFIGURATION │ │ Device Discovery, Enumeration, BAR Configuration, Drivers │ │ (Uses Configuration TLPs) │ ├─────────────────────────────────────────────────────────────────┤ │ TRANSACTION LAYER │ │ TLP Generation/Consumption, Flow Control, Ordering │ ├─────────────────────────────────────────────────────────────────┤ │ DATA LINK LAYER │ │ LCRC, Sequence Numbers, ACK/NAK, Retry, DLLPs │ ├─────────────────────────────────────────────────────────────────┤ │ PHYSICAL LAYER ← YOU ARE HERE │ │ ┌───────────────────────────────────────────────────────────┐ │ │ │ LOGICAL SUB-BLOCK: │ │ │ │ Encoding, Scrambling, Framing, Ordered Sets, LTSSM │ │ │ └───────────────────────────────────────────────────────────┘ │ │ ┌───────────────────────────────────────────────────────────┐ │ │ │ ELECTRICAL SUB-BLOCK: │ │ │ │ Tx Driver, Rx CDR, Equalization, Impedance, Signaling │ │ │ └───────────────────────────────────────────────────────────┘ │ ├─────────────────────────────────────────────────────────────────┤ │ PHYSICAL MEDIUM │ │ PCB Traces, Connectors, Cables, Retimers │ └─────────────────────────────────────────────────────────────────┘

// 8b/10b Encoding Overview Purpose: - Ensure DC balance (equal number of 1s and 0s over time) - Provide special control characters (K-codes) - Enable clock recovery (sufficient transitions) Mechanism: - 8 data bits encoded to 10 symbol bits - 20% overhead (only 80% efficiency) - Two versions of each symbol: RD+ and RD- (Running Disparity) - Transmitter alternates to maintain DC balance Special K-Codes (Control Characters): ┌────────────┬────────────┬──────────────────────────────────┐ │ Symbol │ Encoding │ Purpose │ ├────────────┼────────────┼──────────────────────────────────┤ │ K28.5 (COM)│ 001111_1010│ Comma - byte/symbol alignment │ │ K28.0 (SKP)│ 001111_0100│ Skip - clock compensation │ │ K28.1 (FTS)│ 001111_1001│ Fast Training Sequence │ │ K28.2 (SDP)│ 001111_0101│ Start DLLP │ │ K28.3 (IDL)│ 001111_0011│ Idle │ │ K28.4 (--) │ 001111_0010│ Reserved │ │ K28.6 (--) │ 001111_0110│ Reserved │ │ K28.7 (EIE)│ 001111_1000│ Electrical Idle Exit │ │ K27.7 (STP)│ 110110_1000│ Start TLP │ │ K29.7 (END)│ 101110_1000│ End Good │ │ K30.7 (EDB)│ 011110_1000│ End Bad (nullified TLP) │ │ K23.7 (PAD)│ 111010_1000│ Padding │ └────────────┴────────────┴──────────────────────────────────┘ Running Disparity (RD): RD+ = More 1s than 0s sent recently RD- = More 0s than 1s sent recently Symbol chosen to move RD toward neutral Disparity error = link error indication

// 128b/130b Block Encoding Structure: ┌────────────┬────────────────────────────────────────────────────┐ │ Sync Header│ Payload (128 bits) │ │ (2 bits) │ 16 bytes of data │ └────────────┴────────────────────────────────────────────────────┘ Sync Header Values: 01 = Data Block (contains data symbols) 10 = Ordered Set Block (contains OS) 00, 11 = Invalid (used for error detection) Efficiency: 128/130 = 98.46% No Running Disparity: - DC balance achieved through scrambling - LFSR scrambler provides pseudo-random bit distribution - Sync headers NOT scrambled Block Alignment: - Receiver uses sync header pattern to find block boundaries - 01/10 alternation helps alignment - EIEOS used for initial alignment

// TLP Framing (8b/10b and 128b/130b) 8b/10b Framing (Gen1-2): ┌─────┬────────────────────────────────────────┬─────┐ │ STP │ TLP │ END │ │K27.7│ Header + Data + ECRC (DLL adds Seq+LCRC) │K29.7│ └─────┴────────────────────────────────────────┴─────┘ STP = Start TLP (K27.7) END = End Good (K29.7) or EDB = End Bad (K30.7) DLLP Framing (Gen1-2): ┌─────┬──────────────────┬─────┐ │ SDP │ DLLP │ END │ │K28.2│ 4 bytes + CRC16 │K29.7│ └─────┴──────────────────┴─────┘ 128b/130b Framing (Gen3-5): Data blocks (sync=01) can contain: - STP token (4-bit): Start of TLP - SDP token (4-bit): Start of DLLP - END token (4-bit): End Good - EDB token (4-bit): End Bad (nullified) Tokens embedded in data stream with 4-bit framing codes

// Flit Mode Framing (256-byte fixed Flits) ┌────────────────────────────────────────────────────────────────────┐ │ 256-Byte Flit │ ├────────────┬──────────────────────────────────┬──────────┬─────────┤ │ Flit Header│ TLP Payload Area │ DLP │FEC+CRC │ │ (2 bytes) │ (236 bytes) │(12 bytes)│(6 bytes)│ └────────────┴──────────────────────────────────┴──────────┴─────────┘ Flit Header: - Flit Type (Payload/NOP/IDE) - Flit Sequence Number (8-bit) - First TLP Byte Offset TLP Payload Area: - Can contain multiple small TLPs - Can contain partial large TLP (continued in next Flit) - TLP packed efficiently with OHC (Optimized Header Compression) DLP (Data Link Payload): - ACK/NAK information - Flow control credits - Replaces separate DLLPs FEC + CRC: - 6 bytes Reed-Solomon FEC - Enables error correction without retry

// LFSR Scrambler Polynomials Gen3-5 (128b/130b): Polynomial: G(X) = X^23 + X^21 + X^16 + X^8 + X^5 + X^2 + 1 Initial seed: FFFFFFh Scrambler runs continuously Sync headers (2 bits) NOT scrambled Data payload (128 bits) XORed with LFSR output Scrambler Reset: - Reset at start of each Ordered Set Block - COM symbol in 8b/10b resets scrambler - Allows receiver to resynchronize Compliance Pattern: Special unscrambled pattern for testing Entered via Polling.Compliance state

// TS1 Ordered Set Structure (16 symbols) Symbol 0: COM (K28.5) - Comma for alignment Symbol 1: Link Number - PAD (K23.7) = not yet assigned Symbol 2: Lane Number - PAD (K23.7) = not yet assigned Symbol 3: N_FTS - Number of FTS required for L0s exit Symbol 4: Data Rate ID - Supported/selected data rate Symbol 5: Training Ctrl - Control bits (Hot Reset, Loopback, etc.) Symbol 6-9: EQ Ctrl - Equalization settings (Gen3+) Symbol 10-13: TS1 ID - D10.2 pattern identifies TS1 Symbol 14-15: Reserved // TS2 Ordered Set Structure (16 symbols) Same as TS1 except: Symbol 10-13: TS2 ID - D5.2 pattern identifies TS2 Key Fields: Data Rate ID (Symbol 4): Bit 0: 2.5 GT/s supported Bit 1: 5.0 GT/s supported Bit 2: 8.0 GT/s supported Bit 3: Autonomous Change (speed change without Re-config) Bits 4-6: Speed being trained to Bit 7: Crosslink supported Training Control (Symbol 5): Bit 0: Hot Reset Bit 1: Disable Link Bit 2: Loopback Bit 3: Disable Scrambling Bit 4: Compliance Receive Bits 5-7: Reserved

// SKP (Skip) Ordered Set Purpose: Compensate for clock frequency differences between Tx and Rx Tx and Rx clocks can differ by up to ±300 ppm 8b/10b SKP (Gen1-2): COM + SKP + SKP + SKP (4 symbols) Receiver can add or remove SKP symbols as needed 128b/130b SKP (Gen3-5): Ordered Set Block containing SKP pattern SKP_OS contains identifier + SKP symbols Scheduling (8b/10b): SKP must be sent every 1180-1538 symbol times ~1.2μs interval at 2.5 GT/s Scheduling (128b/130b): SKP_OS sent every ~370 blocks Interval: 370 × 130 bits ÷ data rate Receiver Behavior: - Elastic buffer absorbs timing differences - Add SKP if buffer too empty - Remove SKP if buffer too full - SKP removal/addition transparent to upper layers

// EIEOS (Electrical Idle Exit Ordered Set) Pattern: Alternating 00h and FFh bytes (creates high-frequency tone) Length: 16 symbols Used to: Re-establish bit lock after Electrical Idle Precedes: TS1 when exiting low-power state // EIOS (Electrical Idle Ordered Set) 8b/10b: COM + IDL + IDL + IDL (4 symbols) Signals: Transmitter entering Electrical Idle Followed by: Transmitter drives Electrical Idle level // FTS (Fast Training Sequence) 8b/10b: COM + FTS + FTS + FTS (4 symbols) Used for: Quick L0s exit (minimal re-training) N_FTS field in TS1/TS2 indicates how many needed // SDS (Start of Data Stream) Indicates: Training complete, data transmission beginning 128b/130b: Special SDS pattern in OS block Followed by: First data block

Detect.Quiet: - Initial state after reset/power-on - Transmitter drives Electrical Idle - Wait for internal timer (12ms timeout) - Or triggered by wake event Detect.Active: - Apply voltage change to Tx output - Measure impedance/current to detect receiver termination - Receiver Detected: - Impedance indicates ~50Ω termination present - Transition to Polling - No Receiver: - Return to Detect.Quiet - Exponential backoff (up to 1 second) // Detection thresholds vary by speed/implementation // Must detect receiver within specific current range

Polling.Active: - Transmit TS1 Ordered Sets continuously - Receiver attempting bit lock (CDR acquisition) - Look for valid TS1 or TS2 on receive - Exit to Polling.Configuration: Received 8 TS1/TS2 Polling.Configuration: - Continue transmitting TS1 - Start transmitting TS2 after receiving TS1/TS2 - Exit to Configuration: - Received 8 consecutive TS2 with: - Lane = any lane this port received TS1/TS2 on - Link = PAD (not assigned) - Timeout: 24ms → Detect Polling.Compliance: - Special test/debug state - Transmit Compliance Pattern - Entered if: - Enter Compliance bit set in received TS1 - Or compliance mode strapping - Exit by entering Electrical Idle Polling.Speed: - Speed change negotiation - Exchange TS1/TS2 with speed bits - Prepare for new speed operation

Configuration.Linkwidth.Start: Downstream Port (Root/Switch DSP): - Transmit TS1 with Link# = N (proposed) - Wait for Upstream to accept Upstream Port: - Wait for TS1 with Link# ≠ PAD - Accept proposed configuration Configuration.Linkwidth.Accept: - Negotiate which lanes are active - Handle lane reversal if needed - Agree on link width (x1, x2, x4, x8, x16, x32) Configuration.Lanenum.Wait: - Receive TS1 with Lane numbers assigned - Verify lane numbering consistency Configuration.Lanenum.Accept: - Accept lane number assignment - Transmit TS2 with accepted lane numbers Configuration.Complete: - Both ends transmit TS2 - After receiving 8 TS2 with: - Link# match - Lane# match - Proceed to Configuration.Idle Configuration.Idle: - Transmit Logical Idle (or SDS for 128b/130b) - Enter L0 (normal operation)

Recovery.RcvrLock: - Re-acquire bit lock - Transmit TS1 Ordered Sets - Exit when TS1/TS2 received or timeout Recovery.RcvrCfg: - Re-verify configuration - Similar to Configuration states - Shorter timeouts than initial Configuration Recovery.Equalization: (Gen3+) - Perform link equalization - Phase 0: Downstream Tx with preset, Upstream evaluates - Phase 1: Downstream requests Upstream Tx changes - Phase 2: Upstream Tx with settings, Downstream evaluates - Phase 3: Upstream requests Downstream Tx changes Recovery.Speed: - Handle speed change - Entered when Changed_Speed_Recovery bit set - Change to new speed, then Recovery.RcvrLock Recovery.Idle: - Final state before returning to L0 - Verify link is stable - Transmit Idle then enter L0

L0s (Standby): - Fast entry/exit (~100ns exit latency) - Per-lane power savings - Tx in Electrical Idle - Exit via FTS Ordered Sets - ASPM controlled L1 (Low Power): - Deeper power savings - Longer exit latency (~2-32μs) - Common clock can be stopped - Sub-states: L1.0, L1.1, L1.2 (L1.2 lowest power) - Exit via EIEOS → Recovery L2 (Auxiliary Power): - Near-off state - Only wake signaling active - Main power can be removed - Exit requires full re-training (Detect) - Used for system suspend states

Disabled: - Link intentionally disabled - Entered via software or TS1 with Disable bit - Transmitter in Electrical Idle - Exit to Detect when re-enabled Hot Reset: - In-band reset mechanism - Entered via TS1 with Hot Reset bit - Causes device reset without power cycle - Downstream sends TS1 with Hot Reset - Upstream enters Hot Reset state - After reset complete → Detect

// Equalization Components Transmitter (Tx) Equalization: 3-tap FIR filter: Output = C(-1) × D(n+1) + C(0) × D(n) + C(+1) × D(n-1) C(-1): Pre-cursor (affects next bit) - boosts high frequency C(0): Main cursor (current bit amplitude) C(+1): Post-cursor (affects previous bit) - compensates ISI Receiver (Rx) Equalization: CTLE (Continuous Time Linear Equalizer): - Analog high-pass filter - Boosts high-frequency components - Compensates for channel roll-off DFE (Decision Feedback Equalizer): - Digital filter using previous bit decisions - Cancels post-cursor ISI - 1-5 taps typical - Non-linear, doesn't amplify noise

// Gen3+ Equalization Phases (in Recovery.Equalization) Phase 0: - Downstream Port transmits with initial preset - Upstream Port performs Rx equalization - Evaluate eye quality Phase 1: - Downstream Port sends EQ requests to Upstream Tx - "Increase pre-cursor" / "Decrease post-cursor" etc. - Upstream adjusts Tx EQ settings Phase 2: - Upstream Port transmits with its settings - Downstream Port evaluates and adjusts Rx EQ Phase 3: - Upstream Port sends EQ requests to Downstream Tx - Downstream adjusts settings - Final optimization Presets (P0-P10): Pre-defined coefficient combinations Receiver can request specific preset Speeds up convergence ┌─────────┬──────────┬───────────┬────────────┐ │ Preset │ C(-1) │ C(0) │ C(+1) │ ├─────────┼──────────┼───────────┼────────────┤ │ P0 │ 0 dB │ -6.0 dB │ 0 dB │ │ P1 │ 0 dB │ -3.5 dB │ -3.5 dB │ │ ... │ ... │ ... │ ... │ │ P10 │ -1.9 dB │ 0 dB │ -3.0 dB │ └─────────┴──────────┴───────────┴────────────┘

// Lane Deskew Problem: Different lane lengths cause arrival time differences Data must be aligned across all lanes for parallel processing Solution: - Use symbol lock (COM detection) per lane - Elastic buffers per lane - Align all lanes to slowest lane - SKP insertion/removal for continuous alignment Max Skew: Specified per generation Gen3+: ~20ns maximum lane-to-lane skew

// Clock Tolerance and Compensation Clock Tolerance: Tx and Rx reference clocks can differ by ±300 ppm Without compensation, buffers would overflow/underflow Mechanism: SKP Ordered Sets: - Periodically transmitted in L0 - Receiver can add or remove SKP symbols - Absorbs clock frequency difference Elastic Buffer: - FIFO between CDR and decoder - Nominal half-full - SKP added when buffer too empty - SKP removed when buffer too full SKP Scheduling: 8b/10b: Every 1180-1538 symbols 128b/130b: Every ~370 blocks (depends on implementation) Flit Mode: SKP_OS in Flit stream Calculation: Clock diff = 300 ppm = 0.03% At 8 GT/s: ~2400 symbols/second drift SKP every ~1300 symbols compensates this

// Retimer Architecture ┌─────────┐ ┌─────────────┐ ┌─────────┐ │ Root │◄──────────────►│ Retimer │◄──────────────►│ Device │ │ Complex │ Link Segment │ │ Link Segment │ │ └─────────┘ 1 └─────────────┘ 2 └─────────┘ Retimer Functions: - Full CDR (Clock Data Recovery) - Data retiming with local clock - Equalization (Rx and Tx) - LTSSM participation - Protocol-aware (understands Ordered Sets) Key Points: - Up to 2 retimers per link (3 segments total) - Each retimer adds ~5-10ns latency - Transparent to software/upper layers - Has own LTSSM (pseudo-port) - Passes through TLPs/DLLPs without modification Training: - Participates in equalization - Each segment optimized independently - Upstream and Downstream independently trained Use Cases: - CEM cables (server interconnect) - Long PCB traces (>12 inches) - Backplanes - U.2/U.3 storage connections