MSI & MSI-X Complete Guide

1. What are MSI and MSI-X?

What is Message Signaled Interrupts?

MSI (Message Signaled Interrupts) and MSI-X are interrupt mechanisms where devices signal interrupts by writing to a special memory address, rather than using dedicated interrupt lines. This provides more efficient and scalable interrupt delivery.

Evolution of PCIe Interrupts

Type	Mechanism	Vectors	Sharing
INTx (Legacy)	Virtual wire (Assert_INTx message)	4 (INTA-INTD)	Shared
MSI	Memory write TLP	1-32	Not shared
MSI-X	Memory write TLP	1-2048	Not shared

2. Why MSI/MSI-X?

Why use Message Signaled Interrupts?

MSI/MSI-X provides significant advantages over legacy INTx interrupts including better performance, scalability, and CPU affinity control.

Advantages over INTx

No Sharing: Each vector dedicated to one interrupt source
No ISR Overhead: No need to query device for interrupt source
Ordering: Interrupt arrives after data (memory write ordering)
CPU Affinity: Can direct interrupts to specific CPUs
Scalability: MSI-X supports up to 2048 vectors
Per-Queue Interrupts: NVMe, network queues get dedicated vectors

MSI vs MSI-X Comparison

Feature	MSI	MSI-X
Max Vectors	32	2048
Address per Vector	Shared (one address)	Individual
Per-Vector Masking	Optional	Mandatory
Table Location	Capability structure	BAR space (flexible)
CPU Affinity	All vectors same CPU	Per-vector control

3. MSI Mechanism

MSI Interrupt Flow

    Device                           Root Complex                    CPU
       │                                  │                           │
       │ 1. Event occurs                  │                           │
       │                                  │                           │
       │ 2. Memory Write TLP              │                           │
       │ ─────────────────────────────────►                           │
       │    Address: MSI Address          │                           │
       │    Data: MSI Data (vector)       │                           │
       │                                  │                           │
       │                                  │ 3. Interrupt              │
       │                                  │ ───────────────────────────►
       │                                  │    (to target CPU)        │
       │                                  │                           │
       │                                  │                     4. ISR runs
       │                                  │                           │

MSI Address Format (x86)

    MSI Address (32-bit):
    ┌────────────────────────────────────────────────────────────────┐
    │ 31:20 │   19:12   │ 11:4 │  3  │  2  │ 1:0 │
    │ 0xFEE │  Dest ID  │ Rsvd │ RH  │ DM  │ XX  │
    └────────────────────────────────────────────────────────────────┘
    
    0xFEE = Fixed prefix for interrupt address range
    Dest ID = Target CPU APIC ID
    RH = Redirection Hint
    DM = Destination Mode (0=Physical, 1=Logical)

MSI Data Format

    MSI Data (16-bit):
    ┌────────────────────────────────────────────────────────────────┐
    │ 15 │ 14 │ 13:11 │ 10:8 │  7:0  │
    │ TM │ LV │  Rsvd │ DelvM│ Vector│
    └────────────────────────────────────────────────────────────────┘
    
    TM = Trigger Mode (0=Edge, 1=Level)
    LV = Level (for level-triggered)
    DelvM = Delivery Mode (000=Fixed, 001=Lowest Priority, etc.)
    Vector = Interrupt vector number (0-255)

4. MSI Capability Structure

Offset	Register	Description
00h	Capability ID	05h = MSI
01h	Next Pointer	Next capability offset
02h	Message Control	Enable, vectors, 64-bit, masking
04h	Message Address	Lower 32 bits of MSI address
08h	Message Upper Address	Upper 32 bits (if 64-bit capable)
0Ch	Message Data	16-bit data value
10h	Mask Bits	Per-vector mask (optional)
14h	Pending Bits	Per-vector pending (optional)

Message Control Register

Bits	Field	Description
0	MSI Enable	Enable MSI (disables INTx)
3:1	Multiple Message Capable	Log2 of supported vectors (0=1, 5=32)
6:4	Multiple Message Enable	Log2 of enabled vectors
7	64-bit Address Capable	Supports 64-bit address
8	Per-Vector Masking Capable	Supports per-vector masking

5. MSI-X Mechanism

MSI-X Table Structure

    MSI-X Table (in BAR space):
    
    Entry 0:   ┌──────────────────────────────────────────────────┐
               │ Message Address (64-bit)                         │
               │ Message Data (32-bit) │ Vector Control (32-bit) │
               └──────────────────────────────────────────────────┘
    Entry 1:   ┌──────────────────────────────────────────────────┐
               │ Message Address (64-bit)                         │
               │ Message Data (32-bit) │ Vector Control (32-bit) │
               └──────────────────────────────────────────────────┘
    ...
    Entry N:   ┌──────────────────────────────────────────────────┐
               │ Message Address (64-bit)                         │
               │ Message Data (32-bit) │ Vector Control (32-bit) │
               └──────────────────────────────────────────────────┘
    
    Each entry = 16 bytes
    Vector Control bit 0 = Mask bit

MSI-X PBA (Pending Bit Array)

    PBA Structure:
    
    ┌─────────────────────────────────────────────────────────────┐
    │ QWORD 0: Pending bits for vectors 0-63                      │
    │ QWORD 1: Pending bits for vectors 64-127                    │
    │ ...                                                         │
    │ QWORD N: Remaining pending bits                             │
    └─────────────────────────────────────────────────────────────┘
    
    Bit X = 1: Vector X interrupt is pending (masked but triggered)
    Bit X = 0: No pending interrupt for vector X

6. MSI-X Capability Structure

Offset	Register	Description
00h	Capability ID	11h = MSI-X
01h	Next Pointer	Next capability offset
02h	Message Control	Table size, enable, function mask
04h	Table Offset/BIR	BAR and offset for MSI-X table
08h	PBA Offset/BIR	BAR and offset for PBA

Table Offset/BIR Format

    ┌────────────────────────────────────────────────────────────┐
    │ 31:3                              │  2:0  │
    │ Table Offset (QWORD aligned)      │  BIR  │
    └────────────────────────────────────────────────────────────┘
    
    BIR = BAR Indicator Register (which BAR contains the table)
    Table Offset = Offset within that BAR

7. System-Level Configuration

MSI-X Enable Sequence

Read MSI-X capability to find table location
Map BAR containing MSI-X table
For each vector needed:
- Write Message Address (target CPU)
- Write Message Data (vector number)
- Clear Mask bit in Vector Control
Set MSI-X Enable bit in Message Control
Clear Function Mask if set

Interrupt Affinity (Linux Example)

    # Set IRQ 45 to CPU 3
    echo 3 > /proc/irq/45/smp_affinity_list
    
    # Or using bitmask (CPU 0 and 2)
    echo 5 > /proc/irq/45/smp_affinity
    
    MSI-X allows different vectors to different CPUs:
    Vector 0 → CPU 0 (admin queue)
    Vector 1 → CPU 1 (IO queue 1)
    Vector 2 → CPU 2 (IO queue 2)
    ...

8. Rules and Requirements

MSI/MSI-X Normative Rules

When MSI or MSI-X is enabled, INTx MUST be disabled
MSI and MSI-X MUST NOT be enabled simultaneously
MSI-X table entries MUST be QWORD aligned
Device MUST NOT modify MSI-X table entries
MSI/MSI-X memory writes follow PCIe ordering rules
Masked MSI-X vectors MUST set pending bit when triggered
Device MUST support 32-bit Message Address

Ordering Guarantee

MSI/MSI-X writes are posted memory writes, ensuring the interrupt arrives AFTER preceding data writes from the same device (producer-consumer ordering).