Ordered sets, DLLPs and TLPs on a PCIe link

A PCI Express link carries three distinct vocabularies at once: ordered sets, which are short fixed patterns used by the physical layer to keep the link aligned and trained; data link layer packets (DLLPs), which carry acknowledgements, flow control credits and power management handshakes between the two ends of a link; and transaction layer packets (TLPs), which carry the actual memory, I/O and configuration requests and completions. They are easy to separate once you know what each one is for, because they differ in who originates them, what they acknowledge, and what happens when one is lost.

The distinction matters beyond theory. Anyone reading a protocol analyser trace, or working through a language reference such as the ABAP syntax and statement basics that a developer keeps open beside the bench, is doing the same job: mapping a formal vocabulary onto observed behaviour. The three layers below are the vocabulary of the link.

What is an ordered set and where does it appear?

An ordered set is a short, fixed-length sequence of symbols sent on the link with no packet header, no CRC and no addressing. It appears at the physical layer, on the wire itself, and it is never forwarded to the data link or transaction layers. The receiver recognises it by pattern, not by parsing.

Ordered sets do the housekeeping that must happen before any packet can move. They establish bit and symbol lock, align the lanes of a multi-lane link, and negotiate the link width and speed during training. They also carry the electrical idle ordered sets that let both ends enter and leave low-power states in step, and the SKP ordered sets that absorb small frequency differences between the two reference clocks by inserting or removing symbols.

In a capture, ordered sets look like repeating short patterns rather than structured frames. They have no sequence number, no credit field and no completion. If you see the same symbol group repeating at a steady cadence during link bring-up, you are looking at training traffic, not at data.

What does a data link layer packet do that a transaction layer packet does not?

A DLLP manages the link itself. A TLP carries work for the device at the other end. That is the whole division.

A DLLP is generated and consumed by the data link layer at each end of a single link. It never crosses a switch to a further device. Its jobs are acknowledgement and retry, flow control credit returns, and power management. The acknowledgement DLLP tells the transmitter which TLPs arrived intact, so the transmitter can release its replay buffer. The flow control DLLP returns credits, so the transmitter knows how much buffer space the receiver has free for each traffic class. Neither of these exists at the transaction layer.

A TLP, by contrast, is end-to-end. It is assembled by the transaction layer, addressed to a function somewhere in the hierarchy, and it may pass through several switches before it is consumed. It carries a memory read or write, an I/O request, a configuration access, a message, or a completion. It has a header that identifies the requester, the tag and the target, and it can require a completion to be sent back.

So the practical rule is: if the packet is about keeping this link healthy, it is a DLLP. If it is about moving data or changing a register somewhere in the fabric, it is a TLP.

How do the three layers divide the work?

The physical layer owns the wire. It serialises and deserialises, recovers the clock, aligns lanes, and exchanges ordered sets with its peer. It guarantees that symbols arrive with the correct framing and that the link is trained and stable. It does not know what a packet means.

The data link layer owns one hop. It wraps and unwraps TLPs, assigns sequence numbers, checks the LCRC, acknowledges good reception, requests replay of bad reception, and reports flow control credits. It guarantees reliable delivery across a single link, hop by hop. It does not know where the TLP is ultimately going.

The transaction layer owns the request. It builds and interprets TLPs, manages tags and completions, applies ordering rules and traffic classes, and handles the split transaction model in which a request and its completion are separate packets. It guarantees the semantics of the operation, not the delivery of any individual packet.

Why the three vocabularies stay separate

The separation is deliberate. Ordered sets must be recognisable without parsing, so they are fixed patterns. DLLPs must be cheap, because they are generated per link and per credit event, so they carry no address. TLPs must be routable and semantically rich, so they carry the full header and the ordering rules. Each layer pays only for what it needs, and each one can be debugged on its own terms. Once the three are separated in your reading, a capture stops being a wall of symbols and becomes three readable conversations happening at different scopes on the same wire.