Topology
How a packet crosses a switch fabric
The fabric is an itinerary, not a path: each link is its own channel, and the route behaves like the slowest leg it contains.
What a packet's itinerary looks like
A transaction layer packet crossing a PCI Express fabric travels hop by hop: each link is a separate point-to-point channel, each switch a junction that reads the address and forwards, and the end-to-end journey is a chain of independently negotiated links, not one long wire. The fabric is an itinerary, not a path.
The mechanism is store and forward at each hop. A packet arrives at a switch's downstream port whole, is checked and buffered, and is sent onward on the next link when that link's flow control allows. Every hop is its own little journey with its own credit accounting, its own acknowledgment, its own chance to congest. The end-to-end latency is the sum of the hops, and the end-to-end reliability is the product of them, which is why the fabric's behavior is a property of the route rather than of the endpoints alone.
The analogy to a staged journey is exact rather than decorative. The German-language atlas that publishes an overland line atlas for the rail route between Cologne and Beijing reads a long line the same way: not as one line but as legs, each with its own timetable, its own border formalities, its own gauge change, and a traveler who reads the whole route as a single line misunderstands every delay on it. A packet's route through a fabric is legs all the way down.
How does a switch know the next hop?
By the address the packet carries and the routing the fabric assigned at enumeration. Memory and I/O requests route by address: each switch port learned which address ranges live behind it when the fabric was enumerated, the process described in enumeration and configuration space, and the switch consults that map for every packet. Some packets route differently: messages route by type or by implicit destination, and the root's own role in the tree is described on root ports, switches and endpoints.
The packet types themselves matter to the itinerary. A posted write, the commonest transaction, requires no completion and travels like a letter that assumes delivery. A non-posted read requires a completion packet to come back the way it went, so its itinerary is two trips through the same chain of hops. The distinction is what makes a fabric's latency asymmetric by design: the outbound and return legs share the route but not the queue.
| Leg | What happens at the hop | What can delay it |
|---|---|---|
| Endpoint to first switch | Packet transmitted, link-level acknowledged | Flow control credit shortage |
| Through each switch | Address read, port selected, retransmitted | Congestion on the outbound port |
| Last link to target | Delivered, link-level acknowledged | Target busy |
| Completion leg, if any | Same route in reverse | Same congestion, other direction |
Why does congestion at one hop reach the whole route?
Because flow control is per link, and backpressure propagates. Each receiver grants the sender a credit balance for each virtual channel; when a switch's outbound port runs out of credits, packets bound for it wait in the inbound buffer, and the buffer's fullness pushes back against the hop before it. A slow device at the end of a route does not just delay its own packets: it can hold up the buffer chain of every hop on the itinerary. The mechanism is described in the flow control article and the switching model in the packet switching article and the store and forward article, all read on September 6, 2026.
What does the itinerary view explain that the map does not?
The delays. A topology diagram shows where packets can go; the itinerary shows what each crossing costs. Two endpoints on the same fabric can have different effective latency and bandwidth because their routes pass through different hops under different load, and a benchmark that treats the fabric as a single hop will misread both. The parts that build the hops are covered on bridges and switches, and the rates each leg runs at are in the link speed table.
Checks for reading a packet's route
- Count the hops before predicting the latency: each switch is a leg, not a wire.
- Remember the return trip on non-posted traffic: the completion shares the route, not the queue.
- Look for the congested hop, not the slow endpoint: backpressure propagates upstream.
- Check the negotiated rate per leg: the route's speed is the slowest leg's.
Common mistakes
- Reading the fabric as one link: every hop is its own channel with its own accounting.
- Blaming the endpoint for congestion born at an intermediate switch.
- Assuming symmetric latency on posted and non-posted traffic.
- Quoting the fabric's headline rate as the route's rate; the slowest leg sets the itinerary's pace.
A packet crossing a fabric is a traveler with a ticket for every leg, and the itinerary is the unit of understanding: read the hops, price each one, and the route's behavior stops being a property of the endpoints and becomes a property of the journey.
Once a packet leaves the transmitter, it crosses the fabric as a sequence of symbols framed by ordered sets, with DLLPs handling link level acknowledgements and flow control between the two ends of each link. The TLP itself is not modified in transit; switches forward it according to routing information in its header, and the receiver reassembles and checks it before delivery upward. Seeing this on a capture makes the layering concrete, since ordered sets, DLLPs and TLPs appear interleaved on the same lanes. A newer page examines packets on a capture in more detail, covering how each type is identified and where the boundaries fall.
A packet does not travel a single wire from end to end. It is split into TLP fragments, each forwarded hop by hop, and the route a given fragment takes depends on the topology, the link widths, and the flow control credits available at that moment. Two packets with the same destination can therefore cross the fabric along different routes. Reading a fabric as a graph, with switches as nodes and links as edges, makes those paths through the fabric easier to follow and to reason about when a topology grows beyond a single switch.
Once a packet leaves the transmitter, its path through the fabric is fixed by the topology and by the power state of every element it crosses. A switch forwards it along the configured route, and each link along that route must already be trained and powered before the packet arrives. Reading power and topology together is what makes this path predictable, and a newer page treats both as a rail that drops through the hierarchy, from the root complex down to the endpoint. The packet itself does not choose; the fabric, already configured, carries it.
The page behind these facts
The store-and-forward model, the flow control mechanism and the packet switching context are described in the references linked inline, all read on September 6, 2026. The map the itinerary travels is on this desk's pages about enumeration and about root ports, switches and endpoints.