Diagnosing a link that will not train

A link that will not train is diagnosed from evidence, not from suspicion. The first pass reads the Link Status and Link Capabilities registers, the LTSSM state, and the error counters at both ends of the slot; only after those readings are consistent does a bench swap parts. Substitution is an honest test when the replacement is the only variable changed and the counters are read again on the same firmware, the same slot and the same traffic pattern.

The same discipline applies outside the server room. A home machine that fails to come up is usually worked through in the same order, symptom first, part last, which is the method set out in the PLATINENBLICK guide to domestic hardware maintenance. The parallel is useful because the failure modes rhyme: a device that is present but not enumerating, a slot that reports a narrower width than its neighbour, a system that reboots under load.

Which counters show a link in trouble?

Four groups of counters carry most of the diagnostic weight.

Link Status (offset 0x12 in the PCI Express Capability structure) reports the negotiated link speed and width. A width of x1 where the slot is wired x4, or a speed of 2.5 GT/s where both ends advertise 8 GT/s, is a training outcome, not an error. It says the link came up degraded, which usually points to a lane, a connector or a reference clock rather than to the protocol logic.

Correctable error counters in the Advanced Error Reporting block show the physical layer struggling. A rising count of Receiver Error, Bad DLLP or Bad TLP entries means the link is training and retraining at the margins. These are the counters that move first when a riser is marginal or a connector is contaminated.

Uncorrectable error status bits, particularly Training Error and Data Link Protocol Error, mark events that forced the link down. They are latched, so a single event from an earlier boot can still be sitting in the register. Read them, then clear them, then read again after a defined workload.

Finally, the LTSSM state itself. Reading it through a debug port or a vendor register tells you where training stopped: Detect, Polling, Configuration, Recovery or L0. A link parked in Polling has not agreed on lane polarity or bit lock; a link that reaches Configuration and falls back to Detect has a width or lane negotiation problem.

What does a retrain count hide?

A retrain count is an aggregate, and aggregates hide mechanism. The counter increments whenever the link leaves L0 and re-enters Recovery, but it does not say why, how often per lane, or whether the retrain succeeded on the first attempt. A link that retrains once an hour and recovers cleanly is a different problem from one that retrains forty times a minute and never settles.

It also hides direction. Recovery can be initiated by either end. If only the downstream port logs the event, the upstream device may be driving the retrain through its own equalisation or power management behaviour. The count alone cannot separate a marginal channel from an aggressive low power policy.

And it hides time. A retrain count read at boot may be inherited from a previous session if the register is not cleared. The honest procedure is to clear the counters, run a fixed workload for a fixed period, and read the delta. Without that baseline the number is decoration.

When is substitution an honest test?

Substitution is honest when it isolates one variable and the counters are re-read afterwards. Swapping the add-in card into a known good slot, with the same firmware and the same traffic, tests the card. Swapping a known good card into the suspect slot tests the slot. Doing both at once tests nothing.

The test is also honest only if the replacement is genuinely known good. A spare card of the same model that has never been validated is a second unknown, not a control. Where possible the bench keeps a reference card and a reference slot that are used for nothing else.

Substitution cannot show margin. A card that trains in a short slot at 2.5 GT/s may still fail at 8 GT/s over a long channel. It cannot show intermittent faults that depend on temperature or on the presence of a neighbouring device. And it cannot show a firmware or configuration problem, because the replacement runs the same firmware.

Symptom table