Why the delay exists
Your trade does not travel in a straight line. It crosses your device, your local network, your ISP, and a series of intermediate hops before it reaches the broker. Every step adds some amount of delay.
That is normal. The problem is not that delay exists. The problem is that the delay can grow or wobble at the exact moment you want a clean fill.
Why the difference matters
Latency on its own is a number. Latency plus jitter is the version traders feel. A route that stays around the same timing feels calmer. A route that swings around feels unreliable, even when the average looks okay.
That is why the connection path sits in the middle of the story. You are not trying to eliminate physics. You are trying to keep the path from adding avoidable delay.