Any distortions that round off the corners of the analog waveform making up the SPDIF signal cause the receiver to derive a jittery clock signal. In any digital logic, there is a transition zone between the voltages representing the logic states "1" and "0" that is neither state. A rounded off waveform spends more time in this transition zone, thus delaying the detection of logic transitions, thus affecting the derived clock signal.
This effect is not constant.
Improperly terminated coax (RCA plugs instead of BNC connectors, or bad transmitter/receiver design) will have reflections giving rise to standing waves which will cause distortion to the SPDIF signal.
Basic CD player design is still rooted 1983 technology. In those days memory and processor were expensive. I think that one could completely rethink the player design today, using current digital technology and its new price points.
Perhaps the CD drive (at 4x or better) could read ahead, and fill up a circular buffer. The data stream could then be clocked out of the buffer at a constant rate independently of the transport This scheme would even allow for retries of uncorrectable read errors, and would break the direct connection between data timing and transport timing. Of course we'd want to dump SPDIF as a connection mechanism...
I bet it would be easy to make using off-the shelf parts as well.