It would help to know a lot more specific details, like the amp in question, the speakers, how large the room, how loud you are trying to play the system, whether you tested the system with a different amp, etc. Just knowing that a speaker is rated as 96 db/w does not give enough detail on how suitable is the speaker for SET use. For example, the Wilson WP 6 had a quite high efficiency rating but it was very difficult to drive and was not suitable with many amps, particularly low and medium-powered tube amps. Is this a problem in both channels playing a mono source? Do you have a way to test the tubes or swap in a different set of tubes?
Severe distortion from being pushed to hard could indeed sound like mis-tracking. But, long before you get to that point you should be able to hear the onset of such problems. First, SET amps will start to compress—the music will stop getting louder even as the volume is turned up. The sound will also become a bit muddled and lacking in clarity.
Good luck on solving this problem.