You are right about freq. response, harmonic distortion and other problems being greater in speakers than any other component.
But an upgrade in other components for just say $1,000 can produce better SQ even in speakers that are not the best (which can cost $120,000).