Speakers will give you the most obvious difference, but in the long run, I think source is more important.
Better speakers will get you a more accurate representation of the source signal you currently have, a more accurate source will give you the most accurate system possible, with any speaker.