Which component to upgrade?

This question could probably be posed under any of the topic options but the amp/preamp section seems to get most traffic. Someone on a currently active thread said he decided to move from an ARC Ref3 to Ref5SE and then up to Ref6 because he felt there was something missing in the music. I'm sure we all feel that to some degree almost regardless of the caliber of the system we own as our ears/brain gets more critical of what it consumes and feel the urge to upgrade. But how do we know which component in the audio chain is the one that is holding back what we miss? Some go the speaker path while others get a better source, DAC, or cable, etc. 
What criteria do you use for deciding what upgrade path to take?