Johnny:
I lean towards the speakers first, myself. They're probably going to be the lions share of the budget, and they're what I think really define the sound of a system. By that I mean that unless you have speakers capable of presenting the sonic nuances of the sources and amplification, you can buy pretty much any gear and it can pretty much all sound the same. Or worse, really good gear can sound awful. This is bad. But besides the sonic issue, speakers play a dominant role in the appearance of and sound in the listening room. The limitations of your listening space can sometimes dictate the type of speakers you're going to want, which then impacts the appropriate pre/amp combinations, and that may impact your source options. My opinion is that good amplification can't really improve the sound of the speakers, but bad speakers can mangle the signal from good gear.
Loathe though I may be to do so, I can illustrate from experience. Newly infected by Audiophilia Neurotica (distinct from a. nervosa, which is the chronic and recurrent strain) I studied and sought, listened and bought, and loved my NAD and Paradigm system. Then I moved. DISASTER! The Paradigms became bright and forward, treble raspy, bass boomy, oi vey! No, nothing was damaged in the move, the characteristics of listening room just changed so much that what once was good now was....not at all good. So again I do the shop, study, audition thing, and finally settle on bringing home some Silverlines to demo. Love them. Fit the new room perfectly, no more boom, no more sizzle. Am almost happy again, except...now I can really hear the NAD - and my a. nervosa begins.... I've since demo'd several amps, and found that the Silverlines allow me to hear the subtle characteristics of the equipment that I'm not certain I would have heard with my old speakers. More importantly, I know what the gear sounds like in my room on the speakers I'll be listening with for quite some time.
And that is why I suggest starting with the speakers.
chas