??? How Much Would You Spend ???

In my experience, speakers make the absolute biggest difference in how your system will ultimately sound

I guess it depends. I tried a few integrated amps in the last few months with 2 sets of speakers. They sounded profoundly different. I practically could not tell in a blind test which speakers were on, although I thought I could recognize them from a mile. (Other than volume, there is a 5db sensitivity difference)

But in general I agree, to find the speakers that sound the best to your ear is where to start. 

Questions like this are about synergy not dollars which is why they are impossible to answer without knowing specific brands. So what are these speakers?

More money doesn't mean better. Go get a Buckeye Purifi and you're set for life. Nothing will sound better. 

The Pre Pre Amp or Pre Amp functions typically as the first stage of adding gain to a Signal produced by the Source. 

The Pre Amp can make a very well produced Signal a less attractive Signal to Send on, the end sound can be perceived as not too attractive. The Pre Amp can also maintain the qualities of the Source Signal and produce and end sound that has an attraction.

What matters is how the Signal is sent on through the downstream devices.

@yoyoyaya stated "  Systems are called systems for a reason so they should be built holistically - a chain is only as good as it’s weakest link etc. "

This in my view is very correct .

In multiple cases the weakest Link is the Room selected for the Speakers to be set up in. As working with the Room to improve the interface/coupling is not carried out or other influences are a factor in the methods that can be used not being adopted.

A System is sending a Signal that has the capabilities to be very attractive as an end sound. The loss of this comes from where the conversion of electrical energy to mechanical energy. Has a sound produced that is not managed within the Room it is produced in a way that controls the produced energy. The produced energy moves through the Space meeting structures and creates an energy that is not a sound that is for the betterment of the listening experience.

Set Up in the incorrect environment, there is not any Speaker that is able to offer a true account of its capabilities. In a good outcome, ( I don't think there is a perfet outcome. That belongs back at the R&D Test Chamber for the Speaker) certain traits of the Speakers attraction may manifest.

For certain Speakers the cost associated and the performance attained through not optimising the Space for energy being released to Produce Sound with the outcome being a non-optimal Room Interface. Will in multiple cases, be a situation where sound being listened to is the least of the benefits of making the purchase.  The aesthetic/eye candy impact of the Speaker is seemingly the most realized benefit to be had from the Speaker Purchase.  

If I didn't already have extra components, and I was starting from scratch?

I would look HEAVILY at the Dutch & Dutch 8C speaker system. They are an all in one solution with streamer, DAC, room correction DSP, amplifiers, all built into the speakers.....just add ethernet, AES/EBU, or analog XLR and you are done.

I've heard them 3 times at shows, and each time have been more than impressed. $15,000 new, or $10,500ish if you can find them used