Assuming that your speakers aren't broken, the source will give the best improvement for money spent. A low priced speaker can at least attempt to reproduce what you feed it. No speaker, no matter how good, can reproduce sound that it is never fed. Getting the info in is paramount. The second thing is preserving it on the way to your ears. This is not to minimize the need for good speakers, as I have been involved in speaker design and building for many years. And I appreciate the importance of them too. I love great speakers, and I would always try to do them justice by giving them the best source possible to work with. However, you can never overcome a poor source with great speakers.
Someone just dropped cash into your lap
If you suddenly came upon a financial windfall (you determine the ammount) but could only spend it in one of two places; source, or speakers, where would you put it and why?