Which is the most important part of a stereo system?


My system consists of a pair of B&W 630's, an old Denon 50 watt reciever (DRA-550) from the mid 80's, a Marantz CD5004 cd player, and now a Pro-ject Debut Carbon DC Turntable.  I'm pleased with the speakers and the cd player and while the Denon sounds good it has some issues and I want to upgrade.

I'm planning on returning the Pro-ject TT and getting a higher end TT.  I'm also looking into getting a new amp pre amp with a internal DAC.

Is the source the most important? The speakers? 

Please Help!
klimt
The most important part of any audio system is your ears. If you can’t hear the difference between two components, buy the less expensive item. If a later system modification exposes the flaws of said component, then upgrade. 

This is the story of my life. I enjoy listening to music and upgrading my system(s). I hope that this never ends. AMEN
 So I’m definitely not in the audiophile major leagues like most of you guys, meaning I don’t now or probably will not ever have 10k+ to spend on my system. However I’m definitely an audiophile and used to build Martin Logan speakers in the 80s when I was at KU in Lawrence Kansas. I definitely got the bug listening to krell, audio research, and Macintosh driving reference Martin Logan systems. That said,  i’ve been very lucky to piece together two amazing systems on the cheap. I’m driving 1970s klipsch cornwall ones, the verticals, with a Chinese Doge  tube amp at 60 W controlled by an old rotel RC 995 preamp and a crappy JVC CD player from the 90s.
 I was lucky to get an 1988 Sota Sapphire turntable for vinyl for a song from a generous guy moving out of state recently. 
 For streaming i use the discontinued chrome audio, and or a dragonfly 

 I’m sold on klipsch horn loaded speakers for their durability, efficiency and the ability to get an amazing premium sound for reasonable dollars on the whole. 
I have quite a few pairs of klipsch (la scals,cornerhorns,hereseys,) and fix them up and resell them to finance the hobby
   So to conclude I maybe have 2000 bucks into the system I described and my opinion is definitely every component means something to the whole stream. I will say that you can put a lot of subpar amps into Cornwalls or  La Scala‘s and get a pretty good amazing sound, at the very least  A lot of sound pressure with low watts as a source. 
 I do know that putting a pass labs, or McIntosh amplifier into most klipsch speakers is  going to give you  top of the line sound , but 2500 to $6000 for an amp at least for me isn’t in the cards.  I haven’t even started adding subwoofers to my systems but know that would be my next move to complete the systems.
 Again I think I’m like the majority of folks that really want the audioNirvana sound but really just don’t have the income to achieve it. But if you’re patient like me and find some good deals you can put together an amazing sound system for a budget that is possible/ realistic for a large group of people.  I’ve played a lot with room treatments and placement and I think in the end that helps quite a bit. 
   I would agree with a couple of the posts that really decent and in my opinion high efficiency  speakers would be first, then the clean amp, with the sources backing up the list. 

What @almarg said. I suggest not spending disproportionately on anything that (1) wears out, or (2) has low benefit per unit cost, like cables.
Deschutes, why not DIY your electronics? I would say that the main thing is to stay away from teflon insulation, because when teflon heats up past about 450F, it begins to decompose into some really evil gasses. And unless you have a fume hood or equivalent protection, well ...