Because we are dumb!
By far the biggest deviation from the (whatever it is) ideal signal chain is caused by the speakers (and the room). Logic dictates that this is where the best bang/buck can be achived (not that more $$$ automatically results in better results).
On the other hand: any reduction in loss/error is a GAIN. So we are striving to make improvments in ALL sections of the signal path. But if someone comes to me with $2,000 in hand and tells me to put together a system for him/her, I would NOT spend $1000 for an amplifier and $200 for cables.