Hi Inscrutable; your question caused me a great deal of thought. Thankyou. After 10 years and about $50K spent, I've come to the conclusion that a really great sounding system can only be built by "trial and error", and taking into account your personal music quality/character preferences and biases (and of course your budget),eg I now know that I prefer music that is a bit warm and rich rather than cool, lean, or analytical.
I also believe that "neutrality" is an elusive, and maybe impossible goal. It may not even be a realistic or desireable objective. Subjectively, what I consider neutral you may consider colored. I suppose neutrality can be measured using instruments, but what the instruments show may not agree with what you hear.
I've done both of what you ask above-- sometimes it worked and sometimes it didn't. IMO, you have to be constantly searching for system synergy and satisfying your own listening preferences. I just paid over $11K for my "dream" speakers, and I'm finding them a bit cool and lean, and very revealing in "my room" and with the rest of my system. Yet prior to purchase, I thought they would fit perfectly with the rest of my system.
So, in a way I'm back to square one, and am trying to figure out how to get a little more warmth and richness in my system, while still keeping the strengths of the new speakers. What do I change to get what I want? I thought all my components would complement these speakers nicely-- but not so.
I think maybe a worthwhile general approach would be to pick out the most important component to you and carefully build your system around it a piece at a time. Knowing this now, I would start with speakers because of the major importance of speaker/room interaction. My system now sounds very, very good, but it has some weaknesses too. Good Luck and Cheers. Craig