How forgiving your system is a result of your whole system, not just the speakers: components, acoustic treatment and the room. So the place to start is with your whole system. If your could create a virtual system under your user ID and put some photos and your components that would give us a place to start. I emphasize photos, because in situations like this it is easy for there to be a glaring reason that doesn’t show up in a list of components.
I have been pursuing high end audio for over fifty years. At times, with good intentions and by careful listening tests before purchases, my lust for details pulled my system from musical and natural sounding over to unforgiving and overly detailed... if you keep going on this continuum then it becomes harsh. It is really easy to do. As you implied, you figured that out as well. Some component manufactures produce products that create incredibly detailed sound, scraping every last nuance from the recording. Nothing wrong with this. but the presentation can become very detailed and dry and make anything but the very best recording sound bad. It does so by emphasizing the details... especially that of the ambiance, presenting fine details out of proportion. A good way to tell is, if the first thing that occurs to you when listening is the ambience and mastering of the recording then there is something wrong. Instead of being pulled into the music and forgetting about the system. When you are pulled to listen to the system instead of the music, you have a problem. Unless, of course that is what you want. Lots of folks enjoy the technology and listening to the violinist in the third row foot movement as it is a solo instrument.
There is also a segment of that works to reproduce music with all its nuance. These then to be tube systems and are forgiving... not of a truly hideous recording of course. From a speaker point of view these are Sonus Faber and Serbin.