I agree with many points in this thread - how to improve the playback system, and not least, the integration with the room. Yet I think many problems of ’complex music’ are inherent in the recording. You can reduce them in the playback but they don’t go away.
I would rather think of the problem like this:
recording problems x format limits x playback limits
Meaning that whatever problem is there in the recording, will be accenuated with a poor format and playback system (even if ’masked’ by low resolution etc).
We could use + rather than x, in this formula. But in my experience, the problems tend to interact. So if the recording is medium but the format and playback poor, they will appear larger. The sound will be even worse.
Maybe we should put x2 for the recording factor. It means a lot. With high dynamic and complex music, recording can become very difficult. Witness, from way back when, Procol Harum: In Concert With The Edmonton Symphony Orchestra, a pioneer effort from 1972. And we still hear recording problems, e g when The National plus guests try to redo the grand Grateful Dead song Terrapin Station, sounding noticeably worse than most of the more simple tracks on this Day of the dead 10xLP box, 2016.
Why? Recording a lot of instruments and voices together, each sounding their best, is hard to do. Integrating them into coherent sound is difficult. A notable "problem area", just mentioned, is integrating a band and a symphony orchestra, pioneered by prog rock bands like Procol Harum, Deep Purple and Pink Floyd. The best recordings succeeded, like Dark Side of the Moon, but then again, this was due to heavy production - we dont hear a lot of individual players.
My best-sounding recent LPs are less complex music. Like Bhatt and Cooder: A meeting by the river. Two string instruments. Or piano and voice, like D Krall or P Barber. My less-complex music sounds great, the problem is the complex music.
For a problem case, compare the Danish group Mew. I like them a lot, and have heard them live, plus on cd, and on lp. They switch from soft (simple) to strong (complex) passages, and the sound is clearly better on the soft parts. We heard the same thing from Mew live, some years ago. The sound on the strong parts was too loud. (cf Mew: No more stories... LP 2009)