Probably mostly the speakers. Soundstage has hundreds of speaker measurements showing large deviations from linearity even on SOTA speakers.
Natural sweet spot in your system.
Something that has haunted most of my systems is finding the sweet spot as it relates to volume. Every one of the systems I have put together the past years seems to have a volume where everything works. I go outside of that on the low or high side and things fall apart. What is the most likely cause? The room, the amplifier or the speakers. How do I increase or lower the volume and keep the same balance in the frequency spectrum.