Natural sweet spot in your system.

Something that has haunted most of my systems is finding the sweet spot as it relates to volume. Every one of the systems I have put together the past years seems to have a volume where everything works. I go outside of that on the low or high side and things fall apart. What is the most likely cause? The room, the amplifier or the speakers. How do I increase or lower the volume and keep the same balance in the frequency spectrum.