It's a power and heat sinking problem. The lower the load the amplifier sees the more heat has to be dissipated by the output transistors. Transistors have a maximum operating temperature and they'll fail at that temp so amps are designed with thermal cutoffs well below that limit if the heat sinks get too hot.
But... if you lower the volume then there is less heat dissipated and if that volume level is loud enough (courtesy of the high SPL of the speakers) and the amp is happy and not overheating at that volume level, then it's not a problem. The amplifier does not care what the SPL rating of the speaker is, that is between the speaker and it's crossover and internal volume -- the amplifier sees an inductive load only.
To give you an idea of the heat sinking requirements:
say I want an amplifier with only one pair of output devices, 50 watts at 8 ohms, biased low A/B, with 36 volt rails. At peak signal input (say 1.2 volts) the power the transistor dissipates at peak is about 45 watts. At 2-ohm loading it's 150 watts. Can't do it, needs more pairs of transistors so the amp is limited in its power output to not lower than 4 ohms.
But if you cut the input voltage with the volume pot down to, say, 200 millivolts -- and that is loud enough for you -- the power dissipated at 2-ohms is 50 watts, which is within the max power spec at 8 ohms.