Which speaker tap to use?

I have a flee watt 2A3 headphone amp that also has 8 and 16 ohm speaker taps. My speakers are fairly efficient with a nominal impedance of 8 ohms but here's the catch. For some reason the amp was designed to only cut out the speakers when switching to headphones if the speakers are connected to the 16ohm taps. If the speakers are connected to the 8 ohm taps they continue to play. For convenience when switching between speakers and headphones I would prefer to use the 16 ohm taps but I'm not sure that's a good idea. I am having a hard time nailing down the sound difference when connecting the speakers to the different taps. What would you expect the difference to be? Am I risking damage to either amp or speakers?