Can you have too much speaker presence?

My dealer says I’d lose too much speaker presence if I went to a smaller speaker. I’m not posting the speaker in question solely because I don’t want this to become an attack on them. I get it, presence means there and in the area, but can too much become and issue, especially when it is centered in a specific frequency range?