Being a bit underpowered at "normal" listening levels will result in some degree of dynamic compression. The result will not sound "wrong" or objectionable, just not as impactful, or with as much contrast between quiet sounds and loud ones. Severe clipping due to insufficient power at high listening levels is another story altogether. Not what we’re referring to here.
On the other hand, the "wrong" high powered amp will sound wrong to you, which would be objectionable.
So, pick an amp that sounds good to you.