no damage for sure but be aware that not all ported speakers are designed to be plugged. Unless your speakers came with plugs originally chances are they are not. It depends on physical properties of the woofer suspension stiffness and travel. In general sealed speakers are looser and have more cone travel and ported speakers tend to be stiffer with less travel. Some are right in the middle and capable of both modes.
Plugging speakers that were not designed for it may restrict the travel of the woofer such that the output will suffer.
Such was the case with some PSB towers I had. Too much bass so I plugged the ports but the sound was not good. See if you can give yours more distance to boundaries if plugging doesn’t sound very good.