I think the OP is talking about accidentally damaging or losing the grill covering if they are left off and stored somewhere like the garage. We can’t help with that. One has to listen to determine which sounds better, and if it is grill covering off that sounds better, and the look is acceptable, the storage issue really matters only for the re-sale value of the speaker. I would keep them off myself, if the speakers sound better that way.
There wouldn’t be an issue if the speakers only come naked, like Audio Note speakers; even the ones well up into six figures are naked.