Removing speaker’s grille will shorten its life due to sensitive drivers exposed to dustI wouldn’t worry about that. At all.
IMO the only thing the grilles really protect against is the curious fingers of toddlers and perhaps some protection from cats.
If you're content with how it "looks", and you don't have to worry about little kids, I wouldn't give it another thought.