I've seen at least one speaker company indicate on their website that the speaker were designed for the speaker grills to be on. I thought it was Paradigm, but I didn't find it in my very quick reference search prior to this post.
It's personal preference, think of it as a built-in tweak.
I preferred the grills to be off on my previous Inifinity IL-50 speakers, but much prefer the sound of my Focal 836v speakers with the grills on (Note: the tweeter is above the cloth grill).
When I went to the dealer to demo the Focal 816v vs. the 836v the salesman had them setup with the grills off. It only took one song before I insisted that we put them back on. The speaker was brand new and really sounded harsh with the grills off. The speakers sound much better after being broken in, but more than once I've pulled the grill covers and always put them back on fairly quickly.
Personal preference at best.