Some speakers are in fact voiced to be heard with the grills on but I would imagine that most high end speakers are not. Listening with the grills on will muffle the tweeter to some degree, the mid range to a lesser degree. The grill frame could also cause reflections and distort the sound. However with some electronics which are inherrently bright this might not be an all together bad thing. The amount of muffling will depending on the type of grill cloth. But with a well matched system, grills off will allow you to hear all of the sound that your speaker (and electronics) can resolve. For me thats the preferred way. I put the grills back on when company comes. :-)
