It has an affect but it's not always noticeable. I remember a speaker that had the grills on that I could clearly tell had missing highs. I took off the grill and found the tweeter was pushed completely in. I pulled it out and it sounded normal again.
