Here is a rule of thumb: if you regularly drive your system into distortion, eventually it will break.
Most commonly, pushing an underpowered amp to the point of distortion causes excessive heat inside the tweeters. The heat melts the tiny wires in the speaker windings to the point they act like a blown fuse and short out. Driving woofers into distortion can misalign their suspensions. Constant abuse can tear them apart. If a woofer cone has jumped out of the gap it lives in, sometimes you can reseat it by hand and everything will be ok.
The occasional pop is probably ok. I doubt any recorded distortion will hurt them, so long as the distortion is reproduced cleanly :^)
