Speaker fed high power then switched to low power?

If someone purchased speakers that were used with a high wattage quality amp ( let's say 1000 or 500 watts) and were then sold to a new owner using a lower wattage quality amp (say 100 or 200 watts) would the speaker sound performance be compromised? Specifically is there any permanent change using the higher power amp to the speaker components that would warrant using a similiar amp? Once the speaker is broken in, is it broke in to it's maximum potentail regardless of power fed to it?