Is it okay to talk around your speakers?

When a system is not on, should you avoid making any noise near the speakers? I am thinking about back EMF caused by the speaker cones moving and then sending voltage backwards into the crossover and the rest of the system - could this cause damage or even minor degradation to the sound quality after a while? Do you unhook the speakers when not in use just to be on the safe side?