Unlikely to matter - a permanent magnet is fixed like the earth's magnetic field it has no impact on ac signals. I doubt the voice coil will carry as far as that.
nonshielded magnet / speaker
can you have a passive crossover close (within 2 inches) of the magnet of a speaker? Does it matter?