The speaker magnet could affect any inductors/coils making their performance non-linear and causing distortion. It depends on many specifics (such as what type on inductor, how it is oriented, how far from the magnet, how strong is the magnetic field from the speaker...etc.) as to how much of an effect.
nonshielded magnet / speaker
can you have a passive crossover close (within 2 inches) of the magnet of a speaker? Does it matter?