Cleaning outdoor speakers

I have a pair of Niles OS-20 that were mounted under our front deck. This was not my ideal place of course but at the time I had no other options. I bring them indoors each winter and clean them up but this year I noticed that the woofer has some dried dirt on it and appears to be stuck to the woofer material. They sound ok but I would like to clean them up. Would it be safe to wet the dirt/woofer to loosen the crud and then gently wipe off? Or, since they sound ok should I leave well enough alone?