Never had to worry about cleaning a speaker driver, never heard of such a thing....I would not touch them with anything...leave em alone....
Drivers and how to clean them: Notes from ScanSpeak
Please use good judgement but if you ever wondered what the actual manufacturers recommend for cleaning your speaker drivers here's one maker's view:
https://www.scan-speak.dk/technical/tech-notes/
https://www.scan-speak.dk/technical/tech-notes/