VPI SAMA Flywheel (small)

I have one of the last units from the last run of the HW19-Mk IV with the TNT upgrades (although if memory serves, Mk IV denotes those upgrades).

Anyway, recently the SAMA began to make noise, which was more prevalent with the VPI turntable line conditioner (SDS) in place (the older version, not the latest one). When the PS is taken out of the system, the SAMA makes less noise, but is still audible. 

So I took everything apart. I took off the access panels on the SAMA and pulled the motor out of its housing...looks normal. I did put a very small amount of very light-weight oil on the motor shaft, which probably accomplished nothing. The motor is completely silent out of the TT and on my bench. (Whew!) Mine came with the small "flywheel" that fits over the shaft spindle of the motor. When I reinstalled it on the shaft, the noise returned (at a lower level now that the motor is not connected to the rest of the 'table, but definitely audible).
I've cleaned and slicked the surface of both the spindle and the inside of the flywheel to no avail - the noise is still there, so I assume the noise is the flywheel "chattering" on the spindle.

Any ideas?
I can presumably put everything back together minus the flywheel and, I assume, solve the noise problem. 
What benefit will I lose by doing so? 
Just remove it and call it a day? 

Thanks.