kijanki,
When you short a oscilloscope and probe to ground, you are creating a loop antenna and creating a differential signal which is not to say that common mode noise and/or common mode voltages are not a problem in measurement. That is why Picoscope makes a differential input scope, and Tek and Lecroy sell high voltage differential probes for about $2,000/each (or more). While you are "floating" you are also a big capacitor to ground which creates a path for common mode signals. Also remember that most oscilloscopes probe grounds are also connected to chassis grounds. Floating oscilloscope inputs (battery powered or otherwise), can offer most of the benefits of differential inputs for many measurements (and tend to be much quieter).
And are you assuming the noise was common mode? Are you sure it was not differential mode?
I had TT long time ago and always had problems with low frequency hum (single ended phono stage).
Current thru this ground wire causes tiny voltage drop, seen by the input as input voltage (input is referenced to BNC GND and not to end of the cable GND).
But again, we are talking a cartridge and absent the shielding and other metal structure (and antenna effects isolated by the cartridge itself), it is a loop. Absent the shield/structure capacitance to ground, there is no other current path, whatever current flows in one wire must flow in the other, and hence whatever drop you have on one wire is cancelled by the other.
Transformers are already used for MC coils, so we know they work.