Clearly a problem in the Rega PSU. Likely there is a poor contact in the on-off switch or simply the noise of switching it is being picked up as an RF signal by the cartridge and amplified. What @testpilot said.
Solutions:
1. Get the PSU tested and serviced by a Rega agent.
2. Try moving the PSU further away from the TT if you can. Indeed, you can test my explanation by moving the PSU to and from the cartridge. When nearer, the sound will be louder.
3. Try shielding the PSU by putting it in a metal box.
@ibmjunkman "Not familiar with the TT. You have to power it off to change a record?" If it is wished for the TT to be stationary when a record is changed then, yes, all TTs have to be powered off. At 33 rpm (but not 45) I change records without powering the TT off. Nothing has ever come to any harm.