That sound upon powering up is highly consistent with "tube rush". In some instances but not yours the sound disappears.
It is a good time to get new tubes, per Rodman 99...You should try to get very quiet selected tubes. A phono stage in particular requires the least noisy tubes you can get, (most of them anyway)