@wturkey Measure the voltage on the output side of that capacitor. If the input circuit employs a discrete device like an FET or transistor, there may be a voltage present to bias that device properly. Getting rid of the cap could thus cause the cartridge to have a voltage across it which could mess with it and it could cause the bias voltage for the input circuit to be messed up too.
You might consider a film cap to replace that. Wima makes a little 10uf film cap rated at 35V IIRC; a little rectangular thing you can get from DigiKey.