However, when I disconnect the inputs from the preamplifier (and leave the other end connected to the amplifier), the noise in my speakers gets louder. Why is this?No doubt.... Don't do that. Having the ICs connected to the amp but not connected to any thing on the other end works like an antenna and will introduce large amounts of RFI and EMI to the input stages of the amp.
Not good on the speakers either.