Amp & headphone amp plugged in simultaneously?

Hello gentlemen, I couldn't don't a solution so I'm asking...my dilemma.  From my preamp/dac (amr-dp777 I tried XLR into my main amp (plinius) and then RCAs into my headphone amp (topping a90) and vice versa.  I also tried rca splitters at the preamp.  When any of these happen my headphone amp gain is substantially reduced.  I take it I'm reducing the impedance even though the plinius is off?

What I'm trying to to is avoid having to unplug the plinius everytime I want to use my headphones, surely there is a solution short of buying a dac/preamp what also serves as a headphone amp?

Thank you for your time.