If the headphone amp has a proper line-level input circuit with appropriate input impedance (say 10K ohms or higher), it will not affect the sound quality of the integrated amp when playing the speakers. Schiit makes some good headphone amps that meet these specs -- the Fulla at $99 and the Asgard at $199 are good examples.
Inexpensive headphone amp paired with an audiophile integrated amp
When one's integrated amp doesn't have a headphone jack and one routes out to a headphone amp, is the quality of the signal compromised by an inexpensive headphone amp?