That should be fine. The impedance of connectors (and cables) refers to what is called "characteristic impedance," not actual resistive (or other) loading, and is relevant only to frequencies that are much higher than audio frequencies (such as the spectral components of digital signals).
Regards,
-- Al
Regards,
-- Al