An attenuator (passive) doesn't control anything but the volume function (voltage usually). In a passive attenuator, it's the source component that effectively drives the load via the volume control + connecting wires. If one has a source with a strong output stage &, say, a compatible impedance TVC (or a carefully designed ladder) there should be no problem in driving the load without an active stage inbetween...
Of course, change the source & you may have problems.
Of course, change the source & you may have problems.