"All amps distort the more their volume is raised"
That may be true but not necessarily through the entire volume range. The higher distortion at the (very) low portion of the volume range is a fairly widely agreed upon artifact of most amplifiers. Same is probably true for most attenuators perhaps with the exception of the fancier step resistors, etc. One other thing to keep in mind is many attenuators do not have linear adjustments by design and the lower portions of the control are coarser than the higher range, etc.