Ah, there IS another unit out there designed specifically for what you want to do although, not much help here, don't remember the name. As I recollect this was a very expensive "cost no object" six channel passive line stage preamplifier. I think it had six unique ganged attenuators and the reviewer raved about it (don't they always?).
Anyway, look around if you've not already spent money as this thing is (was) out there.
Anyway, look around if you've not already spent money as this thing is (was) out there.