Why am I experiencing listener's fatigue?

You are probably midranging yourself to death. B&W, tube equipment, Cardas all have reputations for great midrange, but a bit laid-back sounding. You might need more excitement at the frequency extremes. If your rolling off the ends, you might be turning the volumne up to make up for the lack of excitement, and the loudness might be causing the fatigue. Maybe some of the tubes are weak, which would also contribute to this. This is just a suggestion, as I'm not familiar with all your components.

If it were me I would switch to a pair of Martin Logan
Quest Z's which would sound incredible with the VTL Stingray.Also a VPI Table would also improve your ears greatly.

None of the prior posts focused on your amplifier.
Try a Conrad Johnson CAV-50 instead of the Manley. It is an EL-34 based tube integrated which may be better match for your speakers and CDP.
Good luck!

Try some DIY room treatments before you start changing your gear.

i'm NOT assuming anything here :-) & I think Onhwy61 & Timf are on the correct path:- Vertigo, what's your definition of listener's fatigue? & what symptoms are your experiencing? The suggestions depend on your answer.

(I did own the B&W DM604S2 for 3 yrs & I know that they are not very discriminating speakers compared to the N-series. Never-the-less I never experienced listener's fatigue. I did, however, experience "white cheek" syndrome from sitting in my chair listening too long. LOL!
I have another friend here who owns N803s driven by a SA-250 Mk4. here, too, no listener's fatigue & I've been over to his for extended periods of time).