Listening critically also gets easier with time / experience for a few reasons. It takes time to build the experience of what a realistic presentation sounds like - tonally, spatially, and effortlessly. And the better one’s source material, room, system, and hearing (or at least awareness of their personal hearing capabilities) are, the easier it is to understand the performance of any of the variables listed so that we may be able to critique what is contributing to what, or what is falling short.
Yes, it can be truly exhausting at times if you overdo it, but it’s a journey, not a destination, and one that, should you choose to accept its mission, you should traverse at your own pace. The key to success lies in motivation and the means to improve the variables for further fulfillment / enjoyment. Once either of those start to wane, it’s time to take a step back and stop taking it so seriously, and just try to enjoy the music. If you can’t do that, you’ll at least either have the motivation to find the variable that needs improvement so you can find more fulfillment, or you just need a time out from listening to music for a while.