Many issued recordings are flawed in some way or other. Perhaps most are. If you are a certain kind of purist who wants to hear exactly what the recording as issued is presenting, then by all means get the most "accurate" components possible, which is something a lot easier to measure with DACs than, say, loudspeakers. If you want to sit entranced by most if not all of your collection, you may prefer something else.
For instance, I remember the early Deutsche Grammophon recordings that were recorded digitally and issued DDD on CD were usually pretty much unlistenable. Things have improved since then, but still...
Is orange a "distortion" of yellow? If you happen to like the color orange over the color yellow, is that wrong?