My interest in music is mostly in the jazz genre. I have often bought records that had music I did not fully appreciate and then pulled them out later to find my taste had either changed or matured to the point that I later found them enjoyable. These got moved from the back of my collection to the front and I as happy to have not disposed of them. I used to pull out and play Miles' Bitches Brew once a year until I finally understood it. Consequently, I never throw anything out unless it is damaged or replaced with a better copy.
Throw it out!
I just discovered a technique that will save a lot of aggravation. When I find a record that is not satisfying me sound-wise (or sometimes musically,) I dispose of it or relegate it to the inactive file. Rather than having it available in my collection to aggravate me, it gives me great satisfaction to get rid of it!