You ain't crazy and you ears are fine. As you mentioned with your experience as to headphones, speakers rule the roost. I think CVs are only OK for rock n roll and are a mess for anything else. There are plenty of good, and inexpensive, speakers out there from the likes of PSB and Paradigm which would bring out your music much better.
Have fun,
Charlie
Have fun,
Charlie