When I am in the mood to listen, which is quite often, I hardly ever have the problem of music "not speaking to me".
It is more often the case that my initial selection may not speak to me, but I can play something else, which may speak to me.
My tastes are somewhat particular, in that I only tend to enjoy music that has the following attributes: very high levels of musicianship, deep and broad ranges of emotional and/or intellectual content, fairly to highly complex, has the ability to take m on a journey.
Once those criteria are met, I have a pretty broad range of music I listen to, i.e.:
Prog - classic symphonic, Zeuhl, avant-prog, Canterbury, prog-metal, prog-folk
Jazz - fusion, post bop, avant-garde, chamber-jazz, M-Base
Classical - atonal, serial, avant-garde, Spectralism, 12 tone
So, if I am not in the mood for something tonal in the prog realm, Yes or PFM for example, I may be in the mood for something more atonal, Univers Zero or Thinking Plague.
Or I might switch genres completely.