To truly enjoy this hobby, I need to learn electronics. Any book recs?

Interesting question. If you want to become proficient at choosing components that sound great then you want to learn about the high end from Robert Harley’s book: The Complete Guide to High End Audio. This is because the electrical circuits are not really direct insight into how stuff sounds… so much of it is material science, vibration control and proprietary stuff. Correctly choosing and setting up systems is about the dimensions Robert’s book speaks to.

Now if you are just interested in electronic design for fun. Different story… go for it. Just be a bit careful, unless you are choosing really inexpensive components be a bit careful on choosing components by the specific design. 
 

I would start by learning how to read schematics.

Here is a "very" basic video which gives an idea of how important this will become when looking @ your own gear (especially basic vacuum tubes based designs).

This said, complex SS gear (like my CAL CD deck) makes my head spin.

DeKay

Rob Harley’s book is a great read and should provide a good foundation for

sound understanding then get a book on temporal masking but first get a book that explains the dB scale!

a book on the dB scale is imperative 

Contact Ralph Karsten at Atma-Sphere. I'm sure he can recommend some books on electronic theory, circuit design, etc. Forget about books aimed at audiophiles, like the one by Robert Harley; that's not technical information, it's hi-fi consumer advice.  

Your basic premise is flawed unless your enjoyment of the hobby is based on how things work, rather than music to be enjoyed, or used as an escape.