You're not crazy. I have a $2400 streamer and a $500 fed into the same DAC so I can instantly AB and I'm struggling to hear a difference. Or I believe I hear one, only to go back and explore again and poof! The difference is gone or subtle at best.
But I keep trying. I realize we trade in nuance and subtlety in this hobby. If I got a subtle but definite improvement, I would pay good money for it. My system is highly resolving so every bit matters (to me).