Do I really need to replace avr with an av processor?

What you are talking about will indeed be an improvement. But not as much as completely bypassing the avr. Because the sad fact of the matter is there is nothing you can buy worse for sound quality than a avr. Even running the signal through it like you're talking about, just the connections, switch, volume control and pre-amp alone will mess it up. Of course you will never know this, until and unless you try and compare and find out how much better all the same stuff sounds with an ordinary 2-ch integrated instead of the avr.  
You're almost exactly right about movie sound, by the way. It really does help if its quality. But it does not need to be surround. Not at all.  https://systems.audiogon.com/systems/8367

The best solution for a mixed use system is to buy a stereo integrated amp with HT bypass feature, that way you can plug your 2 channel source (DAC or phono) in one input, and plug your AVR into another input, both will funnel sound to your front speakers depending on which source you select.