I tend to not be so anal about seating and positioning, but then I use speakers that are somewhat forgiving as they are time and phase aligned.
Duke Elliington said:
'If it sounds good, it is good'.
So, if your 'distracted' listening moments sound good, I would leave well enough alone.
Bob