The headrest causes the early reflections you mention, particularly if it extends on either side of your head just past the ears. If you can modify the size of the headrest so that it supports the back of your head only, you can eliminate the sound problem. If not, you will have to lean forward in the chair to reduce the reflections. Time for a real listening chair ??
Your Listening chair
How elaborate is your chair? I use a beach chair and use two throw pillows to reach the tweeter height.Very transparent, acoustically. Only problem is the back support just ends up little below my neck level.And does not provide head rest-which is painful for longer sessions. My question: How important not to have a head rest on your chair? I have experimented with a head rest and sound becomes somewhat non-transparent loosing its ambience. Without the head rest sound is more open and natural.