Keep in mind that toe in alters high frequency response, the level of side walls reflection, and the relationship of direct and early reflected sounds.
^^ This! +1
Early side wall reflections are interpreted by the ear as harshness. So you toe in the speakers to reduce the amount of sidewall reflections, allowing the direct radiation to mask the side wall reflections. Sometimes you might even have to have the speakers cross in front of you to do this.