The off-axis response of the tweeter + the room are the balancing acts. I do think the idea that all speakers need to be laser calibrated to point to the listener is a myth we should deal with more.
The correct answer IMHO is balancing your preferences with imaging.
Some speakers are meant for little toe-in (Revel, Magico) and throw a great image over a wide area IF they are not constrained by side walls.
Some speakers have raspy tweets on-axis which sound great off axis.
I've found Focals to need minimal to no toe-in for instance.
Also, it's not just toe-in, but some speakers just sound better on mid range axis, so speaker tilt can really vary. B&W's are IMHO like this.