Professional speaker management systems at the top of the scale in performance all have adjustments for time alignment. When you start talking about several stacks of drivers it starts getting a bit more complicated. There is a masters level course in the subject at one ot the TX universities, professor wrote a text on the subject. I got the books somewhere can not remember the professors name.
If you setup a speaker system in a anechoic chamber you can definitely hear the change with changes in timing if you are sensitive to the issue. The crossover points seem to be noticed by most people even in the a good listening room. To me it just sounds noisy at the crossover points.
I am working out a design for amps and such a management system on a budget. If interested check out "mini DSP", they have off the shelf solutions that are not too expensive. The DSP chips they use would be a good place to start a study of the issue, easy to work with.
I like the idea that the system could be used to make a system with a series of appropriately placed divers and apply a spread spectrum digitization of someone talking and aim the system at particular person in a crowd and that person would be the only person that would hear the person talking everyone else will only hear noise. Nice toy for spies. There is a patent on such a system.
So yes, it makes a difference.