If you slightly reduce the amount of energy to the top by using a smaller ga cable you are disrupting the balance of the speaker the way it was designed.
Hifichic, I agree with the electrical theory but I question whether there would be an audible effect of the two different sizes because the current demand is typically lower to the high frequency drivers due to their higher sensitivity and lower resistance. I think the penalty could possibly be higher if the MF/HF and LF wires are the same gauge but the resistance of the LF wire prevented adequate current transfer to the generally lower sensitivity bass drivers, especially in the case of long runs. That said, most audiophile wire is likely oversized for the required current flow, and I am sure the 6awg of AZ's Double Barrel is probably not needed....except for welding.