In my opinion it would absolutely be an issue but perhaps not as you may anticipate regarding timing, tempo, and signal arrival.
A 4 minute and 36 second musical track will still take exactly 4:36 regardless of the matching or mis-matching of amps.
If you had a 'fast amp' for the mids and highs, you would simply have potentially a more sluggish and ill-defined sounding bass region. Or vice versa.
The fast really refers to the amp's agility and speed to go from a dead silence to an initial attack, esuing decay, and picking up hopefully every little bit of nuance and inter-transient silence in between.
As far as I know, the slower amp's signal will arrive at roughly the same time, but it most likely will skip some to much of the nuances and inter-transient silence that the other amp will pick up.
So in the end, the slower amp will sound as though the tempo is slower even though it's the same. But only because it's running the musical information together.
In my limited experience, most amps are not fast.
Therefore, if you have the wherewithall to purchase what is known as a fast amp, I'd do so without hesitation. You're system will be that much better off. Assuming the fast amp does not have other significant deficiencies.
-IMO