All drivers have a bit of a built in "dynamic compression" due to the suspension. So a high efficiency of the same sd, may in fact be more dynamic as it is in a lower excursion so not suspension "compressed". This may not be noticeable at all (should not be) at medium levels if you chose the correct speaker, but as you crank it up, it may become more noticeable. This is still within the gap. A cheap or poorly designed driver will also get increasing compression if the coil moves out of the gap. To be complicated, gap strength is not linear.
Amplifiers also have a non-linear gain with respect to power, and it can differ with different loads.
So to each their own and enjoy. Build a Kleinhorn and power it with a Schiit REKKR. Use an active crossover on your mains to take the deep bass out of them so they are not trying to produce midrange while approaching XMax. Pick an amp with sufficient overkill as to be on the more linear portion of the transductance curve and sufficient dynamic current.
Do be careful lumping technologies together. All ESL are not the same, all direct radiator, all horn, all panel...