Bass Driver Size - how much better is an extra inch?

The "speed" a driver is capable of is indicted by its frequency response curve. A 15" driver pushing 100 Hz is every bit as fast as a 6" driver pushing 100 Hz and if the motor is designed correctly the transient response can be just as good or better. This concept of "speed" or fastness need to be thrown out.

6-8" woofers can perform admirably down to 50 Hz or so. You can force them to go lower but forget about energetic bass below 50 Hz. Small woofers are being used to support the tower concept that the ladies like. Using multiples helps but none of these speakers do well below 50 Hz. If you think they do it means to me that you have not heard a system that accurately produces low bass. I use a subwoofer array that uses 4 12" drivers with a total of 8000 watts in a 16 X 30 foot room. If I play a 10 Hz sine wave the entire house shakes and buzzes. You can actually see some items shaking. Everything upstairs and in the shop is singing. If I turn it up loud enough your vision blurs. Little woofers simply can not do this, physically impossible. What they really are is midbass drivers which they do wonderfully well. For the best performance you cross over to subwoofers at 80 Hz. If you use a symmetrical array around the main speakers you can cross higher. Good midbass drivers can run nicely up to 2500 Hz or so and there are many tweeters that will easily go down to 2500 Hz. So, you really only need a two way speaker and subwoofers. 
The minimum is four 10" subwoofer drivers or two 12" subwoofer drivers.
Subwoofer drivers have a hard time making it to 500 Hz for design reasons. 

Right, most people (women) do not want big speakers. 

Assuming proper driver design larger woofers can go lower and have less distortion because they do not have to move as far to create the equivalent output of the smaller driver. There is a limit. As the woofers start getting over 18 inches (some say 15") it gets harder to control the motion of the cone creating more distortion not to mention the size of the enclosure gets prohibitive for home use. I think 12 and 15 inch woofers do fine in the home environment. There are 15" drivers now that will work in a 1.5 cubic foot enclosure which is darn small.

Many people notice that for some reason the midrange opens up when subwoofers are added. The low bass gives you a different midrange perspective. If a two way crossover is used distortion in the main speaker is reduced. If a lower order crossover is use enough midrange may come from the sub to change the perspective but I notice this even with steep slopes and we are talking about 10th order. The problem with steeps slopes is the crossover becomes more obvious. The compromise is 4th and 6th order which works great at least in my case.