If you listen to only certain types of music and never exceed certain volume levels, then I imagine the truncated frequency response, limited dynamic range and high distortion levels of an 8" bass driver really wouldn't present a problem. If you want to go loud, you'll need more than 8". If you want to go low, you'll need more than 8". If you want to go loud and low, then size really does matter.
