Alot depends on what you are looking for out of your system.
If you are a "bass freak" and must have prodigious bass response down into the lowest frequencies, then the single driver route is not going to be what you want, at least at this time(unless you use a subwoofer, which is a "band-aid" anyway because integration becomes a problem).
If you feel that you can be happy with a bass response that gets down into the 40-50Hz range, then single-driver speakers can become a contender for your use.
The advantages you speak of are real, and there are real reasons for them. High-efficiency is a factor, but there are also other factors in what makes the single-driver format attractive to some listeners. It is coherence, imaging, soundstage, very fast speed, transparency, open and non-congested sound, excellent driver control, and an overall "naturalness" to the sound quality.
However, these drivers are not perfect, and it is a hard task to get such a wide frequency range out of one driver. Therefore, there will be some anomalies in the frequency response over the wide range, and also could be some other anomalies not related to frequency response. Also, single-drivers, due to the limited driver area(typically 8" diameter) have limited SPL capability, generally topping-out at around 106-108db max.
So, like anything else, it is a "trade-off" of getting some advantages, but having to accept some disadvantages. What it boils down to is, if you can accept the reduction/loss of the lowest octave of the spectrum(lowest bass) and some relatively small irregularities in the frequency response, and a somewhat smaller "sweet spot", and the limited SPL, in order to get the advantages, then you are someone who could live with a system like this.
In my mind(and with my system), these trade-offs are very worthwhile. I feel that my Lowther EX3/modified Voigt Pipe speaker system brings me closer to the music in a lifelike way that I have not experienced from many other speakers.
However, it is important to have a proper amplifier for this kind of speaker, that presents a very coherent signal for the speaker to work with. Also, the high-effeciency requires that the system work very quietly, because any noise will be heard from the speaker. Generally, low power SET amps are used for this, but they have to be good ones(with very good output transformers) or they will have the rolloff and bloat that is a characteristic of amps using cheap output transformers. Also, any mismatches of components or cables will immediately be heard, since this type of system is ruthlessly revealing of any problems.
There are great rewards from using this kind of system, but it requires careful study and matching for it to be at its best. If it falls short due to improper system construction, then you will not be satisfied, and possibly conclude that it cannot be a real contender as an audiophile system. This happens quite often, as people want to "get their feet wet" with SET/single-drivers, and try to spend only a little money on the "trial", and thus get the predictable poor outcome. It is really an "expert level" type of system, that requires alot of knowledge and experience to get it right, and also may take quite a bit of money. My bare drivers alone cost over $1k.
So, to sum up, yes this can be a very satisfying type of music system, if you can live with the inherent drawbacks, in order to get the inherent strengths. And if you have the fortitude to spend the money and time to assemble the kind of system that can display these strengths in the best way.
Nothing that is really worthwhile is easy, and that is a very applicable statement regarding this topic.
I know a couple of folks who tried it, and went back to "normal" systems, but in each case they did what I warned about, and bought lower priced stuff that wasn't good enough. All the people that I know who have gone into "the deep end" with SET/Single-drivers have no desire to go back to what they previously had. Basically, once you hear what can be done correctly with this kind of system, you're hooked, and you stay there. Everything else sounds like it has a blanket over the speakers, in comparison.
