In my opinion the "bookshelf + sub" format arguably has greater potential because of the greater low-end extension/output/flexibility that you get with separate subs. But the specifics matter - most "bookshelf" speakers are designed to have acceptable low-end when used without subs, so they trade off efficiency and/or max SPL capability that might be nice to have if you're going to use subs anyway.
So the "best" bookshelf speakers might not be the "best" for this application, because you really don't need your bookshelf speakers to extend much if any below about 80 Hz. And if home theater is on the menu, then max SPL probably matters. So imo you'd be looking for "bookshelf" speakers that go pretty loud but don't necessarily go very low. Not that these are the only things that matter of course, but imo there is no point in spending money for deep bass from your bookshelf speakers when higher output capability would better serve you.
Duke
dealer/manufacturer