How to choose speakers that won't overpower your room?

Being familiar with REW you should be able to accommodate any speaker. What causes people to experience room overload is simply the inevitable room modes. It’s the peaks that are responsible, peaks that can be 15/20 dB higher than the average in room SPL. These peaks will obviously take much longer to decay and this leads to the 3 complaints of: boomy bass, one-note bass or slow bass.

Get the room dialed in and that trio of problems disappears. Room treatment and a couple of small subs will sort out not only the bass but the full spectrum

The REL sub you own I think has phase flip only and this complicates matters in so far as finding the best position for it. Consider something like a pair of SVS SB1000 PRO or any other subs that provide variable phase adjustment so that you can place them where they are unobtrusive. or where convenient and not be restricted. It is better to have sealed subs and not ported because ports are frequency invariant and are bass sources that can’t be tuned. REW and mic are your best friends. Choose the speakers you desire and note that when the peaks are tamed and the nulls filled in you will be hearing much more than before and even at low SPL will sound great. Good Luck at the show.

Thanks for the additional comments, I will be pursuing some room treatments. @sawbuck I built a custom Pi/Digi streamer a few years back with good success. I nigh have put it back in to see how it compares to the Innuos. 

The Pi /  Digi Signature is an excellent solution, I used Moode Audiio as my streamer software. At the time I was also using a MiniDSP SHD going out to a Crown Amp with very good succes

Just stumbled onto this thread, very interesting responses.  I admit to having always believed the "small room = small speakers" axiom, so it's really enlightening to see so many responses indicating that this not necessarily true with some of you successfully running floor standers in what I'd consider a small room.   I stumbled here because I'm considering upgrading, and my room is 11x10, so I'm in just about the same boat as the OP.  I've been trying to rationalize some Wilson Benesch Endeavours in the room, but my "old" knowledge keeps getting in the way, with the continuing thought that they're just too big even though they're technically monitors on stands.   I don't mean to hijack the thread in any way, but I'll just say that it's a bit reassuring to think that maybe I don't need to be so concerned about WB Endeavours in an 11x10 room and I'm glad I found this discussion.

I gave up on subs, maybe it’s in my head but I convinced myself they only blended with my speakers at a specific volume! I even got so bad I bought one of those velodyne subs that came with a volume remote, still no joy, but that’s my insanity. 
 

I was offered a pair of Dali Kore’s for a really hard to refuse price, but refuse I did because I figured they’d overload my room, damn. 
  I heard the KEF ref 3’s in a few rooms at axpona and if anything I thought the mids needed to be tamed most in those little rooms. 
 

Idk of any math calculations to predict bass response, most of the time it’s room nulls I get. You state to listen at fairly low volume so you ought to be with any practical choice