I would imagine at some point, the limitations of the receiver will outweigh the benefits of purchasing higher-end speakers?Yeah, IMHO you’re already there as the AVR just doesn’t justify higher-end speakers and will be a significant bottleneck to good 2-channel performance. To meet their competitive price points, mass market AVRs use cheap components (power supplies, etc.) so both its preamp and amp sections are severely limited in their respective performance levels. And no, it’s not hard at all to hear the difference between a mass market AVR and decent dedicated stereo electronics with good speakers. If you’ll only be using the 3700 I wouldn’t spend more than $1000/pair on floorstanding speakers. I’d recommend the SVS Prime Towers as they have an excellent Prime Center speaker to match and that’s much better than most center speakers near its $350 price, and a good center speaker is really critical to fully enjoy the HT experience. And SVS offers a completely risk-free, 45-day trial so you can see if they work for you in your home and with your electronics. Read the reviews that are universally positive.
However, there is another option if you think you might get more serious about 2-channel down the road. Your Denon has pre outs so you could buy a good stereo integrated amp with a HT Bypass feature and seamlessly integrate it into your HT setup (with the 3700’s pre outs) such that the integrated alone powers the front L/R speakers for 2-channel and the AVR is completely out of the signal path. If that’s a possibility then you could absolutely justify getting much higher-end speakers— it just depends how serious you wanna get into 2-channel. Just my $0.02 FWIW, and best of luck.