Speakers "Too Big" for room

Reasons I can think of:

1. too much bass
2. Incoherent sound.  Large speakers are usually best listened to from a distance for the sound to “gel” together well.  Some larger designs with concentric or more closely located drivers may sound more coherent in a more near field listening scenario.  

You can do it effectively, just lots of acoustical treatment / dampening may be required so not to over load the room. You can basically acoustically enlarge the room by heavily dampening the walls, floors, and corners. The volume at which you listen will also play a role in how much treatment is required. 

I agree with @ghdprentice …

He has described my current set up…

OP, Go for it! 

I love the infinity line and have many.you would have to spend alot of money today to replace something like those infinity speakers.enjoy the hunt.

The fundamental issue is near field listening to large speaker arrays designed to use power is like sitting next to a jet engine. Hence the art of bookshelf speakers.