What makes the biggest difference in sound quality?

Every Gear including the Room, I think the room is very important if this is not done right no matter type of gear you use it will never reach its full potential.

Audiophile tends to fill the space with diffusors but i think a 60-40 Ratio works well.

60 percent absorbers and 40 percent difusors

Your Room,

Your Speaker (Plus Placements)

Balanced System

Good Power running to the room, Grounding Properly.

Room and Speakers obviously, however the Pre-Amp is the heart of your system which can really make or break it.