There is a story about a man who was worse off than you were. He spent a decade or two and about one million dollars creating the ultimate stereo and listening room. It had only one seat in the ultimate sweet spot so only one person could listen to this system. He burdened his family, costing them their free time helping him build the system. When he finished, he came down with ALS and has just under a year to live.
If such a dream system has only one seat possible to approach perfection, he would have been better off using headphones and having time and energy to hear the music over the quality of the system. Even in live concert halls, the acoustics varies from sometimes arbitrary sound qualities.
This is why you should put priority on headphones.