What else are you going to do in life? Live, overcome obstacles in life, take care of yourself and your family. Enjoy life. what does that mean? If you love music, then by all means, listen to music. If certain electrical equipment or systems drive you out of the room, then the upgrade path it there to fix that until the sound is wonderful. That is not or should not be the only enjoyable thing in life. Art, music, traveling, cooking, spending time with friends and family, helping others, bettering yourself, etc. Here we go. Now you are talking. If it makes you happy and doesn't hurt anyone and you can afford it (that's the hard part), then it really shouldn't be anyone else's business.
Enjoy your system, music and life. If you love watches or cars, either collect them or go to car shows and enjoy. Worrying about what other's think is not cool. There will always be something that others think you are doing that is stupid. It is not their life.
Is it worth it? only you can answer that about you. Did listening to music and upgrading to get "there" with my system hurt others? or prevent me from taking care of my family? Nope. Do I enjoy coming home, turning on my system, getting a good book and a good glass of wine and listening to old and new music. You betcha.
enjoy