I sincerely share your grief and recommend you headphones. Where your speaker fires, it doesn't matter at all. All will depend on your neighbor tolerance and degree of sound isolation. If you have neighbor upstairs and hear him talking or watching TV program you're familiar with...(Can you imagine AGT everyday??) than probably you have an approximate measure to worry about your total listening dB. To your surprise or not, there are choices of components for every case of listening situations especially when you want to listen lower than concert or band levels. I would kinda look for pure class A amplification 10...15w per side if you're not the owner of such yet.