I've done probably 50 sale transactions out of my home. Never had a bad experience. Probably the worst thing that ever happened was a guy who stayed 45 minutes, talked up a storm, then told me he wasn't going to buy... he was definitely an audio person, and there was no 'ulterior motive' - just, don't waste that much of my time; you usually know within the first 5 minutes if you are going to buy or not...
And I have met close to 50 mostly very nice folks and gotten to share audio info / stories / etc. with them; a couple it ended up I bought something off of them later on because I found out they had something I wanted when they came to my place.
While I can certainly understand that WHERE you live might have some bearing on how cautious you feel you need to be... this does show that there are a lot of paranoid people out there. And (full disclosure) I say that as a 60-year old white guy, fairly clean-cut, living in UT (a gun-friendly state) and owning firearms myself. I have never felt the need to carry a firearm when selling anything out of my home. I HAVE gone to a few homes and had that done to me (homeowner answered the door packing and making it obvious) and, quite frankly, those particular folks weren't the same "very nice folks" I mentioned earlier. Instead they acted nervous, suspicious, and quite frankly if I knew what I was walking into - I would have decided against being around them, as I felt like A) they seemed to be generally 'angry' individuals that were almost looking for an excuse to use their weapon; and B) seemed like at least a few of them probably had legitimate mental issues with paranoia, etc.
But, in the end, I do agree with others who have said you should do what feels best for you and your situation. The only thing I would add is that I think it would be extremely rare that someone is pretending to want to buy audio equipment as an excuse to either rob you or 'case' your home.