Some people are just weird, so you do have to allow for that random factor.
But:
(1) Don't judge a book by its cover. I never dress "up" for an in-person used audio purchase. I think it helps if I need to negotiate the price down. Plus I drive my pickup as some things are hard to haul.
(2) I'm sure it's not a bulletproof method, but I have found it is very easy to tell the con-people from the actual audio lovers with just a little bit of conversation. You have to go beyond "non-smoker first owner" and just ask what kind of amp are they using with those speakers...that type of thing. I've met some extremely nice people. One time in buying a McIntosh piece I met the guy who ended up custom-wiring a lake house I was building.
Caution and awareness are very recommended. Paranoia and rudeness may rob you of a good potential experience, however.