I second the first post's recommendations of Academy Records (on 10th street between Lafayette and 3rd. They'll let you try records and have a good selection and are very knowledgeable). If you're brave, you'll go to this place:
The Thing
1001 Manhattan Ave
Brooklyn NY
They have probably the largest collection in NYC, but it's a pretty gnarly experience. I think everything is 2 bucks and under, and I'd say bring some rubber gloves and just start sifting through. You can find some CRAZY stuff. I know this sounds crazy, but I'd also check craigslist. I purchased a whole collection of 900 records in mint condition from some guy who just wanted to dump them. Costs me 50 cents a record, and I'm talking primo stuff!
Otherwise,
Junk
Intersection of North 9th St. & Driggs Ave
Brooklyn, NY
The Jazz Record Center
236 w. 26th street room 804--huge selection
Manhattan
Westsider Records (big collection, good stuff)
233 W. 72nd Street, New York, NY, 10023
(212) 874 1588
Anything else is on google!
Cheers!