Ketchup,if you have the time,you can hop on a NJ metro from Penn Station,and in about 45 minutes you would be in Princeton,NJ(take the Northeast Corrider line.Very easy).Here you wil be within a very short walk from "THe Princeton Record Exchange".Though this LP/CD shop has been better in the past(only the classical section has diminished),it is STILL a real haven for very reasonably priced jems,in classical and particularly JAZZ.The jazz section is quite large,while you can find a good variety of classical discs in the lower bins,requiring some back work.A very huge CD inventory as well.I've been going there for 20 yrs.
Most importantly,their pricing "blows away" the NYC prices,and the sales tax is lower too.I just picked up a supremely mint copy of Oistrach/Fournier doing the Brahms Double cto,on the original,first pressing Columbia(EMI)Sax series,for 75 bucks.That is a STEAL,as this disc sells for way over 200 bucks on all the lists I've seen.This is only one example.Standard English Londons abound,for about 0ne or two bucks a pop.The Jazz section has a huge variety,with hard to find titles,at very reasonable prices,as well.Not to diminish Albert's recommendation of A Classical Record, a fine inventory there,but they are quite a bit pricey,to a seasoned collector.You will pay top dollar.Their prices are "fair" to those used to paying top buck.
BTW-Princeton is a stunningly beautiful little town where you can have a lovely lunch or dinner,while looking for musical gems.I do this regularly,and it enhances my lifestyle.
Best of luck in the "Apple",but the Real Deal is in Joisey!!