Help Me Build a Vinyl System

As far as the question about how many records you own goes, while being a fair question, it's kind of a 'what came 1st, the chicken or the egg". The common train of thought is that one shouldn't have more money invested in the equipment than in the music BUT....you gotta start somewhere right!!?? You could easily go to Audio Advisor and get a decent start-up turntable AND Phono preamp on the cheap (much less than 5K!). Than you can go get yourself some records with whatever's left over. Just be prepared for the madness that will take you over once you realize you prefer vinyl. THAT's when the money starts to flow!

The man wants to try vinyl and has some cash to do it right. Good idea. There are millions of records around, that's not a problem. Cleaning is not a problem either with a $600 machine. There are no big problems with vinyl, just some playing with the set-up. The difference with well recorded classical and jazz compared to digital should be huge.
For $5000 used there are so many options. What kind of sound are you looking for?

Go for it. Get a VPI.

Here's my recent experience - I started from relatively zero and now have a fantastic vinyl playback system and decent record collection for under $5k (used VPI Scoutmaster, Soundsmith Zephyr, Sutherland Ph3D.)

Vinyl's gotten me both amazing sound quality and tons of fun.

You can get mixed lots of 100 classical LPs here on Audiogon for about $2/record, $5 - 10 for fancy labels. Mixed bag, but part of the fun is exploring what's there. I'd never pay $20 each for 4 different recordings on CD of the same piece. But, its easy and great on vinyl. On the other hand, the high quality reissues - Music Matters Jazz, Acoustic Sounds, Mosaic, etc - are simply amazing quality, though expensive ($50 each)

Let's forget your statement "...to start with"
You gonna get it right once and for ever!
1) : Lenco L75 (less than $200 ebay) + (about $350 ebay) plinth with removable armboard (jec965)
2) : Zeta tonearm ($900 Audiogon)
3) : Denon DL-103R Panzerholz uwe body soundsmith level 1 ($375 Audiogon)
4) : Einstein TT CHOISE phonostage ($2650 Audiogon)

Ps : You must be very quick to grab 2) & 3) ASAP !

Lots of good sound advice.. Rauliruegas spelled it out perfectly...I started in the same position as you are now.
I have spent approx 15K on my vinyl system and lps in just over a year. My budget was 5K...good luck!
I hate cleaning records too!