Vinyl (under most comparisons) sounds smoother and more natural than CD. You trade off some surface noise, detail, and distortion, but in many (or most) cases vinyl is a more pleasing sound to the ear. What's more, there's lots of good vinyl (little recent/popular music, but lots of old classical and 60-80s titles) available for very inexpensive prices (in San Francisco, there's a good selection of nice $1 records, many used record stores, and lots of good reissues out there).
On the downside, you have pay for another front end, you must put a lot more care into cleaning and storing records, and you can't pop them in the computer or car when you want to listen outside of your home system. I don't think vinyl is the right move for everybody, but I'm very happy having taken the plunge recently (I had only a few old records left over).
The best way to get into vinyl on a budget is to find a nice quality used rig and receiver. If you can afford a Rega 2, 3 or NAD 533 (about $100-500 used), I think that's a very good starting point. For somewhat less money you might find a decent Dual, Pro-ject, or Thorens rig. If you need to go new, there is a significant price bump, but I like the Music Hall MMF-5 at $500 or the Rega 3-2000 + Elys at about $700. The Creek OBM-8 is a good phono stage for $200 new (some people have reported RFI problems, but I don't have any and live in downtown SF).
Regards, RJS