First and foremost, you must sample a wide variety of music regardless of the medium you wish to use, and it is helpful to have some guidance because the field is so wide open and varied.
You can get good guide books on collecting. One of my favorites is "1,001 Classical Records You Must Hear Before You Die." Not only does this book list very good recordings of specific pieces, it is organized chronologically by yea of composition so you can sample music from different eras to get an idea of what you like and what you need to hear more of to understand. Another good guide, though quite long out of print are the various Penguin Guides to Classical Music. There are many on-line listings that are useful too. Particularly if you are interested in modern, late 20th century and 21st century music, Google The Guardian lists and articles on the subject.
It is impossible to make specific recommendations because each individual's tastes and interests vary so widely. A friend in my office wanted an introduction to classical music so I lent him a selection from Renaissance to 20th century music. His favorite was something unusual: Harry Partch's "Delusion of the Fury."
If you become serious about classical music you MUST have access to digital medium because there is so much more available that way than on vinyl. New music and new recordings of old music died a long time ago on vinyl. Some of the reissues on digital medium sound better than the original vinyl issue, such as, most DG recordings from the 1970's (they had great artists under contract but put out bright and thin recordings that were improved upon when reissued on CD).
