My two cents, conductor and sound engineer matter. So when I write about "recordings," I am thinking of the best combination of recording quality, artistic expression through recording, and the interpretations of the music by the conductor.
Of course, you'll want to consider which periods and styles of music float your boat. "Classical" can be anything from Renaissance choral polyphony to atonal 20th century, if we're sticking with a Western definition.
An inexpensive vinyl journey is to explore anything on the l'Oiseau Lyre (Decca) label, in particular the work of Christopher Hogwood and his early music group. His Mozart, Bach, and Vivaldi interpretations are superb, and the recording is, to my taste, among the very best out there. The Mozart symphony box sets (7 vols.) are the best Mozart symphonies, imho.
Others have mentioned Deutsche G., which is certainly huge. But I prefer, as recordings, Philips, Decca, Erato, and Argo. For more recent smaller labels, again for earlier music, try Harmonia Mundi, Hyperion, and Gimell. Also, there is a lot of inexpensive classical on CD too!
Lots of great advice here about how to expose yourself to and learn about classical music, but don't necessarily be put off if you don't like a piece or a compose at first; it may just be the recording.
Paul
