You are in for a great adventure!
The are some caveats: performance quality and recording quality rarely match up. You can find most great performances in mediocre quality, and the best quality recordings often have material that (as many say) should never have been recorded, and we would not have lost anything. As the saying goes, they most often press only that material in high quality that would otherwise never sell. Vice versa, if people want to hear it, why bother with quality, just press more of the weak recordings. This is a kind of pun, but unfortunately stems from life. For example, Angel and Seraphim have excellent interpretations of the Baroque era, but their recording quality is generally the lowest of all. (Except for the newer Angel digital, which is pretty good, and you will mostly find that on CDs - but their older analogue recordings are quite poor in quality, using second hand master tapes.)
First, your task is to figure out what you like.There are landmarks, who are the giants of classical music.
Bach, Mozart, and Beethoven is the immortal trio upon which practically the rest of classical music is founded upon. I would advise to get acquainted with them first, to get a strong foundation.
The modern composers (such as Stravinsky, etc) offer a very different perspective.They mostly use the same instruments, but the purpose of the works is just about the opposite.Bach, Mozart and Beethoven is all about meditation, beauty, harmony, exploring your inner universe and linking it to the outside world. When you find such works in a good interpretation and good recording quality, it leads to powerful healing and inner growth. Restores you, even saves your life when you have a super stressful job or going through desperate times in life. They also give inspiration - for example, Mozart was Einstein's muse...
Modern composers do the opposite: they explore the outside world, take you for an ecstasy ride, they delve deep into excesses, horrors and terrors of history. Drama, drama and more drama - the more grand, ostentatious, unpredictable, the better. They are suitable when your life is spot on, no worries, and you need some excitement in your life, and want to experience extremes. Listening to much of this music will gt you energized and ready to act, but often leave you unnecessarily agitated and unbalanced. Just to stick to Einstein's opinion, he used very harsh words to describe many modern composers, I am very civil with my descriptions.
That is the general tendency, but there are always exceptions - there are plenty of healing works in modern music (Gustav Holst has that effect on me), and some WTF moments for baroque composers. But, the general tendency is there, and the vastly different expectations of the ages, what people expected from their contemporary music in eg the XVI century and today are vastly different.
I second HM, DG, Telarc, Teldec, EMI, DECCA, London, Telefunken, RCA LSC as superb sources. You can also go for Hyperion, Unicorn, Delos, L'oiseau Lyre, these are superb, smaller labels of high quality.
Although I have a sizeable classical CD collection, I mainly listen to classical on LPs. Do not be afraid of the condition... yes, sometimes LP used condition is terrible, but you can find pristine early pressing treasures which completely redefine quality listening experience. Get 2-3 copies of the same LP for 1-2$ each, keep the best and you are still under 5$ budget per record. It will be a great experience: how vast a gap there is between two copies of the same recording. While digital format (to me) is perfect for pop and electronica music, and modern sounding recordings, but classical connects much more on LP. Digital somehow convey too much control: can work well for modern composers, but spoils Baroque ambience completely. Although there are some labels that do a pretty good job on CD with baroque classical music, Dorian comes to my mind with fantastic classical CDs.
For LPs, a secret is boxed sets. They cost very little, and are most often in mint condition: maybe Side A of the first record was played once, and that's it. That's the flip side of box sets as well - you can have the best recordings, and they are worth next to nothing if you try to sell them. The exception being the sealed box sets, which go for ridiculous money. Once you remove the plastic cover, you have knocked off a zero from value.
In your place I would start discovering Bach first, then a little Mozart and Beethoven. Then Tchaikovsky, Stravinsky, Ravel, Holst, Mahler, Shostakovitch, Mussorgsky. You can also expand to medieval music and renaissance (going back in time). That's also a very different kind of music
Also, be aware of the conductors. Their interpretation impacts the piece greatly. For example, a Mozart piece can be conducted to sound as a Beethoven piece, or Bach to sound like Khachaturian. There is a great nimbus for such performances (called modern interpretations), but you want to stay away from those at first, to get a feel for the core and the tradition.
Have a great journey!