What sounds best: Vinyl or CDs

IMO, it all depends on the recording and on the playback equipment. Since you are just getting started, I would start investing in equipment that would really give your system a boost in SQ. IMO, going from a Yamaha CD player to a rega isn't going to get you ahead. I would invest in a really good quality external dac then hook up a CD player, music server/streamer to it. As for vinyl, get your wallet out. You will need a good phono preamp (much better than using 1 inside a receiver/integrated amp), then a good tt/arm/and cartridge. You see all these reviews comparing vinyl to digital and the reviewer always prefer vinyl. But when you look at their analog equipment, they can have up to $250k in a tt/arm/cartridge/phono preamp, and they compare this setup to a $15k digital setup. It takes quite a bit of $$$$ to build a good analog playback system, much more than building a really good digital system. I propably have 3x more invested in my analog setup (retail prices) compared to my digital equipment and I have vinyl albums that blow away my digital albums and I have cds/dsd albums that sound much better than vinyl. 
1 option when you go analog, you can setup a 2nd arm with a mono cartridge and get superior sound over its equivalent stereo album, if the album was produced originally as mono, not converted, for example, miles Davis kind of blue and the Beatles mono collection.

I’d like to offer my two cents as well. I own an Avid Diva II SP TT with an Audio Technica AT ART-9 MC cart mounted to an SME 309 tonearm setting on a 3" thick maple base with sorbothane feet. I use a Simaudio Moon Neo 310LP phono preamp for the setup. I also own a Bryston BCD-3 CD player. Both pass through a Parasound JC 2 BP preamp and Parasound A21 amp. My speakers are Focal 1028 BEs. In the world of analog, my TT setup is maybe 2.5 stars, my digital is perhaps 4 out of 5 stars. With that out of the way, I’ll say that my low end analog at three times the cost of my CD player does produce better sounding music. My statement strongly depends on the quality of the mastering of the media. The cleanest,most precise vinyl will edge out the very best of my CD collection, by a little bit. The point is, an analog setup will win a shootout with a digital setup most of the time, but there’s a steep price to pay for equipment and proper setup. You could definitely get more out your system with a better cart and preamp that won’t break the bank. A move up the Ortofon MM line to the 2M Black would be an enormous improvement from where you are at. It is probably the best "all purpose" cart I’ve ever owned. The 2M Red to the 2M Black is an honest night to day change  Attach a SEPARATE phono preamp (Graham Slee makes decent stuff that’s MM only) and you’ll have something that will almost certainly put a smile on your face when playing well recorded vinyl.

It all depends on the system(s) you are using and the quality of the recording. Overall, a record should not sound "much worse" than a CD or digital file. In my current set-up I prefer vinyl over CD or SACD. I am sure this is due to the equipment used. I use a Marantz UD7007 for CD and SACD playback and a Denon DP59L with ZYX Fuji cart for Vinyl. In my set-up vinyl often sounds better, but again, it also depends on the recording. To compare, the Marantz is a 700USD device for playback and the Denon / ZYX combo is over 3000USD. I find vinyl more dynamic, alive, and with a better sound stage than CD or SACD in my set. As mentioned in several other posts it is crucial to have your arm and cart set-up properly and have the right cart / arm combo. Best to use a table specific / cart specific protractor for this, or alternatively use a generic protractor. Also, if you should have some budget to up your cart I think it would be worth the effort. Hope this helps in your search for better vinyl playback.

Recently I became aware of the Dynamic Range Database available on line.  Check it out.  Interestingly,  according to this database, the vinyl version always scored highest on the scale of dynamic range over the many CD versions.   Some versions of the same music in CD might match the vinyl's dynamic range, but rarely exceeded the vinyl.  I believe CD compression explains much of the perceived inferiority of this format.  Generally speaking, I believe a typical LP would rarely sound worse than the CD version, but a well engineered CD can come close to matching the LP,  assuming the best playback systems from both formats. This assumes dynamic range is an important criteria for your listening, which for me, more dynamic range = more realistic.

However, on super extreme, well engineered sources, with the very highest dynamic range, and on an unusually capable playback system, I think the story changes, and digital can exceed vinyl and tape.   My professional level system (JBL M2, JBL Sub18, 3 Crown ITech5000HD stereo amps) is capable of super phenomenal  dynamic range (1250 watts to each compression high frequency driver, 1250 watts to each 15" woofer, and 2500 watts to 18" sub) .  Dynamic range capability was this system's major design criteria, and it is frequently used in movie and music production.  I have carefully tuned my listening position using measurement, speaker/sub placement, and acoustical room treatment to handle this extreme dynamic range capability.   When I play the SACD of "Bucky Pizzarelli-Swing Live" by Chesky, which has the highest dynamic range in my collection, and is what I consider a "super" disc, and play it at a generous SPL, the effect is jaw dropping.  I think, how is that even possible.   I don't believe vinyl is technically capable of this super extreme, and considering the base notes being produced in the room, I wonder if you would be hard pressed to  even put the TT in the same room as the speakers. 

Unfortunately, really, really good sources with extreme dynamic range are rare.

It depends on the LP or CD.  There are vinyl recordings that sound awful and ones that sound great.  Same with CD recordings.  Some CD recordings are phenomenal.  Dick Hyman Plays Fats Waller - Direct-To-CD is as close to live as one might ever hear.  Yes it is a CD.

I think where this gets tricky is when you have a CD and a vinyl of the same recording.  If the vinyl is brand new and the best pressing done from the batch - possibly an early pressing you may find the LP sounds better.  However every time you play the vinyl you are shaving some of the fine detail off of it.

Perhaps a better question would be - can you hear the difference between a new vinyl and a digitized recording of that same vinyl.  Using a fancy Benchmark ADC1 A-to-D I "record" my records.  So far I have not had 1 person able to hear the difference between the LP or the Digital file.  And I have a very accurate system - full range electrostatics, Benchmark mono-blocks etc.  

I like to tell people there are different types of playback systems.  One type is the analytical reference system - this is what I have in my studio / playback room.  The other is a performance system - this is what I have in my car - it doesn't actually act as a playback reference it creates its own performance.  So for some things the music actually "sounds better" in my car!  But that is because the car is hiding shortcomings and exaggerating other things.  Sort of the same way a guitar amplifier and speakers are part of the music not a playback component.

So to get back to your question...  If the music you are listening to is only available on an old LP then that is what you have to listen to.  Conversely if only available on CD then there you go.

I have three music sources in my system:
A turntable, a CD player and my computer.

I have three playback systems:
My good one in my studio.
My car.
My iPhone with headphones.

Given the fact that I don't have a turntable in my car or iPhone it reduces the choices down to digital.