What to upgrade first for better sound quality?

Having done some fairly serious auditioning of the MAC MA600, I have concluded that it is a 'mid-fi' product at best. Treat yourself to something from Audio Research, Jeff Rowland, Conrad Johnson, Jadis etc. MACs are built like tanks, but what you want to match with the f-30s are true audiophile components. In all fairness, though, the high-end MAC tube gear is a big step forward from your integrated amp.

Did a little web surfing, but do you have a $6000 Mac MA6600 with a $3800 VPI Prime table all driven by a $700 Ortofon cartridge? Ivor Teifenbrun taught us all decades ago that the balance of system investment must lean towards the front end, the best possible table/arm/cartridge/phono pre-preamp, and then diminish investment (if necessary) downstream. I would recommend two things right away, invest in a serious phono cartridge worthy of what's downstream and then add the amazing Herbies Audio Lab turntable mat for $80. The Herbies mat on my Basis/Graham/BenzMicro/EAR $15000 front-end stunned me.

Yes I have the MAC MA6600 retail $6,500. My first inclination was to get an external phono amp that could run both MM and MC cartridges. This way I could use my current cartridge later upgrading to a MC. I was leaning towards either the Herron VTPH-2, Chinook or an AR. Biggest issue here was matching up with input impedance of only 20k. Herron looked like it might be the way to go. I do think about going tube at some point and don't believe I really 200w per channel. Listening room is say 12x18. This hobby can be daunting at times but I love it.

You can always get more hung up with things like a cartridge, then oops the tone arm is not the best for this cart. and the whole set up doesn't work optimally with this table etc. etc.. It is kind of humorous though that an $80 mat made such a big impression.
You really haven't said what the sound was missing or described this poor quality sound. It might be useful to know what your hoping to hear that your current set up is failing to provide.
If the McIntosh amp is really mid fi as described by others then you know what you need. My own philosophy is that the speakers are the most important elements of a sound system. I see how they sell very expensive cartridges now.

I'd also strongly suggest you do a LOMC cartridge. I'm a big Dynavector fan, but there are several in the $2K range that are very good. Do some auditions. Consider a Sutherland 20/20 for a head unit - very nicely engineered and very reasonable second-hand.

Good luck & happy listening!