I think you should do both eventually, but in your system, the amplifier would probably get you the biggest bang for the buck for now. There are many fine amps made by McCormack, Sonograph (and Conrad Johnson who makes it), Bryston ST Series, (and many others) that would be a nice step up, for not a lot of money.
In addition, one way to upgrade the 2 channel audio CD performance would be to add an external DAC (Digital to Analog Converter) and use your DVD player as a transport to play the CDs. There are used MSB Link DACs for as little as $150 to $200, that would be a big step up from your Toshiba alone. There are many other fine DACs as you move upstream, such as the Bel Canto DAC-1 ($500 to $600 used).