Is it worth buying a headphone amp?

Thanks for the responses, folks. It seems like a headphone amp will benefit me greatly, but I'm still wondering if I should buy better headphones first. I did a little experiment this morning and plugged my Senns into my Macbook (I'm not a computer audio guy yet) with its output impedance around 30 Ohms, and despite the crappy sound card the sound was worlds better than it was through my NAD. Still not a great impedance match, but it was better.

Though the bass was diminished when plugged into the laptop (I think the impedance mismatch with the NAD emphasizes the mid-bass because of the Senns' high impedance spike around 100Hz), the mids and treble were clearer. Cymbals sounded like metal rather than papery electronic hash, though resolution still wasn't excellent. A brushed snare on Lucinda Williams' "Fruits of My Labor" sounded like vague background clatter (think paperclips being shaken in an empty OJ-concentrate can), while it resolves perfectly through my speakers. Not sure if that's a limitation of the headphones, though I do see that they're capable of better sound than I was getting.

The Portal Panache is new to me and looks like a great product. That might be a good future upgrade for me. For now, it's a little out of my reach. I've read a bit on Head-Fi but haven't joined yet, so I appreciate everyone's input here. Still don't know if I should put the Schiit or headphones on the list first (ideally I'll eventually do both)--but I'm open to further suggestions! Admittedly I'm still not a huge headphone fan, as listening fatigue sets in pretty fast for me even at low volumes, but they're good for keeping peace with the neighbors.

Well, folks, new headphones it is. The HD555 are excellent for the money, but to start with I think I'm to going sell them and look for a used pair of AKG K702. I'll buy an amp later, when my budget allows. The Schiit Asgard is still at the top of my list, though I am concerned by the 6 Moons review, which characterizes it as being a bit lean--and lean is the opposite of what I want in a headphone amp. Any other suggestions around $250? Thanks for the input, all.

I appreciate the cautions about the AKGs. I am a little worried that I might find them lean or bright, and due to geography I'm not going to be able to audition them before buying. That said, I thought the AKG K501 was a pretty amazing headphone when I owned it. Its treble was more present than Sennheiser, but I wouldn't have called it bright the way I would for something like the Etymotic ER4P (I got rid of those after a month or so). The K501 did need a LOT of break-in, which I understand is true of the K701/2, too.

I know I really liked the HD600 when I owned them--deeper bass than the K501 with a warmer and smoother sound overall. I might just buy another HD600, except for two things. One, used prices for the HD600 have seemed to rise a bit lately, making the K702 the better value. And two, I just plain think the K702 looks cool! Probably the wrong reason to buy headphones unless I listen in front of a mirror, but still...

Update: I sold the HD555 and bought a mint pair of HD580, which hadn't even been broken in and already sound fantastic. The upgrade is significant--the HD580 sounds very much like the HD600 I used to own and is a much more refined headphone than the HD555. Bass goes deeper, midrange is much more natural, and highs are grain-free.

As far as a headphone amp goes, I may buy one later on but have found a great solution for now. One of the problems with the headphone jack on the NAD C372 integrated is that the gain is so high that 'phones are either too quiet and unbalanced or way too loud. Luckily, the amp has a second set of pre outs with variable gain. I've run a set of IC from that pre out and attenuated its gain by about 6dB. Now I can turn up the volume knob into a range where it works better, and the headphone sound is great.

My speakers sound better this way, too. Before, I could never go past 9 o'clock with the volume, and now I can turn it up toward 10-11. The headphone sound is a bit cool and dry, but the whole amp sounded that way when it was new, and the headphone stage hasn't been used enough to be fully broken in. I'm going to break it and the 'phones in over the next couple of days.