You should absolutely do it!! Age should have no impact on your decision at all.
I play the drums, and I'm 65. I do it almost everyday, and it always gives me great joy, though there are indeed times of frustration when I'm trying to play something and can't quite get it.
As for the question about skip the practice pad, go straight to the kit, why does it have to be either/or? My opinion is using both would help you get better sooner, and by getting better I mean enable you to play things you hear and function around the drum kit, and that would lead to more pleasure for you.
Playing along with records, cd's whatever is a good way to grow your playing, (and a lot of fun) but you do need to realize that you're not gonna just sit down at the kit and magically play anything you hear. You're gonna have to put in some time on developing your ability (building your "chops") so you are physically able to play the things you want. Doesn't mean you can't also slam away, it does feel awfully good.
Go for it, there's a very good chance you'll see the opportunity to fall in love with drums, (as I did) and if you get that feeling, finding a teacher and taking lessons would be something you might want to do. It wouldn't be an onerous task, it would be a joyous pursuit.
Btw, you can teach yourself, and these days there are lots of ways to get the info you need online. I was self-taught before the computer age. I spent 5 or 6 years working at it and actually ended up playing in bands that were fairly successful, and had a great time doing it. I got to the point where I couldn't grow enough on my own, so then I found a teacher (several actually) and got very serious about it. I'm not trying to make this about me, just trying to show you there are a number of ways to do it.
Good luck, I hope you take the plunge