I always wear gloves. Kevin Deal, IMHO, one of the top authorities on tubes and who I've known for over 40 years, says the oil from your hands won't hurt the tubes in any way...I still use gloves.
To quote Kevin:
"As for the glass itself, despite what some people might tell you, the oil on your fingers will NOT damage the glass or leech through it. Tubes are not halogen bulbs. Tubes may get hot to the touch, but your finger oil will not cause the tube to break when heated up, nor will it affect tube life or sonics. It will not go through the glass."