Sharpie!!!!
Red, Blue, or Black the most common. You can control the darkness (brightness to the LED)through layers of coating, not to mention that you can combine colors to create what you want. What about removing them later you say? For a glass or plastic surface(LED,etc),use ISP(isoprophyl alcohol) on a soft cloth or cotton ball. For paper like material...you're on your own. It is definitely permanent on paper products!!!!
hope this helps