Clean your stylus well to remove the layer of burnished-on vinyl particles and crud. It's there - guaranteed.
Use something stronger/more abrasive than a dry brush, goop or gel. Search for "magic eraser" if you need a cheap and effective idea or use a sliver of very fine sandpaper, like Linn has always recommended.
Don't worry. All you did was use 4 of the thousands of hours worth of objective playing time on your cartridge. Subjectively, you didn't even notice! ;-)