Sounds like you have qualitty socket savers. the idea behind socket savers is that they should fit snugly into the amp socket and stay there so that you can put lots of tubes into them--perhaps vintage tubes with dirty contacts, without degrading the sockets in your amp.
Some socket savers stay with the tube rather than the amp and don't give you full benefit.
So generally you don't remove the socket saver. but if for some reason you need to remove it, pull harder.
Jerry