Beryllium is toxic, period. Skin contact usually only raises a rash, but forming the stuff is always left to specialty shops, which is why you can't just buy some to play with. Beryllium copper alloy is mostly copper, which is why it is safe to use.
I heard the $4000 Focal Utopia Headphones yesterday
Very nice. Very expensive. Beryllium drivers. Reviews a bit hyped I think. Who would pay that premium otherwise? Not sure I could live just with any headphones still.