I started with piano in grade school. Parents made me take accordion lessons. Played French Horn in Jr High band. In High School band picked up trumpet, baritone, sax, and clarinet. Well once you can read music and develop the embouchure its down to fingerings.
But this was not "playing around" this was playing music. Anyone can hit a key or get a sax or clarinet to screech and squawk.
Musical instruments are by nature quite loud. As much skill as it takes to play one well it takes even more to play one well and at low volume. In truth, its pretty much impossible to play one at really low volume. Yet it takes real skill to play one at a level anyone really wants to hear.
So the reality of musical instruments are they take skill and practice, practice, practice, without which they are pretty much noise. Really loud, obnoxiously loud, noise. Practice however takes time, patience, commitment, and perseverance.
Now you understand why more people don't do it.