Go for it. I started out slow by building a subwoofer first. Despite the fact that a sub is a simple box/amp/driver, there is a whole lot more to it than slapping those components together. I learned a ton. I met many great contacts as well. By the end of the design phase I was using subwoofer computer models for calculating the box dimensions to achieve a particular "Q" rating that I wanted to achieve, pretty cool stuff.
Later, I wanted to try out a single driver speaker (the ones in my System now). After reading a TON of info on the WEB, I decided to build a pair and settled on a proven design complete with detailed plans. I learned a great deal about resonance, porting, soldering as well as the many different types of resistors, coils and capacitors that are available. Here too, I met many new folks some of who have become good friends. Even if the outcome would have not been to my liking, I would have come out ahead with the knowledge I have gained and the friendships I have made.
Several weeks ago I was plotting the frequency response of my system on a test tone CD and a SPL meter. If you would have told me I would be doing that a year ago, I would have thought you were nuts!
Considerations:
1. Try out a relatively simple design first.
2. Try out a proven design, don't go to the self design route first, you will be amazed the things you have taken for grated.
3. Undertsand your wood working limitations (tools and ability) don't exceed these.
4. Have patience, don't be dissapointed at your mistakes, learn from them.
5. Take the time to meet and ask people on this site and others for help. You will be amazed at what you don't know.
Granted, my single drivers are much simpler a design than multi drivers are, but start something and enjoy the ride.