I did this. My Take
1) I wouldn't even consider going down this path without a proved speaker design from a reputable designer.
2) It is more about the integration of the speaker drivers and the voicing of the crossover than just picking drivers you like. The drivers may be the easiest part.
3) The box is a big deal and you do need to have good wood working skills.
4) The crossover soldering is also harder than you think, I had the advantage of having someone at work who does it for a living do it for me.
My story is I used Troels Gravesen's designs. I think most see him as the best. I made 3 different speakers, bookshelves (Scanspek), center(Seas), and the Faital 3C-15. Bookshelves and Center took 3 months each, the Faital more like 5 months. My woodworking joints were impeccable, they need to be. I also spent a lot of time on finishing so they would be acceptable in the home.
I never thought in my life I would have such a good sounding system, honestly. The bookshelf/center is meant for the family room entertainment center, I have them in the garage and I am in there every night for an hour looking for an excuse to stay out there, they now have about 1000 hours on them and they get better and better. The 3wc-15's are absolutely amazing, I just melt into the music and can sit there for 3-4 hours at a time.
If you are committed, have a great attention to detail and it is important to you, do it. If you really do a good job and use good veneers, you can make them look pretty nice, you can also send to a cabinet mater for finishing if WAF is an issue. I built a paint booth in the garage and was able to get a really nice finish.
If you do it, let us know how they come out.