It's all "important", but I would focus on matching an amp with your speakers.
Most Important?
After speakers where do you focus the lion share of your funds. I known garbage in garbage out. Would you buy a really great amp and an ok cd player of vice versa?