Hey OP.
Your count of amplifier channels is correct. You would need 1 channel per driver.
In an ideal situation, the speaker crossover is entirely removed. However this is complicated for a number of reasons. Crossovers are more than just frequency filters. They also provide level matching and equalization. DSP crossovers are supremely convenient for this as they can do all that plus delays.
I suggest before you experiment on anything fancy consider a simple test rig to learn more about this. Get a pair of inexpensive 2-ways or kits, a miniDSP and a pair of cheap amps.
Best,
Erik