Bad idea, though you could get lucky and have it work OK. Though it will no longer be the designer's intended performance.
A single driver might be 8 ohms (or 4 ohms, etc) but once you put it into a network with other drivers and crossovers, and the net impedance often becomes something else. Usually the result is quite complex versus frequency, which is why speakers are rated at "nominal" impedance to make a nice single number.
If you had 8 ohm driver parts in a speaker rated 4 ohms nominal, and your replace them with 4 ohms driver parts, you'll usually be lowering the nominal impedance well below 4 ohms, possibly into the danger zone for your amp.