Newbee, a reduced damping factor between the amp and speaker can definitely influence the control of the bass driver(s).
Exactly how the loss of control manifests itself may not be exactly predictable, because sometimes the speakers(especially ones with lower mass and strong motors), may have very good inherent damping within the speaker design itself, or it may have poor inherent damping within the speaker itself. The better the inherent damping characteristics of the speaker itself, the less dependent it is on the amplifier to control it. In some cases, a lower damping factor can actually improve transient response.
In some extreme cases, a very low damping factor can cause the frequency response of the amplifer to rise and fall with the variations in the speaker impedance curve. We try to avoid this by proper matching of the impedances, with a goal of at least 5 to 1(speaker to amp impedance). This keeps the frequency response curve relatively flat within a db.
With 4 ohm speakers and a tube amp, it is possible that the impedance curve of the speaker in question is stressing the amplifier at certain frequency points, even at moderate volume levels. If the speaker's impedance dips to a point where the amp is uncomfortable(or approaching instability), then it could definitely have audible effects at these frequencies. Since, with multi-driver speakers and the crossovers involved, there could be impedance swings at anywhere along the spectrum, it can be determined by getting the impedance curve charts, and looking to see if there is a problem area.
Many tube amps do not have very low output impedances(especially if zero or little feedback is used), and reactive speakers with 4 ohm nominal loads could easily bring the amp to some problematic conditions.
Generally, if the tube amp has an output impedance of less than 1.6 ohms, and the speaker has impedances of 8 ohms, you will have the needed 5:1 ratio, as long as the speaker doesn't have any really reactive dips that could compromise this ratio too badly. If the amp has an output impedance of under 1 ohm, then it should be able to handle relatively benign 4 ohm loads. If the 4 ohm load drops to 2 ohms or 1 ohm, then the amp is going to have trouble.
Additionally, since tube amps typically produce lower power as the impedance drops, then it could go into clipping much sooner than when it is driving higher impedances.
As you can see, there could be a variety of reasons for what is happening in your case, and it is the matchup, and not necessarily the quality of the products that may be making the difference.