I actually wasn't thinking that your amp was clipping . . . I was speculating (as have a few others) that it's the subwoofer drivers themselves that are overloading.
If the drivers are exposed on the bottom, and there are four ports, then I was wrong . . . this IS a reflex enclosure . . . and it's alarmingly big. That is to say, by conventional practices, the guy who designed it maybe didn't know what he was doing? It's probably a minimum of 19 cubic feet or so internal volume, even subtracting for internal bracing, driver magnets and baskets, etc.
For comparison, the JBL 2235 was a 15" of the era with a low Q and low Fs (i.e. might be similar to your drivers in concept) and a pair of them work well in a reflex cabinet of less than 8 cubic feet. If I was to install a pair of 2235s in a cabinet similar to yours, I would expect it to behave much as you describe, with uncontrolled cone excursion at certain frequencies.
So I would do your research on this thing's background and design to try to determine if it indeed CAN work correctly. If it's a one-off piece, then you need to figure out EXACTLY which drivers you have, and see what their common configurations were. And even if you do determine that it's intelligently engineered . . . the drivers themselves could be shot. I'm assuming you would recognize rotted foam surrounds, but another very common issue in older, downward-firing systems is spiders that have lost their springiness and have started to sag (like so many other things do with age, when they are called upon to defy gravity).
No amount of stuffing, or electronic compensation, can make it right if you have worn-out drivers in an ill-conceived enclosure.
