Speaker cabinets: How important is inert

First let's assume that the best school of thought is for speaker cabinets to have zero characteristics of their own, i.e it's completely silent.

A lot of the premium speakers like Rockport, Magico, YG, Kharma, Wilson all boast custom cabinets which are supposed to be "dead," which will let the drivers do their jobs without having the cabinets interefere. There are also premium speakers that uses braced MDF like TAD, Tidal, the lower Rockport lines, Avalons, etc which are supposed to be almost as good.

I'm not in the market for speakers, but everytime I look and listen to different speakers, I almost always prefer the big heavy duty cabinet speakers, and not the slim shaped refined looking speakers.

So my question is - do these custom epoxy or sandwiched or aluminum or whatever cabinets make a HUGE difference over plain MDF or braced MDF, or is it just marginal? How much of the secret to a good sounding speaker is in the cabinet engineering versus the drivers?