Affordable vs. ultra expensive speakers - what's the difference?

It’s difficult to judge a pair of speakers based on the frequency response plot.  There are just too many parameters.

It’s difficult to judge a pair of speakers based on the frequency response plot

True, that's why I look at more than the FR plot. If I can't find extensive measurements on a speaker I don't even look at it. 

Nobody has mentioned the true difference - your ears. 
In a true double blind test scenario, nobody here could tell you what speaker they were listening to - no matter the price. 
 

buy what sounds best to you- measurements do not mean a thing if you can’t hear the differences - then simply enjoy. 
 

The difference = a lot of money for about 2-5% improvement in sound.....

The curve of diminishing returns.

Will it sound $48,000 better?  No

Will it sound better than a $2,000 speaker? Yes

For the wealthy people, $50k means nothing. For the average joe, there are many other ways to spend that money for good usage. 

Sound systems need to have synergy with their components. A great sounding system could cost less than 3k in the used market.