How to assess and quantify system improvement and changes?


In one of the currently popular threads, a number of members (all of whom I respect) have made significant claims regarding system improvement.

For example: A recent post mentions "a double or triple improvement in my system." A conversation a few days ago with two audio buddies had them quantifying a DAC change as a 10% improvement by one and 30% by another.

This, by virtue of our pursuit, occurs throughout our discussions in the various threads.

Starting this thread to see if we can mine the collective and come up with guidelines and outcomes that are reproducible, relevant and reliable for comparisons and discussions between audiophiles.

Thanks!
david_ten
Post removed 
My problem with trying to quantify this sort of thing is that in why experience the process is very non linear

By this I mean it goes something like ...
  1. "My systems sounds great, can't imagine it getting much better (diminishing returns etc etc)"
  2. Introduce component X or tweak Y
  3. "wow, I never paid attention to/I was not aware of that (coloration, timing error, gross soundstage aberration etc) before - it's amazing how much better the system sounds without it"
  4. continue tweaking to improve this dimension
  5. Return to step 1 and repeat
So each step 3 can feel like a very large change, then followed by a series of smaller incremental changes and the large change is driven less by the specific quality of the change/new component that highlighted it but more by the fact that it became highlighted

Said another way we train ourselves to appreciate (and to some extent compensate for) the shortcomings in our systems, we "listen around" them and it's only when the need to do this is removed that we realize how much mental effort we were putting into doing this audio error correction ... ultimately then the best test of any change is does it make it easier to listen into the music and get to the intent of the artist
I make changes in equipment and equipment positioning and the result comes down to...........better AAAH, or worse DOH! Percentages are for bank loans.
Art is not science let alone exact science. Or, if you wish, it might be a science, that is beyond our comprehension. So yes, human condition, condition we can live with, I think.
Post removed 

There's a fundamental problem here, as there's no benchmark that any of the judgments can be keyed to.  These are subjective judgments being made by individuals about unique combinations of components in unique listening spaces.  Try codifying that!

From a different perspective.  Replace a small, inexpensive 2-way with a large, high-quality floorstander.  More bass, right?, and better sound generally.  How much more bass?  How much does the increased bass enhance the overall listening experience?  (These are two different judgments.)

Then replace some throw-away 39c ICs with really expensive ICs.  Is there are much of a change?  Where?  In what areas do you hear it, and how do you quantify it?  Compare experiment 1 and 2.  Does the change in 2 equal that in 1?

Most improvements are incremental. People exaggerate, often without knowing they're doing it, to make a point.

A few years ago I dropped a pair of $200 speakers into a system that I'd previously been running with 5K speakers.  What struck me most was how decent, relatively speaking, they sounded.  No, not as good as the 5K, but as everyone always points out, not nearly 25x as good.

What I will say is that reviewers' hyperbole set people up for hearing more significant improvements than they actually achieve when they bring the component home.  Then the search for mitigating circumstances begins...

Post removed 

good luck with this one.


really! I wish you do figure it out, but the variables in this quest are innumerable.


the predominate issue is Audiophillia Nervousa is a self diagnosed disease. 


as such, EVERYTHING then becomes purely the beholder’s viewpoint or assessment on every account.


in fact, the. variable factors extend farther than the equipment itself. one can have the exact same gear in one setting, remove it from that venue, transport it to some other home, or setting, in some other likely dissimilar room, and the results will probably be perceived differently. further, merely purchasing either a single piece, or even a ’turn key’ setup you have auditioned at a dealership, and put it into your own arrangement , and chances are it will sound different.


another separating issue is simply the fact few if any outfits will have identical pieces in tow. how then, can a proper account of how ABC aids or detracts from, any particular rig? or for that matter how good is the ABC gizmo on its own merits  ever to be reviewed?


if the aforementioned items are not enough to frustrate this affair, add in our ears are not hearing things exactly the same way or as well, as other ears!


OK, then bring in the meters and micrometers. of course! but which meters? which rulers? which scales? how will this or that actually get measured?


then too, which words will. be used in surveying or appraising the thing? humans are barely able to communicate these days and suddenly we are all supposed to be crystal clear oh the piece, context, syntax, and findings?

 

synergy. is yet another factor which will ever detract from acquiring a status quo for quantifying performance in terms of gains and losses. its a proven fact, not everything ever made plays well with every other thing ever made.


I’m not even mentioning circumstances surrounding electrical power anomalies which vary  from one installation to another.  . that is another bag of worms all by itself.


in years past, telephones were in short supply. regularly, to talk to a friend or relative, you very often  had to  travel to their home. In so doing one experienced at least in one fashion, the ‘charm of distance’. 


back then distance was often a desired boon for its yield was greater privacy. 


privacy used to be highly valued, now everyone wants everyone else to know what they ate for breakfast, or where their cat  coughed up a fur ball. and when.  


in the often tail chasing hobby of higher end audio, the ‘charm’ lies in the preferences and perceptions of the one paying the bills to be the boss.  his or her insights on individual items, or as an ensemble are the only views that matter.


the only rule of thumb I’ve discovered which generally points to  performance, though not empirically, is simply the ‘cost of likewise  goods’.


gear makers on average have been quite studious on pricing their products, mostly.  subtracting those whose view points lend to them over valuing their equipment, tier pricing seems to equate to tier performance. IMHO


paying to get to some of these tiers of upper end  performance can  bring on many tears. gnashing of teeth and wringing of hands are other participants in the aftermath.  especially if system synergy is not elevated from the alteration or addition of the newly arriving  new bright and shiny gizmo


exactly where these supposed ‘tiers’ separations exist is yet one more facet that lends itself to the charm’ or chagrin, of this hobby.


bear in mind, the notation on the ‘self diagnosed’ theme. 


charming, isn’t it?


I think this is almost impossible to achieve, because it is always based on subjective assessments.  We don't all weight various attributes the same, so, if I were to make a change that to me seemed to be a 24.8% improvement, someone else would say, no, I clearly hear a 32.3% improvement.  For me I am satisfied to place improvements in the following categories.  1)  Immediately obvious, even with casual listening. 2) A clear improvement in one or more attributes based on several minutes of critical listening.  3)  Relatively minor improvement revealed by careful listening to music that I have used for years to assess system/component performance.  4)  I just can't hear any difference, no matter how I try.  5).  Deoptimization in one or more attributes. 
I would love it if someone could make any headway in this endeavor, but I know of no way to quantify subjective measures of performance. This "equation", if you will, would probably have to be broken down into many parts with exact definitions of all the terms involved. I wouldn't even know how to begin.
I’ve been guilty of hyperbole also, but I realize I was doing such a disservice to so many. Now I try very hard to put things in perspective and frequently say, "there are no absolutes in audio, only preferences. In addition, I submit at a certain level night and day differences don't exist.
@inna  funny and thought provoking.  And I get the point. : )

@ebm  I'm honoured. I figured a "Let me think really hard about that and get back to you tomorrow!" : )

Seriously though, take a stab at it if you feel it's worthwhile.
President Trump is 10% more beautiful than Chinese president but 1000% less so than French president Macron.
Extrapolate this and you'll get scientific method par excellance.
@twoleftears  Agreed on the needing luck! : )

Your point on 'hyperbole' is well taken and applies. However, as @cleeds  correctly points out, it's present on both sides... whether the descriptors are positive or in favor and / or those negative and against.

I have been guilty of this and I've been 'reining' it in, so to speak, as best as I can for myself. Disciplined control....

@slaw  Of course. Each of us will describe the things we try in our own ways and to varying degrees (relative to others) and that's one of the reasons for this thread.

I'm proposing we work on defining the 'metrics' we tend to throw out there when we describe our system or other systems.

Perhaps something of this nature has been worked out and is available? If so, can a member in the know share the links, etc.

How we share what we experience is critical to our hobby. I'm asking for a methodology that can be (more) accurately understood and be (more) consistent across audiophiles. 

twoleftears

Audiophiles almost by definition deal in hyperbole; to become a reviewer, first you need to pass a test in hyper-hyperbole.

That sounds like hyperbole to me.

Notice how systems are so often "transformed" by substituting, e.g., wooden speaker cradles for ceramic ones ...

Do you have any actual examples of that, or is it just your hyperbole?


Good luck with this one.  Audiophiles almost by definition deal in hyperbole; to become a reviewer, first you need to pass a test in hyper-hyperbole.

Notice how systems are so often "transformed" by substituting, e.g., wooden speaker cradles for ceramic ones...