Amir and Blind Testing

deludedaudiophile, you seem very defensive about this topic. I get the impression that you manufacture a product and are trying to rationalize not using certified or calibrated test equipment. That would be sloppy work indeed. Any manufacturer publishing specifications for their product should be backing up those specs with data from certified, calibrated instrumentation. If not, they could find themselves in hot water I would think if not able to defend their published claims.

Deludedaudiophile, good luck with your battery venture.  You might want to learn more about metrology.  You can take some courses or learn more from the ISO or MIL Q Specs about test equipment certification and traceability.  These are industry standards and plenty of calibration labs around the country to get your test equipment calibrated and certified for a reasonable cost.  Turnaround times are pretty quick if no repairs are needed.  

So in my vanity, I will take several readings on my bathroom scale but accept only the lowest reading.  I don’t do a statistical calculation of the group of readings.  That’s the very definition of biased testing, I think.  And what’s it matter?   When I go to the Doctor’s office they will not accept my weight based on my scale’s readout.  They take their own measurement on their scale.  No one believes the test data except for the one who took it…

I watched one of his videos a year ago. He tested some expensive speaker cables and dismissed them as comparing no different to lamp cord. His followers seem to eat these types of videos up. I challenged his conclusions with first, was his analyzer calibrated and traceable back to the Bureau of Standards. His answer was no, and that calibration is not necessary since it was a new piece of equipment.. Anyone who publishes data with uncalibrated equipment is immediately suspect. Secondly, anyone who publishes data without the information needed to duplicate the testing makes the data meaningless.

That was as far as I got before getting endless abuse from his followers so I moved on. He tends to test everything with his new (last year) analyzer. To someone with a hammer everything looks like a nail.

Wow, another self proclaimed expert.  Spectrum and frequency analyzers do need to be calibrated.  Since I was an Engineering Manager for several years with one of my responsibilities being test equipment design, calibration and repair in a major Aerospace Company I know that first hand.  All test equipment should be traceable back to the Bureau of Standards or NIST for serious test work or analysis.  Nitpick over the name if you like.  And guess what- equipment loses its calibration.  Protocols were required to deal with product previously tested on equipment found to be out of calibration by more than a certain amount.

I use my iPhone or iPad apps to measure things but I treat is as comparative data, not absolute and I always qualify that as reference only.

And speaking of correlation, that is a big bugaboo.  Correlating test equipment is the biggest pain ever.  The question is; how can two piece of test equipment, both being calibrated give two different measurements for the same part?  The answer is yes.  That's it.  Yes.  So the old saying goes; "a man with two watches is never sure what time it is."  A lot of time and effort spent over the years getting test equipment in two or more locations to correlate.

Then we talk about subjective evaluations and the results will be all over the map.  Subjective evaluation is often used where measurements cannot give a complete picture.  We do it all the time in the audio world and I'm sure that most if not all audio equipment manufacturers do subjective testing as well.  A jury must be trained to listen for specific characteristics and typically receive very specific guidelines detailing how to rate a component or system.  In my experience, the environment is closely controlled and the subjective evaluations are repeated by the panel several times over a few days.  Our moods and previous environmental exposure can affect the results.  So we Audiophiles are typically self taught in the art of listening.  These forums and articles can provide instruction as well.  I'm not saying I'm a great expert at subjective listening.  I'm in my mid 60s now and recently I was at the Tampa Audio Expo with my son who is 31.  He could hear things in the various rooms that I just could not pick up.  And based on that I am telling myself now that I really need not spend so much on upgrades these days.  I have hit my limit of hearing.  But I have no doubt that others can hear differences that make sense to upgrade.  I have a 45 year history of that.

Learning how to listen to our stereo systems is very important.  So back in the day I could walk into a real live brick and mortar store and hear a state of the art stereo.  Then I would go home and listen to my system and I would think, "wow, my system sucks."  Now I would go back to the store and listen again to see why the stereo in the store sounds better.  Is it the amp, the speakers, the preamp, the source or the room?  What about the sound is better at his store vs my no good stereo at home? This could also apply to a friend or group of hobbyists where you have the opportunity to listen to their systems.

An equipment manufacturer might want to bring in some mics and measurement equipment to understand why the sound is better but most of us learned to depend on our ears and our notes.  In fact, without the opportunity to hear other stereo systems - be it at a friends house or a store, our systems can become quite off and we not even realize it.  At least until we get in our car one day and think, "Man! This car stereo sounds better than my home stereo."  Been there too.

For example, about two years ago now I got to hear the Alexia 2's in Atlanta for the first time.  I went home and thought, wow the bass in my stereo system sucks.  I went back and listened some more and started hearing the differences between the  Alexia's and my speakers.  I was just about ready to drop some serious $$$ for new speakers but decided to first work on my room, speaker placement and I also added subwoofers which took another several months to position and tune.  I'm at the point where my system has bass almost as good as the Alexia's but good enough at least that I'm no longer motivated to upgrade.  It took me several months of work, including putting in a wood floor but I can say from experience that speakers are not the number one influence on the sound.  It is the room and then the electronics.  Of course, the speakers have to be reasonably qualified.

My point is some actual measurements are needed at times but critical listening skills can guide us to build exceptional systems.  

Oh, and the other thing- it took months to find the best position for the speakers and subs because what sounded good one day sounded terrible another day.  It can take time and patience to get dialed in.  I also spent some time listening to the Wilsons over a year plus and I did buy some other gear at that store since I spent so much time there.  I still think they are great speakers.

deluded audiophile, your behavior- argumentative, disrespectful of others, spending 5 seconds reading about something on the internet and thinking you are an expert, indicates that you are a young adult living in your mom's basement somewhere who has accomplished nothing.  Step away from the keyboard and go outside and play.  The sunshine will do you good.

Most of you on this forum likely do not know or have ever heard of a gauge R&R.  Most also likely do not understand the concept of accuracy and precision.  That's not a slight.  This is a difficult concept and much work has been done to define it and apply it to test measuring equipment.  I want to start with something most of us know quite well- the bathroom scale.  If you are like me, we have a love/hate relationship with our bathroom scale.  It's a simple device that can either make or break our day and yet we typically do not think twice about whether or not it is telling us the truth.  What do I mean?  Well, for starters I can get on my bathroom scale three times consecutively and get three different readings with a range of 2 or more pounds.  Even worse, I find that I can move the scale around on the floor and get even more variation.  This is one of the newer scales with a digital readout to tenths of a pound.  While my bathroom scale indicates a precision of 0.1 lbs, the repeatability is much worse which implies the accuracy is likely off by a few pounds.  I don't know because my bathroom scale has no reference back to a standard.  I notice the scale at the Doctor's office has much better repeatability.  I see just 0- 0.1 lbs variation if I step off and back on again and the Doctor's scale has higher precision based only upon the display showing hundredths of a pound.  But I have rarely seen a calibration sticker on the scale in the Doctor's office.  I have seen stickers on the scales at a research dept and at the hospital.  Probably because they publish reports.  Accuracy is typically not well defined.  Typically, gages are rated accurate to within a certain percentage of full scale.  Let's say a bathroom scale is rated to +/-0.5% of full scale.  (Not likely that good for a $30 scale)  That means the manufacturer is stating that any reading will be (for a 400 lb scale) within +/-0.5% of 400 lbs or +/- 2 lbs.  So I could have lost one pound overnight but my bathroom scale might tell me that I gained one pound!  Isn't that frustrating.

What's my point?  Let's say you go to the butcher shop and you buy a 10 lb ham.  Then you stop by another shop and just to see, you weigh the ham on their scales and find it only weighs 9 lbs.  Wouldn't you be upset?   How about you stop at the gas station and buy 10 gallons of gasoline only to learn you actually got just 9 gallons.  Well, take comfort in knowing that by law those scales and gas pumps are calibrated back to a standard.  If you look at the scale at your butcher shop you should see a calibration sticker.  The same goes for your local gas pump.  Take a look on the face plate of the pump for the calibration sticker.  

If we count on these everyday items to telll us the truth then why not expect the same regarding measurements of stereo gear.  Knowing that calibration of the equipment to a standard was done, what test equipment was used, and also the procedure so that the measurements can be duplicated or verified by someone else is crucial to know that the data is telling us the truth.  Also important is to know how these particular measurement data relate to how the piece of gear performs.  For example, I can measure the resistance of two different speaker cables with an Ohmmeter or even a resistance bridge for more precision but still conclude no difference.  So why do they sound different?  Some speculate that better cables reject RF noise.  Sounds reasonable to me.  So why hasn't someone published test data showing the RF rejection characteristics of different cables?  Maybe they have but I just have not seen it.  This would not be easy testing.  It would require a Faraday cage and some sophisticated measurement equipment.  Still, we cannot and should not take every measurement at face value and make conclusions from that about what we are or are not hearing.  I had my own saying in Engineering:  "No-one believes the test data except for the person who took it.  Everyone believes the calculations except for the person who made them.