Double blind test- over a month- could this be a reliable test for any equipment?
Summary of AES:
Ten years of A/B/X Testing
https://www.audiosciencereview.com/forum/index.php?threads/aes-paper-digest-sensitivity-and-reliabil... |
Seems like a parlour trick to me designed to confuse people and mask real world and even audibly measurable differences, let’s see how we can get NULL results on everything possible+1 Audio is not life and death, so there is nothing seriously wrong with following subjective impressions, but only as long as you remember that - that is all they are - subjective impressions. As are the impressions of those under test. Their hearing is still completely subjective whilst the testing is being touted as objective. There's a lot of conflating going on there. The parlor trick of A/B testing can reduce any desired result to that of chance. You're not listening for pleasure. You're listening under test conditions. No matter how "objective" you think you can be, that very aspect is constantly weighing in on your conclusions. The very nature of a test implies the ability to discern so it's inherent that you expect change. Now, factor in the "cheat" of implying a change, only to not do so. That is the reason the mind may perceive a change in a quick test scenario. It's just a trick. Any legitimate evaluation is done over the long term with equipment and music you are familiar with. It is then that you can be comprehensive and confident in your evaluation when you change something. All the best, Nonoise |
Thier certainly is a difference between R@D and testing to come up with a product or test Airplanes compared to how a normal human would listen to music at home. In what scenario would a person take back an AMP/Cable that they heard a positive difference in normal listening conditions but got a Null result in a DBY/ABX test? Regardless of a the DBT or ABX test I’m going to keep whatever products end up sounding better in my system. I just don’t see the actual practical use for them for normal consumers cause no matter the test results I’m keeping what sounds better in my system |
cd318421 Yes, let’s make ourselves even more anachronistic by putting our heads down back into the sands of subjectivity. Down with blind listening tests! Not our reality. On the other hand we can be grateful that aeronautics, automobiles, broadcasting, computing, engineering, healthcare, and satellite navigation don’t follow suit! >>>>>Do you think Boeing used blind testing on the software for the 737s that crashed? On the other hand if you are looking for truth, then the scientific method is a better way. One that has been adopted by the civilised world for centuries. >>>>>Double blind testing is not inherently the scientific method any more than any other kind of testing. Nice try! Besides results of subjective listening is empirical data, also part of the scientific method. 🤗 |
Yes, let's make ourselves even more anachronistic by putting our heads down back into the sands of subjectivity. Down with blind listening tests! Not our reality. On the other hand we can be grateful that aeronautics, automobiles, broadcasting, computing, engineering, healthcare, and satellite navigation don't follow suit! Audio is not life and death, so there is nothing seriously wrong with following subjective impressions, but only as long as you remember that - that is all they are - subjective impressions. On the other hand if you are looking for truth, then the scientific method is a better way. One that has been adopted by the civilised world for centuries. |
For me DBT and ABX is just completely pointless from a normal end user perspective. Let me put it this way if you do any type of test that is different than your normal listening conditions and conflicts with your experience in normal listening conditions are you going to change what choice you make? If I test Cables/Amps/ Pre pro in my normal setup in normal listening conditions and like the difference it makes during normal listening and then do an ABX/DBT and get the usual null results I’m not going to change the equipment back to the ones I liked less due to a DBT/ABX. Why would I downgrade my normal listening experience due to ABX/DBT? If my experiences are consistent where this cable sounds better to me before and after I do an ABX?DBT why on earth would I go back to something that sounds worse in normal listening conditions? Seems like a parlour trick to me designed to confuse people and mask real world and even audibly measurable differences, let’s see how we can get NULL results on everything possible |
| Post removed |
It takes us a little longer to absorb our fullest impressions aurally. Visually we catch on to a lot right away...aurally not so quick, we are predominantly visual creatures...but, yeah, we still have inherently good auditory memory. DB test all you want, but there will never be a substitute for living with your gear for many months or even years. It’s certainly not the fastest result, but as tests of perceived audio quality go, it’s the most revealing. The A/B-ing can tell us what differences in sound there might be, but it may tell us much less about which differences we might be willing to live with for now vs those for the longest run. What sounds fine enough to us today might sound more tiresome 5 years from now. Those are the chickens that most often come back to roost. But all of that to me is just the nature of the beast. |
The reason musicians can play an entire song or album or opera without a musical score in front of them is because humans inherently have good auditory memory. I don’t recall seeing Keith Richards or Jimmy Page using a musical score during a concert. 😬 same for movies. Humans can recall entire scenes from their favorite movies with ease, even after many years. Video, music, dialog, sequence of events, what have you. |
The demand that double blind tests be repeated a number of times in a row, sometimes as many as ten times, greatly increases the probability the person tested will fail. This is why The Amazing Randi never had to pay out $1 million in his Million Dollar Challenge. Not only for psychics - Randi’s primary target, a target 🎯 that Johnny Carson subsidized in his will - but also audiophiles, e.g., Intelligent Chip and exotic high end cables. It’s because when differences are small or subtle, which they frequently are, the pressure of having to be correct ten times in a row is too great. But Randi is a nice guy and knows a good thing when he sees it - audiophile sacred cows. 🐄 🐄 🐄 |
| Post removed |
recall and building a point of reference Some of you guys may know of my cousin the late Doc Watson. Riding in a car with Doc was a trip. Of course you know Doc was blind, yet he could tell you exactly where you were on a road he traveled often. For example Doc could tell you where every pot hole was on any of these particular roads, or even tell you if this road had recently been worked on. He knew the curves, distances and could tell you approximately how fast you were going. Our sensory recall ability is developed through experience and revisiting that experience. It's something we do every day on auto pilot but it's also something we cultivate as we focus on any particular skill we wish to perfect. When someone says "they can't hear or it doesn't exist" we should consider that this statement is probably true for them. They either have no desire to hear or they are focused on other areas that to them fits their point of view. We've seen a lot of listeners grow out of this, but we've also seen a lot of folks not interested in developing that part of their brain's pattern and activities. I tend to take audiophiles at their word. If they can't hear it, no biggie. It only waste time trying to convince them. If they someday open up that part of their experience tool chest, cool. If not, just as cool, it doesn't change reality for those who do experience. another thing If others make some type of derogatory comment about what you know or have experience with, which is common on forums, consider the source. They more than likely just haven't taken the time to do or their brain is wire differently with other priorities. When someone says BS, prove it, nonsense or those types of statements, you can pretty much take it to the bank they haven't done for themselves or they really can't hear it. These same posters share these same comments on any similar topic, it's their mo. It's a hobby of doing and teaching ourselves how to do and if you notice many times we go through the same types of learning curves as we advance on our own listening journey. mg |
It is very common for those who are immature, inept and ineffective, especially in writing, to think they are funny, entertaining, or amusing while trolling, boring, and annoying. It is odd to think, conjecture, or contemplate how anyone unable, unwilling, and incapable of anything other than one, no two, no three adjectives strung between, around, and betwixt conjuctive clauses could have anything to say that anyone man, woman, or child, would ever want to know. |
millercarbon"So get your facts straight." That is an odd, nonsensical, irrational, statement from one who freely, openly, and casually admits, acknowledges, and promotes the use of mind altering substances known for they're ability to intoxicate and then goes to assert here that his reason, logic, and opinion have value, worth, or credibility. |
mzkmxcv@clearthink .Tell that to Toole. You are welcome, free, and permitted to tell Toole what ever you desire, wish, and believe I told you and that was in specific response, reply, and reaction to your absurd, dangerous, unsubstantiated "claim" that is potentially risky, hazardous, and dangerous. |
millercarbon"My system sounds a whole lot better right after spraying anti-static " This sounds like a dangerous, unwise, reckless practice and is the type of thing I would expect to be suggested, promoted, and recommended by those hear who also routinely, regularly, and frequently engage in the consumption of various types of mind-altering intoxicants. |
Our perfect auditory recall period is about 10sec. If there is a minute difference you didn’t pick up immediately, it’s impossible for you to notice a change the next day. If I changed the volume of your setup by 2dB, I doubt a single person on this forum could notice it if they didn’t listen to the change for a full day, whereas if immediatly listening, the change is easily audible. Fascinating how you know what everyone else is able to hear. Especially since, according to what you wrote, you aren't able to remember anything more than 10 seconds. My system sounds a whole lot better right after spraying anti-static and demagnetizing. Then within half an hour or so it needs to be repeated. This wasn't something anyone told me. This was something I learned by listening. And being able to remember more than 10 seconds. I had some friends over one time. This guy didn't believe any of this either. He brought his wife and I told him, have her bring her favorite CD. That's all. I played her music and then asked if there was one track on her CD she wouldn't mind hearing twice. No other clue. They listened and when the track was done I stepped out, walked to the garage, flipped all the breakers off, came back and played the track again. They had no idea what was going on. Immediately when the song was over the wife exclaimed, "What did you do??? It sounded so much better!!!" I didn't even ask, she just blurted it out. That was a lot more than 10 seconds. Another time the same guy, like I said he just refused to believe. So when no one is looking he moves the chair a bit to one side. Because I had told him how critical symmetry is to imaging, so he's gonna have this big laugh I guess. Only, I happened to sit there to listen and set the volume for the next person. Just sitting and listening I noticed right away something was wrong. Checked the balance, nope. So either the chair or one or both speakers got moved. In no time flat, and the guy was watching and couldn't believe it, I had the chair back where it belonged. Entirely from listening. And remembering. For a whole lot longer than 10 seconds. I could go on and on. Parties swapping power cords for a whole room full of people. Takes a lot more than 10 seconds to swap a power cord. Everyone hears the difference. Heck, forget audio. Ask any mom if she knows what her baby sounds like. They remember EXACTLY what sound they make. Lewis Hamilton, you can bet he remembers how his car sounds from one lap to the next. 10 seconds. Right. |
mzkmxcv"Our perfect auditory recall period is about 10sec" This can be reliably, consistently, and repeatedly shown to be misleading, innaccurate, and false of course if you are under the influence of drugs or alcohol as is so commonly, casually, and frequently promoted, advocated, and recommended on this sight then the claim, assertion, and statement could be true! |
@alexatpos Our perfect auditory recall period is about 10sec. If there is a minute difference you didn’t pick up immediately, it’s impossible for you to notice a change the next day. If I changed the volume of your setup by 2dB, I doubt a single person on this forum could notice it if they didn’t listen to the change for a full day, whereas if immediatly listening, the change is easily audible. You noticing differences is simply noticing things that already existed, but were simply not overtly audible. I could play you a song 5 times, and you would likely hear some change if I told you there was one. |
divertiti22 posts03-19-2019 10:26amNo testing is 100% reliable when it comes to sensory perception, but blind tests are a hell of a lot more reliable than sighted tests. The human mind, perception and memory are all extremely fallible and vulnerable to influence. This is the reason why eye witness testimonies have proven over and over again to be the least reliable type of evidence. >>>>>There are many fallibilities, including operator error, improper test procedure, misterpretation of test results, operator hearing issues, system mistakes. Besides one test means nothing. It’s the preponderance of the evidence that one should evaluate, not just one test. People oft have ulterior motives for saying something like, “I bet you can’t prove it/hear it in a double blind test!” Most of the double blind challenges are empty threats. |
No testing is 100% reliable when it comes to sensory perception, but blind tests are a hell of a lot more reliable than sighted tests. The human mind, perception and memory are all extremely fallible and vulnerable to influence. This is the reason why eye witness testimonies have proven over and over again to be the least reliable type of evidence. With that said, there is an intangible side to audio that isn't easily discerned or understood in a shorter time. I have experienced times where I heard a difference, thought it was an improvement, but then the change annoyed me over the long term, or I missed how the music flowed previously. I suggest you combine blind testing of differences with longer term living with the components to get a full picture. |
OP wouldn’t you rather get to know the performance of a recording that you have experienced for many years and be able to recognize things changing from viewing your stage? It’s so much more fun to know a soundstage intimately and tune it in different ways to explore different aspects of the same recording. First you have to know how to get a soundstage consistently, but once that is an easy task, then it’s all about hearing the recording from many different points of reference. Sometimes you may want to spread the stage way out and other times not so wide and deep, but more condensed. There’s so many ways to listen to the same recording with all of them being right. Once you get to that type of control, double blind tests and other tests like that become boring and meaningless. mg |
And the sound changes all by itself day to day, hour to hour, month to month.But if conducting the test over a month changes should average out. I would say if you cannot identify a difference in that scenario it is not worth the upgrade. I think ideally it you could create a computer controlled AB that would only change (or not, double blind) after making a choice. Review the results and have an additional data point to make your decision. You can always revert to feels if you like. |
This has been discussed ad nauseam. There are many perfectly good reasons why any test is unreliable, including double blind tests. In addition, there are many perfectly good reasons why testing over periods of time is also unreliable. For one thing test results oft depend on whose ox is being gored, but also the system used for the test, external factors, and a whole bunch of other things. And the sound changes all by itself day to day, hour to hour, month to month. So, the question naturally arises, when you check on the sound a week later or whatever, what are you actually hearing? Yeah, made the scene Week to week Day to day Hour to hour The gate is straight Deep and wide Break on through to the other side Break on through to the other side |
In my experience, sometimes its possible to hear ’big and obvious’ differences right away, but on the other hand, sometimes it took me much longer time to hear and actually understand how some piece of gear affects the overall sound of my system. Even more, if the ’differences’ were not so big in the beginning,longer listening helps me to determine if something sounds 'right' or not. So, the blind test could help, but (imho) I would not consider it as essential for choosing some piece of equipment |
@millercarbon Just no. Have you ever intensely focused for 10 minutes straight? Because that’s what you are doing with double-blind listening, you are trying your best to be hyper aware of any sonic differences, hence why you need to take breaks. There is nothing flawed with this method. That’s a cop-out for people who want it to be true that their $2000 speaker wire actually made a difference. The test needs to be double-blind, quick-switching (~10sec or less), and level matched. Now, being double-blind and quick-switching pretty much means the test needs to be automated. |
Well, no, that's the point. Its not reliable. In fact its been proven not to work. This after all was the basis for some of the early codecs that were claimed to be scientifically tested identical sounding before everyone who actually heard them said hey no its crap. At the very least, if you can't stand to be doing it for long that alone ought to be enough to know there's something wrong. |
The double-blind audio data (e.g. codec) tests conducted by researchers and sound experts will be rapid comparisons of the same short segment. Typically you will pick a few seconds out of a 30 second track, and rapidly switch back and forth in order to easily detect differences. When we compare gear, we often do the same. 10-30 seconds of the same segment of music, repeated back and forth as quickly as possible between the two pieces of gear. It's the most reliable way to identify real differences. In any event, you can only do this for so long before you need to take a break. Unless the differences are so large and obvious, it can take a lot of concentration to go through this and people get tired. |
I once read that a/b-ing by switching back and forth during a track was the quickest way to missread true differences (as opposed to whole track with A, then whole track with B). And this was my experience...but a few people on another forum told me I was flat wrong. I still do not believe the brain can detect the finer differences with rapid switching. Right. Yet that is exactly what many people do believe. Once long ago I let these poor souls so mislead me into this nonsense that I refused even to waste my time trying to compare interconnects. Surely, their BS went, in the time it takes to change them you would forget what you heard. So don't waste your time! Then this one dealer advertised they had this switch that would let you switch between two interconnects. They could even pretend to switch if you wanted, a true double-blind! 200 miles later when I got there the switch was broke. Well I'm here... waste of time... but might as well.... So after listening a few minutes to one I got up, changed the source input, changed the interconnect, changed the source input back and.... freaked out! Instantly, immediately, from still standing BEHIND the amp I heard the difference PLAIN AS DAY! That was back when I could hardly hear anything compared to now. So imagine how I feel hearing about how you have to switch fast, or double-blind, or any of that bull. Are you old enough to remember Mr T and his signature line? I pity da fool. Listen. Trust your ears. Pity those who don't. |
| Post removed |
Audio is a thousand little religions, all of them right, and all of them wrong. I once read that a/b-ing by switching back and forth during a track was the quickest way to missread true differences (as opposed to whole track with A, then whole track with B). And this was my experience...but a few people on another forum told me I was flat wrong. I still do not believe the brain can detect the finer differences with rapid switching. |
| Post removed |
