It's almost impossible to AB anything without bias, for a variety of reasons...
1. When presented with 2 options, the second is almost always preferred because we naturally tend to evaluate the second relative to the first (but not the other way around). Since the second is usually the 'new' one, we are biased to the new.
2. Warm up time. As most know it takes 30 minutes or so for something to warm up. The interim delay is too long to retain realistic recall.
3. Visuals really matter. We are strongly biased to something that 'looks better' whether we realize it or not.
4. We are biased to the new and/or more expensive, again whether we realize it or not.
5. It's impossible to get a good sense of things in a short time. Long listening sessions are needed. Again, by the time you've listened to the second for a while, you've completely forgotten what the first sounded like.
Not sure it's possible to truly compare in AB testing. Of course huge differences in sound are readily apparent, but subtle ones much less so. Lesson - don't make changes unless they are huge ones. Small 'tweaks' are purely emotional.
1. When presented with 2 options, the second is almost always preferred because we naturally tend to evaluate the second relative to the first (but not the other way around). Since the second is usually the 'new' one, we are biased to the new.
2. Warm up time. As most know it takes 30 minutes or so for something to warm up. The interim delay is too long to retain realistic recall.
3. Visuals really matter. We are strongly biased to something that 'looks better' whether we realize it or not.
4. We are biased to the new and/or more expensive, again whether we realize it or not.
5. It's impossible to get a good sense of things in a short time. Long listening sessions are needed. Again, by the time you've listened to the second for a while, you've completely forgotten what the first sounded like.
Not sure it's possible to truly compare in AB testing. Of course huge differences in sound are readily apparent, but subtle ones much less so. Lesson - don't make changes unless they are huge ones. Small 'tweaks' are purely emotional.