What is the most challenging music to play on a stereo?

It depends on what you are testing. There is no one genre that does it all. In terms of low bass performance nothing beats the right organ piece. In terms of imaging nothing beats a great string quartet recording. In terms of dynamic performance, percussion pieces and NIN. If you want to get a divorce, Slipknot. 

Solo Piano

String Quartet

Female Voice.

Many, if not most systems have problems with sibilance. Female voice and violins commonly bring this out. Only the great systems can cruise through these without biting. Great does not mean expensive. I just heard a system built around Harbeth P3s that was wonderful. Piano due to it's complexity and sustain.