I had a similar issues (my sweet sounding crystal clear amp went all grainy and hard sounding) with a Audio Research CA50 and B&W 805s's. The bottom line is they just do not work well together. I ended up swicthing to a McIntosh 2275 and all was well. The CA50 is now happily driving a pair of AR S310HO's with Sequerra Ribbon Tweeters with no problems.
CAV-50 and 804 Diamonds
I am demoing B&W 804 Diamonds. I have a Conrad Johnson CAV-50 and pretty sure it's not a good match. I'm having to constantly bias the tubes and unless I get it right it produces a fairly grainy sound at volume. The 804's need WPC of 50-200, while the CAV-50 produces 50 WPC I'm replacing Coincident floor standers which is a fairly efficient speaker. I really want to stick with the 804s. I guess the point is I'd like comments on match with the CAV-50 and/or what would you replace it with. Thanks