For your CD, DTS, and Dolby Digital you will probably get the best results using a digital cable which will cause the DACs in the 5800 to be used instead of the 3800. I have a 5800 and its DACs are better sounding than most DVD players. For DVD Audio I think Cardas Neutral Reference would work well to connect the analog multichannel outputs of the 3800 to the 5800 External inputs. I have used the Neutral Reference with the 5800 before, and it is a pretty good match.
It is OK to hook both digital and analog cables up at the same time. Just set up the receiver input for DVD to a digital input and select the external input when you want to listen to DVD-Audio.
It is OK to hook both digital and analog cables up at the same time. Just set up the receiver input for DVD to a digital input and select the external input when you want to listen to DVD-Audio.