How does the digital bypass on home theater processor work

I think you might be thinking of this backwards.  The idea is to run the front L/R preouts from the prepro to an analog input on the Mytek.  That way only the Brooklyn is in the playback chain for critical 2-channel listening and the prepro is completely out of the circuit.  I don't think your DAC has a HT bypass function, so you can just choose a reference volume level on the Mytek and use the prepro to balance the channels. 

Doing it the other way, the Mytek's signal would have to go through the prepro before going to the amp, which kind of defeats the purpose.  Hope this helps. 

Given what you're looking to do, and that unfortunately the Mytek looks like it only has one analog input that presumably will be taken by your turntable, you'd just run the analog out from the Mytek into the front L/R analog inputs on the prepro.  It's unfortunate because the output from the Brooklyn will have to pass through the prepro rather than passing directly to the amp. 

Maybe someone else has a thought, and just spitballing here, but if there's a high quality RCA 2 to 1 switcher that could take the outputs from the turntable and the front L/R outputs of the prepro and plug that into the analog input on the Mytek that could possibly be a workaround.  If you were confused before, I'm sure this only makes things worse. 

If 2-channel is relatively more important to you, a potentially cheaper way to go would be to get a decent AVR (Anthem, Yamaha, Marantz) to handle multichannel processing and amplifying your center and surround channels and buy a good integrated stereo amp with a HT bypass function.  With this setup, although you wouldn't be using the preamp function of the Brooklyn, only the DAC and integrated would be in the signal path for 2-channel and you could switch to HT with the push of one button.  This is the generally the way most people do it when combining a stereo and home theater system into one.  I think what you're trying to do is seamlessly pass through the signal from the Mytek through a prepro using an analog pass through (bypassing the volume control) on the prepro.  I'm not sure that pass through function exists on a prepro (unless maybe there's a way to use the tape loop to do it?), but regardless you'll end up having to send the DAC's signal through the prepro, which is really what you should be trying to avoid from a more purist perspective -- hence the recommendation for an integrated amp above. 

Now that I've probably got you thoroughly confused you I'll just sign off and leave you twisting.  Heh heh.  Just kidding -- I really am trying to help here.  Oh, and don't drink the water. That one's simple.