Audirvana

Hi, you'd need to connect the computer to the DAC in order for the Audirvana to detect it as a DAC.

https://www.denafrips.com/audirvana

The Denon DRA-800H is a network receiver, and there is no USB connection to the computer. It's unlikely the Audirvana can detect it as a DAC...

Hope this helps!

If I got a usb dac and connected it via rca would that bypass the denon internal dac

Yes, and that is your solution, unless your computer includes a digital optical output, as the DRA-800H seems to have digital optical inputs (great for connecting digital audio from your tv, typically up to 24/192 resolution).
It will be easy to find a better DAC than the one onboard in your receiver.
One inexpensive option:
https://www.schiit.com/products/modi-1 for $100.

Hi Leroy,
Another possible option:  
Your DRA-800H  seems to fully support a Wired LAN connection, as shown on page 40 of your manual.  Try a direct connection from your computer's ethernet port to the LAN input port.  If you can get your LAN set up and functioning, that might be the best of all worlds.  Looks like an interesting receiver, good luck.

I've hooked up an optical from pc to the denon and it still doesn't detect the denon as a dac 

Do you have guest network? Generally when the 2 components are NOT on the same network, these issue happen. Make sure your Wireless and Ethernet network are the same. Or at the least your wifi is not connected to a guest network.

Just a thought, as it happens sometimes between my laptop connected to the ethernet and the iPad which is wirelessly connected.

Both are hard wired to the router, pc and denon