I have a PS3 and tried it with the Integra 9.8. They worked together fine once the proper PS3 AV setup settings are selected.
You have to do what Kal suggested and select the apropriate track from the disc menu first.
A BIG part of your issue is the PS3 setup. You mentioned you connected it via HDMI and the 9.8 did not automatically select the right mode for your Blu-ray discs. That is because the person who told you to setup the PS3 to bitstream is wrong. The PS3 physically cannot bitstream the newer codecs (DD TrueHD, DTS MA, etc.).
You have to let the PS3 decode internally and send LPCM data to the 9.8. When you do this, the 9.8 will not show you DD TrueHD, DTS MA, etc. on the display since it is not decoding internally. It will say either PCM, 5.1 PCM, or 7.1 PCM on the display. It simply an issue of where decoding is taking place, either in the PS3 or in the 9.8.
I do not think anything is wrong with your unit, it is a PS3 setup issue. This drove me nuts for a few hours until I figured it out. If you want to send bitstreamed data to the 9.8 from the Blu-ray player, you need to get something other than the PS3. I got the Sony BDP-S550 and it will bitstream to the 9.8 and the 9.8's front panel lights up with the correct mode automatically once you select it from the disc.
Both the PS3 and 550 are excellent Blu-ray players. Its just a matter of where you want the decoding to take place and other personal perferences.
Good luck!!