Unfortunately, Blu-ray content on the Xbox 360 can be a major pain. Some content may work without any transcoding, but others will require it due to the inherent playback limitations of the Xbox.
You may be best off maintaining a transcoded copy of the source just for your Xbox extenders and then keep things simple for the HTPC and maintain an unmolested BDMV or ISO which you can use to derive the lossless audio track and ship off as PCM to your AVR.
