Another possibility is the I2S is not synched properly ,sadly there is no standard for I2S, why as like many things in Audio in digital no Standards where there should have been for years for example all recordings should have exactly the same spl level when mastering ths was proposed way back in the 60s on tape
but no, why isn’t the -I2S - HDMI all be a standard instead of several possible pin outs ? Which makes this a pain. In the Asss, but this is what you may have to do trial and error .