I just purchased a Terminator II and had this same concern. I emailed Alvin and he explained that the Terminator II uses an adaptive buffer to reclock the incoming signal. This helps reject jitter from poor sources.
This buffer can cause variable latency with some of the more common inputs. The USB input should have limited latency. Alvin didn't explain why this is.
I use my Terminator II with my PS5 and I have no latency issues.
I do use a Mutec to reclock before the DAC. This makes the jitter level coming in to the DAC likely very very low which could be why I don't see much latency.
YMMV.