I tried it without the buffer first and I will continue to listen both with and without. I owned the icOn Pro4 (balanced version) AVC preamp for awhile (link to review) and, for whatever reason, I didn't like the sound of the Pro4 with the buffer.
In the case of this Hattor ARP, the buffer does change the sound somewhat but, unlike the icOn Pro4 where I simply didn't like how it sounded with the buffer, the Hattor ARP it sounds pretty good both with and without, just different. I will probably be able to describe it better after I spend more time but, for now, the buffer seems to add a bit of body/fullness to the sound without a noticeable penalty.