As I understand it, the buffer is the amount of memory a software player reserves to stream music into RAM, then into the USB converter. For you this would be a setting in iTunes or Pure Music. I'm a PC using JPLAY, and the current beta has different buffer settings for different sampling frequencies. JRiver also has settings for prebuffering, which delays playback for a few seconds while music is loaded into memory.
There are other things that could cause the pops/clicks, like too many processes going on your Mac mini, but from the looks of your system it's a dedicated machine, so that's probably not it. Also, not having enough RAM can be a culprit.
Since you got the A2 going, do you still hear the clicks?