Audio Midi Setup.app can be found in the Utilities folder inside Applications on your Mac. Drag the icon into your dock so it will be handy.
In Audio Midi, select Audio Devices at the top, then set Default Output to the appropriate output device (built-in output if you are using the Toslink, a named device if you are using something like a USB DAC). Don't worry about System Output. Where it says Properties (left side of window), select the same output device you chose for Default Output. Then, under Audio Output (lower right portion of the window), you can set the word length and the sample rate. When people say to set playback to "native resolution," they mean to put those settings at the same sample rate and word length as the music file you are playing. Typically that is 44.100 and 16-bit. The options you can choose from are limited by the output device you have selected. My advice is to chose 24-bit word length if it is allowed, but keep the sample rate at 44.100 for standard "redbook" CD. After you change the settings in Audio Midi, quit iTunes and then reopen it and you are all set.
In Audio Midi, select Audio Devices at the top, then set Default Output to the appropriate output device (built-in output if you are using the Toslink, a named device if you are using something like a USB DAC). Don't worry about System Output. Where it says Properties (left side of window), select the same output device you chose for Default Output. Then, under Audio Output (lower right portion of the window), you can set the word length and the sample rate. When people say to set playback to "native resolution," they mean to put those settings at the same sample rate and word length as the music file you are playing. Typically that is 44.100 and 16-bit. The options you can choose from are limited by the output device you have selected. My advice is to chose 24-bit word length if it is allowed, but keep the sample rate at 44.100 for standard "redbook" CD. After you change the settings in Audio Midi, quit iTunes and then reopen it and you are all set.