iMac in a home Hi-Fi system.

Apple lossless or Aiff are the best quality. Either one is fine.

You will need an external DAC using toslink(optical) via the headphone out jack on the mac. Then a set of RCA's from the DAC to any free inputs on the pre-amp.

Under 'sounds' select the external dac for output.

If you have a Cd player with 'optical digital in' you can use that in place of the DAC. Either way you will need a toslink cable - toslink to toslink rca mini.

In the setting of the audio input and output: how to select sampling rate and quantization?
44100, 48000, 96000 Hz? and 16, 20, 24 bit?

Yes Vlady, what Meiwan advised is correct. In essence from your appleloss source which is digital, you need to have it converted by a Dac to analog. Fom there you can plug it into your analog system pretty much normally. But as to your second question about upsampling.

Vlady:> How to select sampling rate and quantization?
44100, 48000, 96000 Hz? and 16, 20, 24 bit?

You definitely want 24-bit, so you will be ready when your source is also 24-bit. There is much controversy as to whether or not higher up sampling rates will be audible on your system.

See: High Definition Audio (HD)

There are many good USB DACs out there as well. But do your reading, as many DACs have poor USB implementation. However, good USB implementation has been found by many to be preferable to Toslink with many DACs.

You should also download a "more audiophile" music playback program than iTunes, although many of them still use iTunes as a front-end interface. This is not an exhaustive review, but Bitstream is the cheapest, and many people like it. You can download it for $10 from the App Store. I use Pure Music on my home computer and Audirvana on my laptop. They each have their pros and cons, but I think I like Audivarna slightly better. Decibel gets some good reviews, and Amarra is considered by many to be the best (or at least the most expensive).

When ripping discs with iTunes, make sure you have the error correction on. Some people prefer using other programs, like Max or XLD to rip discs. I think iTunes works fine.