HOWEVER I realized that when I rotate the potentiometer, the L channel changes a bit the sound too, not as much as R but it still gets its volume changed... How can I prevent this? How can I control R channel volume without affecting the L channel?
Usually with an OP-Amp on the RHS and the LHS, so that the impedance is buffered.
You could run each into its own OP-Amp and then through a small resistor, like maybe 100 ohms, and then tie those 2 channels together to become one output..
If you did that, you would probably want a pot on each OP-Amp to alter the gain, and then you could balance it towards the left or right.
As it is now, one channel will pull the other channel up or down, as they are fighting each other when the image is not centered, and they are probably being fed with a around a 600 ohm output impedance, so each has an equal amount strength,