When you go lower ohms - I hate saying lower load because technically that’s a tougher load for the cartridge - you can start to lose noticeable signal, like 1 dB and more as you slide down below a 10x coil-to-load ratio (about half a dB at a 20x ratio).
This is because of the voltage divider formed between the cartridge coil and load. This only applies to voltage-amplification mode phono stages and SUTs, as intactaudio will point out (though current mode stages are still much less common).
So you may need to adjust the volume slightly on different loads, to be fully fair in your comparisons. You can often notice the difference going from say 50 ohms to 200 ohms (e.g. adjust the volume up almost a dB for 50 ohms load) or vice versa. But at some point, throwing away too much signal (too low of a load resistance) is definitely bad - you’re lowering signal-to-noise ratio for one.