The impedance of the speaker can easily make a difference. In some cases can change the class of operation too (lower impedance pushes the amps more towards the B region; yet another reason to avoid low impedance speakers if possible).
With a lower impedance load more current can flow through the output devices at idle. Since they already have a quiescent current (idle current), depending on how hard the output circuit is biased ('hard' meaning more bias current; perhaps the designer wanted more than just one or two watts of operation in the Class A region) reducing the load impedance can easily raise the temperature. I think you will find that with the other amp on the same load that it runs warmer too, just not so much that you notice it.
With a lower impedance load more current can flow through the output devices at idle. Since they already have a quiescent current (idle current), depending on how hard the output circuit is biased ('hard' meaning more bias current; perhaps the designer wanted more than just one or two watts of operation in the Class A region) reducing the load impedance can easily raise the temperature. I think you will find that with the other amp on the same load that it runs warmer too, just not so much that you notice it.