Low power, class-A solid state and high impedance speakers

It is my understanding that running a solid state power amp into high impedance loudspeakers (above 8 ohms nominal) will yield lower power output but also lower distortion. Assuming that the loudspeakers are sensitive enough not to suffer from the amp's lower power output, are there any downsides to doing this? Any other upsides besides lower distortion?

For example, a Pass Labs XA25 (rated 25wpc 8 ohms; 50 wpc 4 ohms) into DeVore O/93s (rated 10 ohm; 93db) or Zu Audio Soul Supremes (rated 16 ohm; 101db)?

I'm finding these combinations quite subjectively satisfying but wonder if there are any other objective performance issues to consider.

I appreciate all the knowledge on this forum. It's been invaluable to my pandemic audio adventures. Many thanks.