What contributes most to a change in how an amplifier sounds?

Output impedience.

If you follow the graphics in Stereophile, you see that the higher the output impedience, the greater the variation in frequency response based on a real life speaker load that is not a simple resistor.

All the organs were deciding who should be the boss....

"I should be in charge," said the brain , "I run all the body's systems, without me nothing would happen."

"I should be in charge," said the heart , "I circulate oxygen and nutrients all over."

"No! I should be in charge," said the stomach, "I process the food that gives us energy."

"I should be in charge," said the legs, "without me the body couldn't go anywhere."

"I should be in charge," said the eyes, "I allow the body to see where it goes."

"I should be in charge," said the asshole, "I am responsible for waste removal."

All of the other body parts laughed at the asshole and insulted him. So he shut down. Within a few days, the brain had a terrible headache, the stomach was bloated, the legs got wobbly, the eyes got watery, and the heart pumped toxic blood. They all decided that the asshole should be the boss.

What is the moral of the story? Even though everybody else does all of the work some asshole is usually in charge.

I gave the answer..., output impedience.

The frequency response of an amplifier will vary with the output impedience when pushing a real life speaker load, and not some fixed value resistor used to determine the power.

The lower the output impedience, the flatter the frequency response across the band. An amp with a high output impedience may vary by several decibels across the band when pushing a real life load.  You can't get around Ohm's Law.

You will hear a difference of several decibels across the band far more than the difference between say 0.01% and 0.001% distortion, or...

No the first graph is better as the power level, and hence volume, of the sound does not change with frequency.

In the second graph, the power goes up and speakers will get louder ~75 Hz and 1-2kHz and have a dip at ~5kHz so you would not attain a flat frequency response with that amp with that speaker load.

Note that the same "real life" speaker load is used for both amps.

If you people would read the measurements section of Stereophile, you would see that I’m correct.

Look at the frequency response for the Moon 861. Remember, we are talking power amps here, so no preamp functions (e.g., loudness) enter into the discussion.

And the discussion:

The output impedance in stereo mode was extremely low, at 0.008 ohms at 20Hz and 1kHz, rising slightly to 0.03 ohms at 20kHz. As the two output stages are in series in mono mode, the output impedances were twice the stereo values. In both stereo and mono modes, the variation in the frequency response with our standard simulated loudspeaker (fig.1, gray trace) was negligible.

Now look at the BAT REX 500 frequency response:

And the discussion:

The REX 500 amplifier’s output impedance, including the series impedance of 6’ of spaced-pair cable, was relatively high, at 0.45 ohms at low and middle frequencies and 0.6 ohms at the top of the audioband. As a result, the variation in the frequency response with our standard simulated loudspeaker (fig.1, gray trace) was ±0.3dB.

So while the response is flat into a load resistor, it varies by ~+-0.3 dB into a real life speaker load. That’s enough of a difference to hear as a difference. And of course every speaker is going to have a different load, so the amp is going to have a different response for each speaker dependent on that load.

Know that a lot of amps show considerably more variation than this.

https://www.stereophile.com/reference/707heavy/index.html