"Do capacitors deteriorate just with age, or does it have to do more with use? Or some combination?"
Film caps can have an indefinite lifespan, longer than you, and I. Electrolytic caps age when used, or not used, in a speaker.
For the most part, electrolytic caps in an amplifier age both when used or unused, but more quickly when used, as the environment is higher voltage and higher heat.