Do caps deteriorate with time even if an amp is not being used?

Lack of use and lack of cycling caps are their worst enemies. Particularly if the electrolitycs are of "exotic" type.

^^^^^^^^^^^
not an amp but still capacitors

I have a Studer tape machine that failed on me a few years ago. It would power up, but when you spooled up a tape and hit rewind, ........nada.

My tech told me "it’s probably the power supply capacitors. I have them in stock bring it over". As I waited and watched in amazement, if you have ever seen the inside of one of these machines; he checked out the power supply capacitors, and they were showing/testing "like new" . Huge things and I remember Made in West Germany labels.

The new, replacement ones were sitting on the table. So do we switch them out I asked. He said "No, I wouldn’t touch those."

I believe it is very dependent on what the amp is and the quality of components that were used. If it is mass-market junk then 10 years may be a big deal. If it is high quality equipment with good components, then 10 years should be nothing.

Electrolytic caps loose rated breakdown voltage and also dry out.  They loose rated breakdown voltage because electrolyte "eats out" dielectric - aluminum oxide (presence of voltage rebuilds this layer).  They dry out - especially when hot.  Each 10 degC temperature increase cuts life of electrolytic cap by half.  If you store gear in cool place caps should be good for 50 years, except for the breakdown voltage problem - that can be fixed by introducing voltage slowly.  SMPS power supplies might not make it possible since some of them don't operate below certain supply voltage. 

Yes. A capacitors two worst enemies are age and lack of use.
Caps rely a lot on memory, and require cycling (turning on/off) to keep their storage memory secure. Charge and discharge.
Not cycling an amp for years can certainly shorten the lifespan of the caps.

I don’t think the capacitors really wear out over time; instead, the electrolytic "paste" reacts with the metallized container or wrapping IIRC.

Other types of caps should not be (as) affected, but 10, 15, 20 years is about it (tho my Sunfire amp is still going strong).

Yes. Caps require replacing around 10-15 years w/ age.

A big YES! Though(often) careful reforming can restore them(their dielectric) to original specs. (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.823.6133&rep=rep1&type=pdf) If you’re going to buy that amp, buy a variac too(unless a friend owns one).  ie: (http://www.ebay.com/bhp/variac)