If the outlet was wired backwards could the amp feed line voltage through the RCA connectors ground wire?
Under most circumstances the answer to that would be no, John. With properly designed modern equipment the ground (shell) of the rca connectors is in common with signal ground, the chassis of the amp or other component, and ac safety ground. All of those are isolated from both the hot and the neutral of the ac wiring. The connection between chassis and ac safety ground is intended to cause the ac circuit breaker to trip in the event that an internal short makes the chassis "hot."
But given all the ways in which the outlet could have been miswired, combined with the possibility (as GS suggested) that the "cheap integrated amplifier" (as the op described it) could have had a defective line filter capacitor, or marginal insulation in its power transformer, combined with the facts that we don't know if a source component was connected, and whether the component(s) had two-prong or three-prong plugs, and whether or not the amp had been used in the recent past, there are probably more possible scenarios than it's practical to enumerate. The op should have the wiring checked out as has been suggested, and also have the innards of the amp looked at by a suitably experienced person, to try to identify what smoked.
Regards,
-- Al
