I agree with Keef

What "Brown Sugar" says is that the frame of mind of contemporary white men pursuing black women is not that different in kind or type than that of slave owners abusing their charges, and the singer doesn't necessarily exclude himself from this analysis. You must admit that this is hard teaching, and something you might not want to confess when the cops are around. Doubts about whether they can still pull it off at their advanced ages may have played a part in the decision as well.