Another thought about the OP's post. Writing in 2004 and bemoaning the paucity of "insightful and intelligent lyrics" also translates into not being able or willing to branch out into other genres. The artists mentioned in the OP were all comfortable, safe, white Boomer artists firmly entrenched in that cultural paradigm of lyrical, non-confrontational poetry which, compared to the world of the first decade of the 21st century, was analogous to the mushy comfort of mashed potatoes and stuffing.
And yet in 2004 you had Nas and KRS-1 perpetuating and redefining intelligent, insightful lyrics; you had Lady Gaga and Prince penning masterful lyrics for themselves and other artists; you had "conventional" singer-songwriters like Josh Rouse, Neko Case, and even Tori Amos all writing powerful, poetic lyrics.
I realize my dinosaur comment was condescending, but anyone who bemoans the lack of anything in contemporary music and then proceeds to reel off artists from decades ago is simply confessing a static perspective.