If you're gonna put Kanye West and Andre 3000, then you have to put a few other rappers who are undisputedly better lyricists than Kanye, and song-for-song and verse-for-verse over their careers have put out about as much good material as Kanye and Andre 3000.
1. Redman (just look at his catalogue, consistency of his material, his influence on rappers and on the game in general, his lyricism)
2. Ghostface Killah (catalogue on Wu group albums, his solo stuff, his lyricism, his guest features)
3. Raekwon (same as Ghostface)
4. Scarface (his Geto Boys stuff, his solo catalogue, his influence on the South, his lyricism and appeal to emotion)
I think if you do song-for-song then Snoop and Busta Rhymes have to be in the discussion (even if they aren't considered the best). If skills are the main factor, then you have to add Big L and Big Pun (especially if you're adding Biggie to the convo).
And if we're talking purely about skill, i'd even reach far enough to put Mos Def in the discussion for best.