I think Xehanort could have been a keyblade wielder but either lost his power or forgot it, seeing as how he'd lost all his memory when he was found. That explains how when his heartless posessed Riku that he was able to get a keyblade out of nowhere, and it also explains how he got a keyblade to give to Kairi toward the end of KH2.
What do you guys think of that? I think it's really very possible.