Considering the "It's been a while friend" line and a couple scenes from Days, there's a slight chance that Xemnas might've also retained Terra's memories, or at least a portion of them.
I love how we keep assuming that it's gonna be Terra who turns out to be Xehanort and not someone whose face we haven't even seen yet.....hint hint.
I shall sum up my own theory:
1. "Xehanort" is the only thing the "young man" taken in by Ansem the Wise can remember, strongly implying it to not be his real name.
2. Yes, Terra's eyes change color from blue to yellow. So? Saix's hair stands up and goes crazy when he looks at the Kingdom Hearts moon. Riku's eyes turned yellow when he turned into "Ansem". But once again, there's a character whose face we haven't even seen yet! Doesn't that leave any room for discussion or theorizing?
3.Based on what we've seen in KH1 and KH2, Xehanort's Heartless is fond of identity theft (comedy drum sound). He takes the name of Ansem the Wise (whom he was apprenticed to). Is it really impossible that he cannot have done so, say, in Birth By Sleep after his first Master was killed?
I think you know what I'm saying. It's really more probably that Vanitas is the Xehanort we know.
Side note: The Terra you fight in KH2:FM+: Would it be too far-fetched to refer to him as, say, Terra's ghost, or unsent?