Truthfully, that is the part he is worried about. The 'whither thou goest' bit. He isn't sure why -- well, he isn't exactly sure why, but it makes his heart pound incessantly, and brings about this feeling of being an idiot.

But he smiles nevertheless because some part of him is relieved by this. He turns to Yvaine, then, and nods. "Thank you," he says to her, holding out his hand.