My brain is an irreplaceable part of me, but one can always find somewhere else to live, so threats to the former are more scary.

Except that your brain has altered itself over the last ten years. You're not the person you were at college, you're just similar to him, and you're certainly very much unlike the person you were as a five year-old. So you're only the you you are now.