I know what you mean. Although, in actual fact, it doesn't do any harm for them to think that, since I'm the sort of person who's bound to make that mistake eventually (although I haven't yet, it seems increasingly unlikely no-one will notice on Sunday), it might be more productive for me to use the occasion for minor social bonding than to demonstrate that I'm marginally more accurate than my interlocutor :)