Privileged misinformation. Because you have inside knowledge of a situation, you're confident you understand it better than someone without that special knowledge. So you feel free to do things which would look utterly silly to that outsider, because you know why they're not such silly things to do after all, and the outsider doesn't.

Only, every so often, your inside knowledge turns out to be wrong. Then you have not only caused something bad to happen, but you've also made yourself look an utter fool to any watching outsiders. ‘I would have known,’ they think, ‘not to do that. What was he thinking?!’

If your inside knowledge had been right, then the results would have spoken for themselves: the outsider might have wondered what you were doing to start with, but when something obviously desirable happened in response, they would have understood that somehow you'd known it would and they hadn't. But if you try that and get it wrong, you'll never convince the watching outsiders that you weren't simply stupid.