The context I've most often come across people critising time spent on thinking is when they wish to contrast it with time spent on doing, or time spent on feeling. The label 'sad' gets applied to activities like playing Magic the Gathering, collecting Star Trek trivia or working out just how often the traffic lights in Cambridge will synchronise in such a way as to form a particular rune in red lights when seen from the sky. The reason, I think, is not so much that they are seen as non-productive, as that they are seen as anti-social.

If you want to fight this viewpoint, perhaps the parallel to draw is with going to the gym to exercise your body. Thinking is exercise for the mind.