Is it just me, or is it really difficult to say that something simply isn't funny?

Whenever I use the phrase ‘that's not funny’, I always feel as if it has connotations of being tasteless, or tactless, or offensive; as if it indicates that whatever joke I'm describing has something actively wrong with it that cancels out any humour or wit it might possess. I feel this so strongly that I find myself unwilling to use the phrase about a joke which merely lacks sufficient humour or wit to have managed to make me laugh; one which there's nothing obviously wrong with, just nothing particularly right with either.

Am I reading too much into the usual usage of the phrase? Are there plenty of people around who would hear me say ‘That's not funny’, and understand by it no more than I had literally said?