Via a friend's Facebook feed comes this excellent article about "the psychology of software engineers and what makes us the way we are." I'm in almost complete agreement with the article, although I do somewhat take issue with this paragraph:

Part of the problem [of software engineers consistently giving overoptimistic estimates and then failing to meet them] is also our fragile egos. We get afraid that if we give an estimate that is "too long", that people will think less of us. "Good engineers" should be able to work faster, they say, and so we acquiesce.

I don't really think software engineers in general have "fragile egos," or at least not to a greater extent than people in any other profession. We are notoriously bad at giving realistic estimates, but in my experience, this is often because we're actually incentivized to do so: deadlines are set before the amount of work required for a project is actually determined, or without regard to it, and then we're made to fit our estimates into too-short iterations while still trying to accomplish all we set out to do. There's a common sentiment that it's better to be under-optimistic and out-perform your goals than the converse, and while that seems true in theory, in practice there don't seem to be many discouragements to failing to live up to estimates that everyone already knows are unrealistic and often meaningless.

By contrast, I think the fear that people will think less of us for giving "too long" estimates is completely valid in a culture where it's commonplace to set unrealistic goals and then break our backs trying to achieve them by any means necessary. I think this especially true at the feature team level when one is working on a large piece of software. What do you expect to happen to a team that estimates up front that its feature cannot be completed within the release window? (Remember, adding more engineers might just make the problem worse.) I think it's safe to say that it won't involve promotions and pay raises.

Finally, as my wife pointed out when we discussed this article, the planning fallacy is by no means unique to software engineers.

All of that said, I think the rest of the article is very good; this part in particular struck a chord for me (emphasis mine):

So, without enough information, changing requirements, not enough knowledge to do the job, and people constantly second guessing us, we trudge into work every day. Being creative people, we put up with all of this because we know that one day people will use our work. That's really what drives software engineers more than anything else: the idea that people we don't even know will be affected by our work. Whether you’re working on a web site visited by millions each day or you’re working on a point-of-sale system for restaurants, the knowledge that we’re affecting people's lives is a powerful driver.