Q:

Is it normal for things to be so cutthroat, and what to do about it?

I am a postdoc in mathematics. I like doing math, but I do not feel like I belong to a community, in the true sense of the word.
My vision of mathematical research is that it should be a massive collaborative effort to solve puzzles and publish the solutions, and no one should be excluded without a good reason. Also people should be allowed to openly talk about each others' results objectively without any feelings hurt.
However, in my field, people seem to see research as a competition. They form factions and play the game of inclusion-exclusion and loyalty matters more than truth, so does pride and it is almost impossible to be sure if an error is an error in a paper without making enemies.
People also treat knowledge as private property (within their factions or to oneself) and tend to keep secrets.
There are two types of secret-keeping generally practiced in my field:

When working on a project one would avoid talking about what they are working on and what are the partial results etc. in fear of being scooped.
When the work is done one should avoid proper exposition in the paper or explanation to the public so that:

they can stay ahead in the competition
they can make their results look as non-trivial as possible.

As much as I do not want to force people to disclose their secrets, I value clear explanations and talking to people about my work in progress. But as I do not belong to any faction I seem to be taking a huge risk on my career in doing so.
The fact that everyone is expecting secret-keeping also makes me reluctant to work with them because I do not want to sign an invisible NDA and unwittingly make enemies when I discuss with others on joint projects (and it is hard to set a boundary on discussion). This is not particularly helpful to my current state of isolation.
Perhaps it is just my field that has an environment incompatible with my belief, or perhaps all this is considered normal in any field. Nevertheless do you think this is a good reason for me to seek a different career path than academia?
Is it feasible to make a career doing open research?

A:

Reality is very complicated. It is not one way or the other, often a mixture, and lots of judicious decision-making is required. Some environments are nicer than other, some people more trustworthy than other. If atmosphere disturbs you, choose your environment and the people to suit your taste, there are more pleasant subfields than others. 
Careers do depend on individual success, so you cannot just plainly discount people trying to keep their ideas for themselves until they are ready to capitalise on that - success, as measured by hiring committees, is individual, not collective. 
Yes, a fully open community is a bit like communism - great if it works, but usually works only to a commune of a limited size, or if the work to be done cannot hope to progress more than differentially faster than the community as a whole  and also if there are no "bad apples" who do not play ball and prey on the goodwill of the group.
That being said, you can encounter very friendly communities, but if that is a priority for you, you need to include that into your criteria for choosing a research topic.

A:

Hello fellow postdoc in mathematics. 
I know what you're feeling and to an extent I agree; however, I've noticed that it's very variable in which field you're in in mathematics on how competitive colleagues are. Even in subfields in my main discipline, the amount of infighting and openness varies wildly. In my experience, in the more "puzzle" type research problems people are very competitive, while in the more "theory building" type research problems people are very communicative and open to dialogue.
If you find that your subfield in mathematics is overly competitive and you dislike that feeling, could you try to go to conferences a bit out of your field and see if that new field is as competitive?
My experience is that being in an intersection of various fields, some are much more open and willing to help than others. Some feel threatened by an "outsider" wandering in and others are happy to have another person joining the research landscape as they need all the help they can get. Only one project in my career have led me to need to be less than 100% open on what I am trying to accomplish, but this is very subfield specific.

A:

Let me first comment on the 2 types of secret keeping you mention:

When working on a project one would avoid talking about what they are working on and what are the partial results etc. in fear of being scooped.

I agree that this is common to some extent.  However, I think most people aren't really that secretive.  My experience is most people are willing to broadly talk about the kinds of problems they are working on, and share details with people they trust.  I trust most people I know in my field, and imagine my colleagues feel similarly.

When the work is done one should avoid proper exposition in the paper or explanation to the public so that:
1 they can stay ahead in the competition
2 they can make their results look as non-trivial as possible.

I've heard people give this as "advice," but I've always assumed it was tongue-in-cheek.  I usually try to present my ideas as straightforward as possible, but when I actually have to write up things in a paper, I find that writing all the details to make everything correct in generality makes it look more complicated than it is.  When I read a paper where explanations are hard to follow, I assume it is either (i) the author was lazy, or (ii) the author doesn't realize/forgets certain things aren't obvious if you haven't spent ages thinking about that particular problem. 
In fact, I've found most senior mathematicians to be quite generous with their ideas because one quickly has more ideas than time to explore them, let alone having time to steal other people's ideas.
Yes, mathematics is somewhat competitive, and some people are more competitive/secretive than others, but I and I think most of my colleagues have found it to be a friendly community where you can be open with a lot of other people.  That said, there are certain fields, and I think certain countries, that tend to be much more competitive than others.  It's possible you would have a better time in a different field, but I suggest talking to a couple people you trust in your area about your concerns.  It could be it is an somewhat of an issue in your area, or it could be that your impression is based on a few unrepresentative experiences/misunderstandings.
As to whether one can make a career of "open research," I think that depends on what is meant.  At this point, until you have a permanent job or attain enlightenment, you probably cannot make a successful research career of only being involved in massive open collaborations like the Polymath projects, though you can be involved in them.  However, if one just means being open with colleagues about what one is working on, then yes, I know people who are very open about their ongoing work.  It is certainly possible to have a successful career like this, though there may be some people you should stay away from.

