Will I look bad if I'm never in my office as a graduate student? I just checked out my office. It looks like a dungeon. There are no windows. It is kind of cramped. Everything looks kind of industrial. I don't think I would like to study/work there. There are other places in the math building that look like much less depressing places to work such as the math library. Also coffee shops, at home, etc. 

I have to have some number of office hours which I will gladly have. But for studying for my classes, working on research, etc. I would much prefer other places. There is no formal rules on where you have to be. I don't have an adviser yet and when I do, he won't be watching over me just looking for results. So there is no formal reason why I have to be in my office other than office hours. 

If I simply don't use my office will I look bad to other grad students or professors? Will I maybe miss out on productive discussions? I can see if my office mates are in the same classes; we can work on the problem sets together.  

What are the potential pitfalls of not being in my office other than for office hours?
Look bad, not necessarily. Will it be good for you? Maybe not.

Assuming that you don't have a really significant reason not to be on the lab, you should be. Stuff only happens with people that show up. If you are not in the lab, you won't talk to people as much. I literally can't count how many good ideas (and papers) came out of random coffee talk, that wouldn't happen on an otherwise "official" meeting.

full disclosure: That's something I didn't do during my phd and I regret it. I had my reasons, probably not really good enough, but I was almost never at my office and that hindered my development. I could have done more if I was there.