Should I list conferences I attended on my CV? I am a PhD student in mathematics, and I noticed that on my peers' CV's, they often list conferences they attended. It seems as if they just attended these conferences, they did not help organize, nor did they give a talk.

Should I list conferences I only attended on my CV, or does it look like obvious filler?
In general, you can put pretty much whatever you want in your CV (unless there is a template, in which case deciding what to put or not should be pretty straight-forward). The question is: what is the point of putting a piece of information in your CV?

Usually, a CV is used in the context of a recruitment process, where the point, in the end, is to give reasons to the recruiter to recruit you instead of someone else. Hence, a piece of information contained in a CV should be instructive, for instance by ensuring that you do have the proper credentials, or by providing evidence that you can be apt to the position you are applying for. 

Perhaps attending conferences is something relatively rare in your field, and you have attended more than the average PhD students? In that case, listing all the conferences you have attended could indicate you are very interested in keeping up with recent advances in your field. It could also show that you enjoy travelling. Perhaps you have secure internal or external funding to attend that conference (i.e., the funding was not automatically given)? That's also good evidence. You could list them if it provides evidence for an aspect that you are willing to emphasise in your application. 

On the other hand, if the conferences you have attended had nothing special, or do not help emphasising any particular point, then you might want to keep the space in your CV for something else.