Can I submit my paper to arXiv with a header that says "Under review at X" when it isn't? Previously I submitted my paper in MS Word format to arXiv.  Now after almost two years, I decided to submit an updated version in LaTeX format.

To create the LaTeX version of the document, I downloaded a template from here that added a line at the top of each page saying:


  Under review as a conference paper at ICLR 2016


Do I have to remove that notice, or is it OK to leave it as is?
Remove it.  It is not appropriate to mislead people as to the submission or publication status of your paper.  You should also remove any other reference to the conference appearing in the paper (e.g. a logo).

Additionally, you'll have to upload your LaTeX source to arXiv in order to have it posted, including the conference style file, so you should make sure its copyright allows this.

(Even if you really had submitted the paper to that conference, you typically wouldn't acknowledge this in the preprint.  The reason is simple: if the paper isn't actually published in that conference, people will be able to infer that it must have been rejected, and this is something you'd probably rather were not publicly known.)

Update: Here is a question from someone who used a template that included the name of Journal X when submitting to Journal Y, and had the paper rejected on suspicion of double submission.  While I think that Journal Y was out of line in doing this, it does illustrate that these sort of misunderstandings can have serious consequences.