yes. definitely stay in the class. you will get a better grade than your test scores. he does that deliberately to make all the students study a lot.
i hate it and i was about to withdrawal. but after hearing your experience, i think i will tough it out.