Can a peer-grading system still work despite malicious students? I'm asking this as a purely theoretical question, out of curiosity.

Let's say you are teaching an undergraduate class with complete autonomy. Instead of grading it traditionally, you decide to have the students grade each other. The goal is to significantly (by at least 50-90%) reduce the instructor's grading burden and promote self-assessment for students.

My initial instinct is that any such system will always be vulnerable to students collaborating to grade all papers 100 or 0, disregarding the rubric. Is this the case? Are there any peer-grading systems that work without the "honor principle"?
The goal of reducing grading effort in this way is malfeasance. As the instructor, it is your judgement that should be the dominant factor in student evaluation. And I'm a big believer and user of peer evaluation. But you can't substitute the judgement of non professionals for your own and expect any good outcome. 

Self-assessment is different from peer-assessment, by the way and it is also a useful and valuable thing to do, but not if the main goal is to make the instructor into a mere clerk. 

In an ethical system in which student evaluate and mark their peers the instructor effort is actually likely to increase, not decrease, since you have to both judge the original papers yourself and the evaluations given by others. It is valuable to do that if done well, but is inconsistent with the goal of reducing instructor effort. 

And of course, if the students need to expend effort to evaluate their peers the results should be part of their own evaluation as well. 

I've often used self and peer evaluation to get some feedback on what students do in a project setting where their work is done outside my direct view. I still have to evaluate the project, of course. But the self evaluation is to have students describe their own main contribution to the work. The peer evaluation is to ask students who were the top contributors to the effort and why they make that judgement. Note that all of the questions are stated positively. What it has taught me is that some students who don't appear to be particularly engaged in the classroom can be major contributors to group work. That is valuable to know and it has to come from students, but it doesn't reduce my effort.