Effective reviewing in top tier Computer Science conferences Most top conferences in Computer Science receive a record number of submissions every year, like AAAI received almost 9000 submissions this year, while CVPR 2020 received 6600 submissions. That would mean atleast 27000 and 20000 reviews respectively. That would mean almost 5000 and 4000 reviewers respectively, assuming 5 papers per reviewer. 

I've also come across a few posts of area chairs (of these top-tier conferences) expressing their frustration of not getting reviews from the reviewers in time, and as a result there is a compromise on the quality of reviews. Is the burgeoning rise in ML/AI leading to falling standards? Is there an effective solution to this? Do such conferences have a way to check the quality of reviewers given that a lot of them are PhD students? 

One criteria mentioned on their sites is that the reviewer should have 1-2 publications in such top-tier conferences, but is that enough? For instance, I have 3-4 publications (not in highly ranked conferences/journals), but I don't think I can provide effective reviews for papers submitted to those particular conferences/journals. Is there some additional criteria as well?
Anecdotal evidence: IEEE ICDM used to demand that all reviewers hold doctor titles, so there were no PhD students in the Program Committee. Then, in one particular year, all PC members were asked to review 17 papers (instead of the promised, and usual at ICDM, batch size of 8-10 papers), in a reviewing period of 16 days (including weekends). That did not lead to high-quality reviews either, and as additional fallout damage, several PC members declined the invitation for next year's ICDM, amplifying the problem.



I am not aware of any reasonable solution from the point of view of conference organizers:


If we put criteria on reviewers, then fewer reviewers will be available, so the available reviewers must each do more work, which reduces their availability for other conferences.
IJCAI this year asked their area chairs to rather aggressively desk reject papers: about half of all papers were filtered out before reviewing even begun. This decreases the load on PC members, but increases the load on area chairs, and it will inevitably increase the already unacceptably high variance inherent in the reviewing process: not all desk rejected papers will have deserved to be desk rejected, which is very unfair on those authors.
ICLR does open reviewing, which means that reviews and reviewer identities are published on the internet. Since people know that their name will be available along with the review they wrote, only those people that know that they can invest a substantial amount of time will accept a reviewing invitation. This means that the reviewer pool will be smaller, which increases the load on individual PC members...
If a conference were to monitor the performance of reviewers, and would somehow* manage to evaluate the quality of the reviews, then what? Suppose that PC members do a bad job. Do you kick them out? Then we have even fewer PC members to complete the job next year, increasing the load on the other PC members...


*how would we do this, though? Should the area chairs assess this? That would massively increase the workload of area chairs, which is probably not a good idea given the overload problems the rest of this post outlines.



As a scientist who tries to publish papers, I see two solutions on how to deal with the problem:


Grow a thick skin when receiving conference reviews and trust that in the long run, the bias towards good papers will outweigh the variance in individual conference decisions.
Submit your paper to a journal instead; if your papers gets an unjust bad review for a journal, you can at least discuss with the editor.