How should we evaluate our peers? This is a quite broad question, to which I have some personal answers, but I think I need more different perspectives.

I am not talking about evaluating a peer's submitted work as a referee, but about evaluating applications ranging from small grants or sabbaticals (which have to be applied to at least in France) to tenure or tenure-track hiring. Such decisions have a small or huge impact on both others career and the overall academic system (by giving certain incentives), and should thus be taken very seriously. On the other hand, the time we are able to use for such evaluations is limited.


  How should we deal with this opposing constraints (evaluation is serious matter but time is limited)? What tools and proxys should we use or avoid in the evaluation process?


I am not asking about how the system should be (I know my answer: more automatic funding, sabbaticals and pay raises to have less evaluations made more thoroughly), but about what one wanting to take evaluation seriously (as opposed to a formal game that only needs an arbitrary answer) can do in the current system. Actions that might imply change in the system are welcome, as far as they can be implemented individually and the plausible outcome is considered rather than the ideal outcome if everyone did the same thing.

Edit in view of comments

Clarification about "the system": if needed please precise in your answer the relevant bits of information about the evaluation system you are speaking about, as of course things evolve and vary from place to place -- but the broad principles should be applicable regardless of the fine details so please keep this as concise as is relevant.

Example of subquestions: it has often been said that impact factor, or more generally the glamor of journal titles should not be used for evaluation. Is there a consensus about this? I don't know. They are used in practice, but are there good alternatives? Are there specific evaluation circumstances when one should use this proxy? 

What about h-index and other citation metrics? When should they and should they not be evaluated? 

The question is not so much about how much time one should spend (which is more or less a given), but how to achieve a reasonable result in that time. Also, what to do in a situation where one feels a reasonable result cannot be achieved?
I'm only going to answer the first part of this question:


  How should we deal with this opposing constraints (evaluation is serious matter but time is limited)? 


It is true that it is often rather time consuming to do these evaluations. But I think ultimately, you need to apply the categorical imperative. In other words, you need to spend as much time and diligence on evaluations as you would expect your evaluators to spend on yours.

For a two-page application for a sabbatical, this may not be very much. But for someone applying for tenure, surely a thorough reading of at least the most important papers, along with some research about the impact of the candidate's work, is appropriate. This is thankless work, as you don't usually get a lot of credit for it. But it is nevertheless something that can't be taken lightly because of its outsize effect on the candidate and the importance of your evaluation. Consequently, if the case is not completely straightforward, if you find that you need to spend a couple of days on it, then that is what you need to do -- because you would your evaluators to do the same.