Google Scholar Syntax quirks: results depend on order of search strings I am working on a systematic literature review and noticed something quite disturbing since I want to base my work on Google Scholar results: The number and actual results returned by Google Scholar seem to depend on the order of the search terms (to some very relevant extent), at least if the OR operator is used.

This quirk poses a problem, to the extent that it sheds doubt on the reproducibility of the search results. How should I deal with this in my systematic literature review? Is there any plausible explanation for why the results depend on the order of search strings? Knowing this might help to find a workaround. The consequences of this behavior could actually be quite important to any academic user who searches for papers in their own research. Naturally we cannot definitely determine what's going within Google's algorithm as it is a blackbox but we still can come up with hypotheses that we might be able to test nevertheless (I cannot think of anything more scientific than that :))

What I found out so far: 

The number of results shown on the top of the results page is known to be a rough estimate that does not necessarily reflect the actual number. Furthermore, only the first ~1000 results/100 result pages are reachable anyway. However, neither is relevant to this question since it happens for far more specific searches with below 10 results as well.

Some examples with their number of results (the results themselves differ too).


10200: ("microarchitecture" "pipeline") OR ("pipeline" "instruction set")
38100: ("pipeline" "microarchitecture") OR ("pipeline" "instruction set")
39300: ("pipeline" "instruction set") OR ("microarchitecture" "pipeline") 
39800: ("pipeline" "microarchitecture") OR ("instruction set" "pipeline")
If you want to use google scholar as a means of measuring the number of papers on a certain topic: don't.

Contrary to what we often see in reviews or other publications, you did some more research than simply copying the number of results from a naïve search, and that research indicated that this number is highly unreliable (as should have been expected). Conclusion: Google scholar is not the right tool for the job. Its algorithms are unknown, and the sources it searches are unknown as well. Maybe it even counts the same paper multiple times, depending on which words you search for, who knows?

Having said that, in practice nobody really seems to mind an introduction stating "a Google search for XXX returned N results", but the question remains how useful such a statement is, given your findings.