Q:

Looking at the top X% of a population

Recently, I came across an article that claims shorter titles of scientific articles tend to result in a greater number of citations, implying concise titles are better. 
While this sounds plausible, the article uses only the top 2% of articles, ranked by the number of citations.
My questions is whether this is a valid approach to answer the research question: Does title length affect the number of citations of an article? 
Can't this only help explain a (small) difference in citations, given your article is already in the top 2% cited articles?
I don't understand what justifies this approach to the problem. What kind of bias does sampling from an upper quantile introduce in general and can any claim be made about the 98% with less (<16) citations?  

The advantage of short paper titles
Adrian Letchford, Helen Susannah Moat, Tobias Preis
R. Soc. open sci. 2015 2 150266; DOI: 10.1098/rsos.150266. Published 26 August 2015  
Same authors, similar strategy:
The advantage of simple paper abstracts
Adrian Letchford, Tobias Preis, Helen Susannah Moat
Journal of Informetrics, Volume 10, Issue 1, 2016, Pages 1-8, ISSN 1751-1577; https://doi.org/10.1016/j.joi.2015.11.001.

A:

Why did they only look at the top 2% citations?
For pretty much any distribution that looks concave, the datapoints in the upper quantiles will usually span a much larger range than those near the middle as the distribution tapers off. The probability distribution over the number of citations is probably a heavy-tailed distribution so the above must hold true. Since the objective of the study you cite was to quantify the influence of a variable (title length) on the number of citations, it makes sense that they would focus on the data near the tail where the sensitivity can be easily revealed. If instead, they had used the middle quantile, they wouldn't see much variation in the number of citations to quantify any effect.
Is their conclusion valid?
In general, whether or not it is okay to restrict your focus on the upper quantile of the population depends on the relationship between the variables your are attempting to model. Let $x$ and $y$ be the predictor and response variables such that:
$${y} = {f}(x) + \epsilon$$
for some function $f$ and independent noise $\epsilon$. Let $y_{(p)}$ denote the subset of all $y$ that lie in the neighbourhood of the $p$th percentile of the distribution $P(y)$, and $x_{(p)}=f^{-1}(y_{(p)})$ be the corresponding set of values of $x$.
Now suppose you choose two different quantiles around $p$th and $q$th percentiles of the population and use linear regression to model the relationship as $y_{(p)} = \beta_p x_{(p)}$ and $y_{(q)} = \beta_q x_{(q)}$ respectively. We have:
$$\beta_p=\frac{dy}{dx}\Bigr|_{\substack{x=x_{(p)}}}=f'(x)|_{\substack{x=x_{(p)}}}$$
$$\beta_q=\frac{dy}{dx}\Bigr|_{\substack{x=x_{(q)}}}=f'(x)|_{\substack{x=x_{(q)}}}$$
If $f(x)$ is linear in $x$, then it  wouldn't matter which neighbourhood you choose. For instance, if $f(x)=x$, then $\beta_p = \beta_q = 1$ and both quantiles will be representative of the full population.

However, if $f(x)$ is non-linear, then the conclusions you draw could depend drastically on the sample you choose. For example, if $f(x)=1/x$, then:
$$\beta_p=-\frac{1}{x_{(p)}^2}=-y_{(p)}^2$$
$$\beta_q=-\frac{1}{x_{(q)}^2}=-y_{(q)}^2$$
So $|\beta_p|$ >> $|\beta_q|$ if $y_{(p)}$ > $y_{(q)}$ yielding drastically different results for different quantiles of the population.

So, unless the true relationship happens to be linear, it would be wrong to draw general conclusions about the full population based on a biased sample.

