Theil index

The Theil index is a statistic primarily used to measure economic inequality and other economic phenomena, though it has also been used to measure racial segregation. 
The first presentation of this method of measuring inequality was builded up on statistical information theory. This original presentation did not seem much intuitive (as e.g. an economist A. Sen proclaimed). The Theil index "T"T is the same as redundancy in information theory which is the maximum possible entropy of the data minus the observed entropy. It is a special case of the generalized entropy index. It can be viewed as a measure of redundancy, lack of diversity, isolation, segregation, inequality, non-randomness, and compressibility. It was proposed by a Dutch econometrician Henri Theil (1924-2000) at the Erasmus University Rotterdam.
Henri Theil himself said (1967): "The (Theil) index can be interpreted as the expected information content of the indirect message which transforms the population shares as prior probabilities into the income shares as posterior probabilities."
An Indian economist and philosopher Amartya Sen (*1933) claimed about the Theil index: "But the fact remains that the Theil index is an arbitrary formula, and the average of the logarithms of the reciprocals of income shares weighted by income is not a measure that is exactly overflowing with intuitive sense."
Formula.
For a population of "N" "agents" each with characteristic "x", the situation may be represented by the list "x""i" ("i" = 1...,"N") where "x""i" is the characteristic of agent "i". For example, if the characteristic is income, then "xi" is the income of agent "i". 
The Theil "T" index is defined as
and the Theil "L" index is defined as
where formula_3 is the mean income:
The Theil L formula represents the logarithm of the geometric-mean of the ratio: (mean income)/(income i), over all the incomes included in the summation. ...obviously a relevant fact for any range of incomes on the same side of the mean income.
...showing that this form of the Theil has obvious, intuitive, plausible and natural justification, rather than just being justified in terms of entropy.
Because a transfer from a larger income to a smaller one will change the smaller income's ratio more than it changes the larger income's ratio, the transfer-principle is satisfied by this index.
Of course, if desired, a weighting factor, such as (mean income)/(income i) could be included in the terms of the summation (as in the Theil-T formula above, with the income-ratios inverted), to weight the index in favor of more strongly counting changes to income-ratios in which income i differs from the mean income by a larger factor.
In Theil T, each income-ratio's logarithm is weighted by a factor equal to that income-ratio's own value. And so, if the income-ratio is 2, then the index's value is affected as if there were two of that person. ...a reasonable weighting if each income-ratio's importance is judged to be proportional to its own value...the factor by which a particular income differs from the mean income.
Equivalently, if the situation is characterized by a discrete distribution function "f""k" ("k" = 0...,"W") where "f""k" is the fraction of the population with income "k" and "W" = "Nμ" is the total income, then formula_5 and the Theil index is:
where formula_3 is again the mean income:
Note that in this case income "k" is an integer and "k=1" represents the smallest increment of income possible (e.g., cents).
if the situation is characterized by a continuous distribution function "f"("k") (supported from 0 to infinity) where "f"("k") "dk" is the fraction of the population with income "k" to "k" + "dk", then the Theil index is:
where the mean is:
Theil indices for some common continuous probability distributions are given in the table below:
If everyone has the same income, then "T"T equals 0. If one person has all the income, then "T"T gives the result formula_11, which is maximum inequality. Dividing "T"T by formula_11 can normalize the equation to range from 0 to 1, but then the independence axiom is violated: formula_13 and does not qualify as a measure of inequality.
The Theil index measures an entropic "distance" the population is away from the egalitarian state of everyone having the same income. The numerical result is in terms of negative entropy so that a higher number indicates more order that is further away from the complete equality. Formulating the index to represent negative entropy instead of entropy allows it to be a measure of inequality rather than equality.
Relation to Atkinson Index.
The Theil index can be transformed into an Atkinson index, which has a range between 0 and 1 (0% and 100%), where 0 indicates perfect equality and 1 (100%) indicates maximum inequality. (See Generalized entropy index for the transformation.)
Derivation from entropy.
The Theil index is derived from Shannon's measure of information entropy formula_14, where entropy is a measure of randomness in a given set of information. In information theory, physics, and the Theil index, the general form of entropy is
When looking at the distribution of income in a population, formula_17 is equal to the ratio of a particular individual's income to the total income of the entire population. This gives the observed entropy formula_23 of a population to be:
The Theil index formula_32 measures how far the observed entropy (formula_23, which represents how randomly income is distributed) is from the highest possible entropy (formula_34, which represents income being maximally distributed amongst individuals in the population– a distribution analogous to the [most likely] outcome of an infinite number of random coin tosses: an equal distribution of heads and tails). Therefore, the Theil index is the difference between the theoretical maximum entropy (which would be reached if the incomes of every individual were equal) minus the observed entropy:
When formula_30 is in units of population/species, formula_23 is a measure of biodiversity and is called the Shannon index. If the Theil index is used with x=population/species, it is a measure of inequality of population among a set of species, or "bio-isolation" as opposed to "wealth isolation".
The Theil index measures what is called redundancy in information theory. It is the left over "information space" that was not utilized to convey information, which reduces the effectiveness of the price signal. The Theil index is a measure of the redundancy of income (or other measure of wealth) in some individuals. Redundancy in some individuals implies scarcity in others. A high Theil index indicates the total income is not distributed evenly among individuals in the same way an uncompressed text file does not have a similar number of byte locations assigned to the available unique byte characters. 
Decomposability.
According to the World Bank,"The best-known entropy measures are Theil’s T (formula_32) and Theil’s L (formula_39), both of which allow one to decompose inequality into the part that is due to inequality within areas (e.g. urban, rural) and the part that is due to differences between areas (e.g. the rural-urban income gap). Typically at least three-quarters of inequality in a country is due to within-group inequality, and the remaining quarter to between-group differences."
If the population is divided into formula_40 subgroups and 
then Theil's T index is
For example, inequality within the United States is the average inequality within each state, weighted by state income, plus the inequality between states.
The decomposition of the Theil index which identifies the share attributable to the between-region component becomes a helpful tool for the positive analysis of regional inequality as it suggests the relative importance of spatial dimension of inequality.
Theil's "T" versus Theil's "L".
Both Theil's "T" and Theil's "L" are decomposable. The difference between them is based on the part of the outcomes distribution that each is used for. Indexes of inequality in the generalized entropy (GE) family are more sensitive to differences in income shares among the poor or among the rich depending on a parameter that defines the GE index. The smaller the parameter value for GE, the more sensitive it is to differences at the bottom of the distribution.
The decomposability is a property of the Theil index which the more popular Gini coefficient does not offer. The Gini coefficient is more intuitive to many people since it is based on the Lorenz curve. However, it is not easily decomposable like the Theil.
Applications.
In addition to multitude of economic applications, the Theil index has been applied to assess performance of irrigation systems and distribution of software metrics.
Application in OECD.
The Theil index is used for measuring regional inequalities by OECD (the Organisation for Economic Co-operation and Development), whereas the Theil index is defined as
formula_52 
where formula_53 is the number of regions n the OECD, formula_54 is the variable of interest in the formula_55 region (e.g. life expactancy, household income, homicide rate...) and formula_56 is the mean of the given variable of interest across all regions. 
The interpretation is following: The Theil index ranges between 0 and formula_57, where zero represents an equal distribution and any other (higher) value represents a higher level of disproportion. 
"Note:" The index allocates an equal weight to each region irrespective of its extent; hence differencies in the values of the index among countries could be partially due to differences in the average size of regions in each of the countries. 