<jupyter_start><jupyter_text>Grammar of Graphics  Introduction The *Grammar of Graphics* is a way of understanding the representation of data that differs from the 'traditional' way of thinking in terms of specific kinds of plot. It was pioneered by <NAME> in the early 2000s, but massively popularised, and very well explained, by <NAME> and applied in the incredibly popular `ggplot` libraries in `R`.* <NAME>'s ["A Layered Grammar of Graphics"](http://vita.had.co.nz/papers/layered-grammar.pdf)* [Review](http://www.r-bloggers.com/the-grammar-of-graphics-l-wilkinson/) of "The Grammar of Graphics" (Wilkinson)* `ggplot2`: [home page](http://ggplot2.org/) The central premise of the *grammar of graphics* is data and its representation are handled separately. This allows for components of a plot to be customised easily to achieve a specific representation that allows you to explore and understand your data, or explains it to others, effectively.If you are coming to the *grammar of graphics* for the first time, it can appear quite non-intuitive, but I hope you will see that it has many advantages over considering graphical representation in terms of archetypal 'types' of plot.In this notebook you will learn how the *grammar of graphics* is applied in the Python module `ggplot`, and how to apply its principles to your own datasets.![grammar of graphics example](images/gog.png)  Python importsThe code in the cell below suppresses noisy warnings from `matplotlib` and `pandas`<jupyter_code>import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Learning Outcomes* Understand the Grammar of Graphics* Use the Grammar of Graphics with the `ggplot` module to produce a scatterplot from *aesthetics* and *geometric representations*.* Use *layers* to produce new visualisations specifically to suit your data, showing data and statistical summaries* Use multi-panel figures to display complex datasets Exercise 1. Producing a Basic Scatterplot In this part of the exercise, you will produce a basic scatterplot of socioeconomic data, and work through understanding the elements of the plot and how they relate to the *grammar of graphics*.  Python importsThe `ggplot` Python module implements the *grammar of graphics* in a style similar to that used for `R`'s `ggplot2` library* `ggplot` module: [project page](http://ggplot.yhathq.com/)and we will use it for this exercise. This module uses matplotlib as a backend, and if you are running in a virtual environment on OSX, it is likely that you will not have this installed as a Framework, so you will need to use the %pylab inline magic.`ggplot` is intended to work well with the `DataFrame` representations of the `Pandas` module - a little like a reimplementation of `R`'s data philosophy in Python, so we also import this module. ```python%matplotlib inlinefrom ggplot import *import pandas as pd```<jupyter_code># Import ggplot and pandas modules
%matplotlib inline

from ggplot import *

import pandas as pd<jupyter_output><empty_output><jupyter_text>Importing dataWe will need some data to work with, and for this we will use data from the `R` package `gapminder`, which describes an excerpt of the Gapminder data on life expectancy, GDP per capita, and population by country.* `gapminder` data: [R documentation](https://cran.r-project.org/web/packages/gapminder/index.html)* `pandas`: [documentation](http://pandas.pydata.org/pandas-docs/stable/)This is located under this repository's root directory in the `data` subdirectory in tab-separated tabular format, as `gapminder.tab`. We can import this to a DataFrame in the variable `gapminder` using `pandas`:```pythongapminder = pd.read_csv("../../data/gapminder.tab", sep="\t")```<jupyter_code># Import gapminder data into the variable gapminder
gapminder = pd.read_csv("../../data/gapminder.tab", sep="\t")<jupyter_output><empty_output><jupyter_text>We can peek at the data in a `pandas` dataframe with the `.head()` method, which shows is the first few lines, along with header and row index information. The `.describe()` method will show summary information on the data in each column.To see what data type each column holds, we can look at the `.dtypes` attribute of the dataframe```pythongapminder.head()gapminder.describe()gapminder.dtypes```* ***QUESTION:*** What data is contained in the `gapminder` dataframe? How is it organised?* ***QUESTION:*** What are the minimum and maximum values in each column?*  ***QUESTION:*** What data types are `country`, `pop` and `lifeExp`? Why do you think these columns have these data types?<jupyter_code># Peek at the gapminder data<jupyter_output><empty_output><jupyter_text>The elements of a scatterplotWe can plot the data in `gapminder` using `ggplot` as though it were any other plotting library, just to render a basic scatterplot. We will do this by plotting life expectancy against GDP per capita with `ggplot`'s `qplot()` function. This emulates the `qplot()` function in `R`'s `ggplot2` library, but has this Python-specific difference:* **dataframe variables must be provided as strings**```pythonqplot('lifeExp', 'gdpPercap', data=gapminder, color='continent')```<jupyter_code># Use qplot to produce a basic scatterplot of gapminder data
qplot('lifeExp', 'gdpPercap', data=gapminder, color='continent')<jupyter_output><empty_output><jupyter_text>2. Understanding a Scatterplot It is likely that you can treat the scatterplot you have just produced quite intuitively. After all, you've probably seen a few scatterplots in your time.The plot has *x*- and *y*- **axes**, and a large number of **points** plotted against those axes, indicating associations between life expectancy and GDP per capita for each point. The points are **coloured** according to the continent on which the datapoint was measured, and these colours are indicated in a **legend**. By quick visual inspection, we can see that the blue points (Europe) are associated with high life expectanct, and red points (Africa) a low life expectancy. So, this representation has value in summarising our large `gapminder` table, and allowing us to infer patterns in the data.This convenience function (the default scatterplot) was quick and easy to draw, and informatics, but it's not very easy to modify or customise the representation. What if we'd like to build a plot *like* this, but different?We're going to consider the plot elements (in **bold** above) in a little more detail, to break them down a bit in terms of the *grammar of graphics*, so we can build highly customisable plots to represent your data. 2a. **aesthetics**Firstly, we note that every *observation* in the data is a single **point**. Each point has an *aesthetic* that determines how it is rendered in the plot. The *aesthetics* we can control are:* *x*- and *y*- co-ordinate* shape* size* colourEach aesthetic can either be constant, or be mapped to a variable. Most obviously, the *x*- and *y*- co-ordinates are *mapped* to the variables `lifeExp` and `gdpPercap`, respectively. The *colour* aesthetic is mapped to the `continent` variable. These two mappings differ, because of the kind of variable being mapped:* `lifeExp` and `gdpPercap` are continuous variables* `continent` is a discrete variableWe could construct many different plots from exactly the same data, just by using different aesthetics For an arbitrary set of points, we could describe *x*-, *y*- and *colour* aesthetics as in the table below:![scatterplot aesthetics](images/scatterplot_data_table.png)This should remind you of a `DataFrame`, and with good reason: the mapping from data to an effective visualisation is in large part one of making an appropriate choice of aesthetics*Aesthetics* essentially create a new dataset that corresponds to your original data, but that contains aesthetic information. Building a plot involves creating *mappings* from your data to produce values in this *aesthetic* dataset. 2b. Geometric representations: geomsIndependently of the aesthetics described above, we can consider different ways of drawing a dataset:* If we draw data with **points**, we have a **scatterplot*** If we draw data with **lines**, we have a **line plot*** If we draw data with **bars**, we have a **bar chart**and so on.These *types* of representation are referred to as *geometric representations*, or `geom`s, for short.![Types of `geom`](images/geom_types.png)* Not all `geom`s make sense for a given dataset, even though they may be "grammatical".* It is possible to combine multiple `geom`s to produce new graphs 2c. layersIn the *grammar of graphics*, visualisations are built up as **layers**. Each layer has *at least* two parts:1. *data* and its *aesthetic* mapping2. a *geometric representation* (`geom`)Additionally, each layer may also have optional statistical transformations or adjustments (e.g. scalings, or fitted curves).The data in a layer may be provided in the layer itself or, more commonly, be provided in a default dataset that is in the base layer for the visualisation. 2d. Recreating the scatterplot with layersOne approach to creating a plot with layers is first to create a *base layer* that contains the default dataset and defines *aesthetics* for it, and then to add additional layers that provide *geometric representations*. We will do this now to reproduce the scatterplot you drew above. Base layerWe create a base layer as an instance of the `ggplot()` Class, and assign it to a variable `p`. The `ggplot()` is created with `gapminder` as the dataset, and with *aesthetics* that map the `lifeExp` and `gdpPercap` continuous variables to the *x*- and *y*- co-ordinates, respectively.```pythonp = ggplot(gapminder, aes(x='lifeExp', y='gdpPercap', color='continent'))```NOTE: This does not have quite the same syntax as its R equivalent.<jupyter_code># Create the plot base layer<jupyter_output><empty_output><jupyter_text>First `geom` layerYou will notice that, although you have created a base layer, there is no plot visible. This is in part because no geometric representation has been defined and, without this representation of the data, there is nothing to be drawn.To produce a figure, we *add* a `geom` to the base layer in `p`:```pythonp + geom_point()```NOTE: This does not have quite the same syntax or behaviour as its R equivalent. In particular, it is not possible (in ggplot v0.10.4) to define a new aesthetic in this layer.<jupyter_code># Use a geom_point() to produce a scatterplot<jupyter_output><empty_output><jupyter_text>This only temporarily changes the plot, as the base plot in `p` remains unchanged. To make a 'permanent' change to a base plot, and to show it in the notebook, we can use the code:```pythonp = ggplot(gapminder, aes(x='lifeExp', y='gdpPercap', color='continent'))p += geom_point()p```<jupyter_code># Add the geom_point() to the base plot in p, and render the whole scatterplot<jupyter_output><empty_output><jupyter_text>This now matches our earlier effort, and we can continue to add new `geom`s to the plot to build up more information:```pythonp = ggplot(gapminder, aes(x='lifeExp', y='gdpPercap', color='continent'))p += geom_point()p += xlab("life expectancy")p += ylab("GDP per capita")p += title("gapminder example data")p```NOTE: This does not have the same syntax or behaviour as its R equivalent. In particular, it is not possible (in ggplot v0.10.4) to render a plot then continue to add layers to the plot.<jupyter_code># Add axis labels and a title to the plot<jupyter_output><empty_output><jupyter_text>***CHALLENGE:*** Can you modify the example above so that the figure visualises how life expectancy has changed over time, as a scatterplot?<jupyter_code># Answer challenge in this cell<jupyter_output><empty_output><jupyter_text>3. More LayersAn advantage to using the *grammar of graphics* approach to producing visualisations is that it is possible to *stack* **layers** of `geom`s to better visualise data. In this subsection we will see how to stack different types of *geometric representation* to obtain bespoke visualisations. 3a. Stacking data representations The challenge answer above doesn't look very good as a data representation. While we can get a sense of the variation between continents, it is difficult to follow data points to track a single `country` over the data period. To see this information more clearly, we can use a `geom_line()` object to render a *line plot*:```pythonp = ggplot(gapminder, aes(x='year', y='lifeExp', color='continent'))p + geom_line()```<jupyter_code># Render a line plot with geom_line()<jupyter_output><empty_output><jupyter_text>This hasn't worked very well. The data have not been grouped by country but instead by continent. The lines go vertically between all points for a single continent in each year, before skipping to the next year.To connect datapoints by country, we need to specify `country` as the grouping variable when we define the base *aesthetic*:```pythonp = ggplot(gapminder, aes(x='year', y='lifeExp', color='continent', by='country'))p + geom_line()```<jupyter_code># Group datapoints by country, and render a line plot<jupyter_output><empty_output><jupyter_text>This looks much better, and now we can *overlay* or *stack* another `geom` - this time to place individual data points, with `geom_point()`:```pythonp = ggplot(gapminder, aes(x='year', y='lifeExp', color='continent', by='country'))p + geom_line() + geom_point()```<jupyter_code># Stack a geom_point() on a geom_line() for the gapminder data<jupyter_output><empty_output><jupyter_text>To an extent (though less so than for the equivalent objects in `R`), we can modify aesthetics independently in separate layers. For example, we can set the colours of all plotted datapoints in the `geom_point()` layer to be black:```pythonp = ggplot(gapminder, aes(x='year', y='lifeExp', color='continent', by='country'))p + geom_line() + geom_point(color='black') ```<jupyter_code># Redraw the stacked plot, with black datapoints<jupyter_output><empty_output><jupyter_text>3b. Data transformationTransformations, scalings and statistical summaries of data are another kind of *layer*, and can be stacked just like `geom`s. In our earlier plot of 'GDP per capita' against 'life expectancy', it could be difficult to distinguish GDP on the *y*-axis:```pythonp = ggplot(gapminder, aes(x='lifeExp', y='gdpPercap', color='continent'))p + geom_point()```<jupyter_code># Plot GDP against life expectancy<jupyter_output><empty_output><jupyter_text>We might consider using a log transformation of the GDP data. We could do this by creating a new column in our data table, filling it with transformed data, and then plotting this new column in place of `gdpPercap`. However, we can apply a scaling directly to our plot with the `scale_y_log()` transformation.```pythonp = ggplot(gapminder, aes(x='lifeExp', y='gdpPercap', color='continent'))p + geom_point() + scale_y_log()```(there is a counterpart `scale_x_log()` transformation). This essentially applies a transformation/mapping 'on-the-fly' to the data, as it is plotted.<jupyter_code># Replot the data with a log y-axis<jupyter_output><empty_output><jupyter_text>Although it may seem abstract, the action of recolouring data is also a transformation (but onto a domain of colours, rather than numerical values). Colour transformations/scalings are applied in the same way. The code below will redraw the plot with one of the `colorbrewer` palettes.* `colorbrewer`: [interactive examples](http://colorbrewer2.org/)```pythonp = ggplot(gapminder, aes(x='lifeExp', y='gdpPercap', color='continent'))p + geom_point() + scale_y_log() + scale_color_brewer()```Many options are available for `colorbrewer` palettes, and some experimentation may be useful when exploring your own data:```pythonscale_color_brewer(type='qual')scale_color_brewer(palette=3)scale_color_brewer(palette=2, type='div')```<jupyter_code># Replot the log-transformed data with a Brewer colour scale<jupyter_output><empty_output><jupyter_text>3b. Statistical data summaries Statistical analyses also transform data, usually by performing a data summary such as smoothing or binning. In the Python `ggplot` module, two layers are available. The first of these, `stat_smooth`, attempts to fit a smooth curve to your data (according to how it is grouped), and provides a visual estimate of confidence interval on the smoothing:```pythonp = ggplot(gapminder, aes(x='year', y='lifeExp', color='continent'))p + geom_point() + stat_smooth()```***QUESTION:*** How has life expectancy changed for each continent?NOTE: These layers do not have the same syntax or behaviour as their R equivalents. They are much less flexible.<jupyter_code># Plot smoothed curves by continent for life expectancy over time<jupyter_output><empty_output><jupyter_text>The second statistical summary layer, `stat_density()` plots a kernel density estimate (KDE) of frequency for a given *x*-variable. The code below plots the frequency of a given life expectancy grouped by continent, for all countries and all years measured:```pythonp = ggplot(gapminder, aes(x='lifeExp', color='continent'))p + stat_density()```***QUESTION:*** How does life expectancy vary by continent?<jupyter_code># Plot smoothed curves by continent for life expectancy over time<jupyter_output><empty_output><jupyter_text>4. Multi-panel figuresSo far we have shown all our data in a single plot. We have used different colours and geometric representations to make our figures clearer, but sometimes multi-panel figures can give much clearer overviews of data, generally as *small multiple* plots.*Small multiple* plots (aka *trellis*, *lattice*, *grid* or *facet* plots) are similar graphs using the same scales and axes, but representing different subsets of a complete dataset, allowing them to be easily compared visually. These plots are fairly easy to represent using `ggplot`, as in this module the `facet_wrap()` or `facet_grid()` object is just another layer of the plot. A single plot that groups our `gapminder` data by country, colouring them by continent, is messy and hard to read:```pythonp = ggplot(gapminder, aes(x='year', y='lifeExp', color='continent', by='country'))p + geom_line()```***QUESTION:*** Can you tell how life expectancy has changed on each continent?<jupyter_code># Group datapoints by country, and render a line plot<jupyter_output><empty_output><jupyter_text>Using a small multiple plot with the `facet_wrap()` layer, we can split plots out on the basis of a named variable. Here we will split by continent:```pythonp = ggplot(gapminder, aes(x='year', y='lifeExp', color='continent', by='country'))p + geom_line() + facet_wrap('continent')```***QUESTION:*** How has life expectancy changed on each continent?<jupyter_code># Group datapoints by country, and render a line plot<jupyter_output><empty_output><jupyter_text>This breaks up the plot into five plots, each with the same *x*- and *y*-axes and scales. This allows for a direct visual comparison, where we can readily see the trends and variation on a continent-wise basis. Similarly, we can split our larger scatterplot into twelve subplots by year, and see at a glance how the GDP *per capita* and life expectancy has varied over time, with a common log scale on the *y*- axis for each plot.```pythonp = ggplot(gapminder, aes(x='lifeExp', y='gdpPercap', color='continent'))p + geom_point() + scale_y_log() + facet_wrap('year')```***QUESTION:*** Can you tell how the relationship between GDP per capita and life expectancy changed in each continent, since 1980?<jupyter_code># Create a facet plot by year of gdpPercap vs lifeExp, coloured by continent<jupyter_output><empty_output><jupyter_text>It is possible to create facets that are split on the basis of more than one variable. For example, we can plot data separately for each continent and year since 1980, with the code below:```pythongapminder_recent = gapminder.loc[gapminder['year'] > 1980,]p = ggplot(gapminder_recent, aes(x='lifeExp', y='gdpPercap', color='continent'))p + geom_point() + scale_y_log() + facet_wrap('year', 'continent', ncol=5)```***QUESTION:*** How has the relationship between GDP per capita and life expectancy changed in each continent, since 1980?<jupyter_code># Create a facet plot by year and continent, of gdpPercap vs lifeExp since 1980, coloured by continent<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08972219495435359, 'fraction_numerical': 0.002601354667713753, 'mean_word_length': 5.923207611281006, 'pattern_counts': {'":': 0, '<': 96, '<?xml version=': 0, '>': 97, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '415670', 'license': 'MIT', 'n_tokens_mistral': 5843, 'n_tokens_neox': 5484, 'n_words': 2664, 'path': 'exercises/grammar_of_graphics/grammar_of_graphics.ipynb', 'repo_name': 'widdowquinn/Teaching-Data-Visualisation', 'size': '20400'}