<jupyter_start><jupyter_text>In this demo, we will be exploring how world developmental indicators are related to a country’s early effort in COVID-19 response.          Visualizations of dataframes beyond simple tables To enable Lux, simply add `import lux` along with your Pandas import statement.<jupyter_code>import lux
import pandas as pd<jupyter_output><empty_output><jupyter_text>Lux preserves the Pandas dataframe semantics -- which means that you can apply any command from Pandas's API to the dataframes in Lux and expect the same behavior. For example, we can load the [Happy Planet Index (HPI)](http://happyplanetindex.org/) dataset via standard Pandas `read_csv` command.<jupyter_code>df = pd.read_csv("https://github.com/lux-org/lux-datasets/blob/master/data/hpi_cleaned.csv?raw=True")<jupyter_output><empty_output><jupyter_text>We can quickly get an overview of the dataframe, simply by print out the dataframe `df`.<jupyter_code>df<jupyter_output><empty_output><jupyter_text>From the Pandas table view, we see that the dataframe contains country-level data on sustainability and well-being.By clicking on the Toggle button, you can now explore the data visually through Lux, you should see several tabs of visualizations recommended to you that includes scatterplots, bar charts, and maps. In Lux, we recommend visualizations that may be relevant or interesting to you across different [actions](https://lux-api.readthedocs.io/en/latest/source/getting_started/overview.htmlvisualizing-dataframes-with-recommendations), which are displayed as different tabs. By inspecting the `Correlation` tab, we learn that there is a negative correlation between `AvrgLifeExpectancy` and `Inequality`. In other words, countries with higher levels of inequality also have a lower average life expectancy. We can also look at other tabs, which show the Distribution of quantitative attributes and the Occurrence of categorical attributes.         Steering analysis with intentLet's say that we want to investigate whether any country-level characteristics explain the observed negative correlation between inequality and life expectancy. Beyond the basic recommendations, you can further specify your analysis *intent*, i.e., the data attributes and values that you are interested in visualizing.  We can do this by specifying our analysis intent to Lux via `df.intent`:<jupyter_code>df.intent = ["Inequality","AvrgLifeExpectancy"]<jupyter_output><empty_output><jupyter_text>Upon printing the dataframe again, Lux leverages the analysis intent to steer the recommendations towards what the user might be interested in.<jupyter_code>df<jupyter_output><empty_output><jupyter_text>By looking at the colored scatterplots in the `Enhance` tab, we find that most G10 industrialized countries are on the upper left quadrant on the scatterplot (low inequality, high life expectancy). In the breakdown by Region, we observe that countries in Sub-Saharan Africa (yellow points) tend to be on the bottom right, with lower life expectancy and higher inequality. ℹ️ Check out [this tutorial](https://lux-api.readthedocs.io/en/latest/source/guide/intent.html) to learn more about how to specify intent in Lux.         Data Manipulation + Vis without changing a line of your Pandas code Lux is designed to be tightly integrated with Pandas and can be used as-is, without modifying your existing Pandas code.  This means that you can seamlessly transition from doing data cleaning and transformation to visualizing your dataframes with no effort. The goal of this section is largely to demonstrate how Lux can help you visualize your dataframe in a realistic scenario that involves lots of complex data cleaning and transformation Continuing our analysis, we are now interested in how these country-level metrics related to a country's early COVID intervention strategy and response, based on the [COVID pandemic policy dataset](https://ourworldindata.org/grapher/covid-stringency-index) dataset.<jupyter_code>covid = pd.read_csv("https://github.com/lux-org/lux-datasets/blob/master/data/covid_cleaned.csv?raw=True")<jupyter_output><empty_output><jupyter_text>We now join the countries dataframe `df` with the filtered COVID dataframe:<jupyter_code>df = covid.merge(df,left_on=["Entity","Code"],right_on=["Country","cca3"])
df<jupyter_output><empty_output><jupyter_text>🤔 Some interesting findings With the modified dataframe, we revisit the negative correlation that we observed previously by setting the intent as average life expectancy and inequality again. The result is similar to what we saw before, with one visualization showing the breakdown by `stringency_level`.<jupyter_code>df.intent = ["Inequality","AvrgLifeExpectancy"]
df<jupyter_output><empty_output><jupyter_text>We see a strong separation showing how stricter countries (blue) corresponded to countries with higher life expectancy and lower levels of inequality. This visualization indicates that these countries could possibly have a more well-developed public health infrastructure that promoted the early pandemic response. However, we observe three outliers that seem to defy this trend.   When we filter to these dataframe records, we find that these countries correspond to [Afghanistan](https://www.who.int/news-room/feature-stories/detail/afghanistan-who-mission-reviews-covid-19-response), [Pakistan](https://www.who.int/news-room/feature-stories/detail/covid-19-in-pakistan-who-fighting-tirelessly-against-the-odds), and [Rwanda](https://www.npr.org/sections/goatsandsoda/2020/07/15/889802561/a-covid-19-success-story-in-rwanda-free-testing-robot-caregivers)—countries that were praised for their early pandemic response despite limited resources.<jupyter_code>df[(df["Inequality"]>0.35)&(df["stringency_level"]=="High")]<jupyter_output><empty_output><jupyter_text>Exporting visualization insight to edit and share To download this visualization insight and share with others, we can click on the visualization in the Lux view above and the button.<jupyter_code>df<jupyter_output><empty_output><jupyter_text>This exports the visualization from the widget to a `Vis` object. We can access the exported `Vis` object via the `exported` property and print it as code.<jupyter_code>df.exported
print(df.exported[0].to_code("altair"))<jupyter_output><empty_output><jupyter_text>We can copy-and-paste the output Altair code into a separate cell. Then let's tweak the plotting code a bit before sharing this insight with our colleagues.<jupyter_code>import altair as alt

c = "#e7298a"
chart = alt.Chart(df,title="Check out this cool insight!").mark_circle().encode(
    x=alt.X('Inequality',scale=alt.Scale(domain=(0.04, 0.51)),type='quantitative', axis=alt.Axis(title='Inequality')),
    y=alt.Y('AvrgLifeExpectancy',scale=alt.Scale(domain=(48.9, 83.6)),type='quantitative', axis=alt.Axis(title='AvrgLifeExpectancy'))
)
highlight = df[(df["Inequality"]>0.35)&(df["stringency_level"]=="High")]

hchart = alt.Chart(highlight).mark_point(color=c,size=50,shape="cross").encode(
    x=alt.X('Inequality',scale=alt.Scale(domain=(0.04, 0.51)),type='quantitative', axis=alt.Axis(title='Inequality')),
    y=alt.Y('AvrgLifeExpectancy',scale=alt.Scale(domain=(48.9, 83.6)),type='quantitative', axis=alt.Axis(title='AvrgLifeExpectancy')),
)

text = alt.Chart(highlight).mark_text(color=c,dx=-35,dy=0,fontWeight=800).encode(
    x=alt.X('Inequality',scale=alt.Scale(domain=(0.04, 0.51)),type='quantitative', axis=alt.Axis(title='Inequality')),
    y=alt.Y('AvrgLifeExpectancy',scale=alt.Scale(domain=(48.9, 83.6)),type='quantitative', axis=alt.Axis(title='AvrgLifeExpectancy')),
    text=alt.Text('Country')
)

chart = chart.encode(color=alt.Color('stringency_level',type='nominal'))
chart = chart.properties(width=160,height=150)

(chart + hchart + text).configure_title(color=c)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09597759674134419, 'fraction_numerical': 0.011328920570264765, 'mean_word_length': 6.44739336492891, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 51, 'https://': 8, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '405153', 'license': 'Apache-2.0', 'n_tokens_mistral': 2297, 'n_tokens_neox': 2178, 'n_words': 757, 'path': 'demo/hpi_covid_demo.ipynb', 'repo_name': 'fenago/lux-binder', 'size': '7856'}