<jupyter_start><jupyter_text>Python libraries for data scienceYou've already seen that you can import modules to extend Python with a wide range of commands.Python has many powerful libraries, or packages (large modules) for data analysis and visualisation.Some of the most popular are listed below, along with various statistics from Github, the open source code-sharing platform where they are developed.The statistics are:  * Commits - individual updates to the code* Contributors - the community of people contributing to the code* Releases - numbered versions of the code* Watch - people watching to be notified of changes to the code* Star - people who have "favourited" the code* Fork - different copies made of the code either to update it or modify it into other software| Type           | Library Name | Commits | Contributors | Releases | Watch | Star  | Fork | Description                                                      ||----------------|--------------|---------|--------------|----------|-------|-------|------|------------------------------------------------------------------|| Data wrangling | NumPy        | 17453   | 605          | 133      | 376   | 6402  | 2567 | Operations on n-arrays and matrices                              ||                | SciPy        | 18640   | 576          | 96       | 283   | 4097  | 2174 | Linear algebra, optimization, integration, and statistics        ||                | pandas       | 16534   | 990          | 88       | 788   | 12855 | 4983 | Quick and easy data manipulation, aggregation, and visualisation || Visualisation  | matplotlib   | 24484   | 685          | 67       | 500   | 6748  | 3171 | Simple and powerful visualisations, but low level                ||                | seaborn      | 2010    | 77           | 13       | 196   | 4597  | 696  | Visualisation of statistical models including heat maps          ||                | plotly       | 2787    | 45           | 8        | 188   | 3020  | 732  | Web-based toolbox for building visualisations                    ||                | bokeh        | 16730   | 277          | 52       | 357   | 7190  | 1846 | Interactive visualisations                                       || Statistics     | StatsModels  | 9729    | 144          | 19       | 221   | 2597  | 1162 | Data exploration via statistical models, assertions and analysis || Data scraping  | scrapy       | 6594    | 276          | 81       | 1640  | 25683 | 6469 | Crawling web pages to retrieve structured data                   |*Source: [DATASCIENCE.COM](https://www.datascience.com/blog/top-python-libraries-for-data-science-in-2017) 2018*  Data analysis with pandasA fundamental package for data analysis is pandas, which roughly stands for 'Python Data Analysis'. Pandas is designed to provide data scientists working in Python with a set of powerful tools to load, transform, and process large-ish data sets. As a result, it has become something of a de facto standard for online tutorials and many of the lessons that you can find online will make use of pandas at some point. The official documentation is [here](http://pandas.pydata.org/pandas-docs/stable/).Let's use pandas to work with a table of data from a local CSV file. We'll use a small file (only 19 KB) so it's easy and quick to work with. The file shows an analysis of housing affordability by <NAME>. It was downloaded from City Futures Research Centre's CityData website: *[Sydney Dwelling Affordability Index](https://citydata.be.unsw.edu.au/layers/geonode%3Aaffordability_index)*.Visit that page now to learn a little about the dataset. First let's cheat by opening the file in Excel or other spreadsheet software to help understand what pandas is doing.You'll find it in the `data` folder, which is in the parent folder from where this notebook is located. **Open `affordability_index.csv` now.**It looks like this:![image.png](attachment:image.png) Pandas is not part of the base Python software, so you need to import it:<jupyter_code>import pandas<jupyter_output><empty_output><jupyter_text>We'll use pandas' `read_csv` method to open the file, read it and store it in a variable called `data`. All in one command!We have to tell Python where to find the file. Rather than an absolute path, we'll use a relative file path, which is relative to the notebook's location. To explain this path (below):* `..` means the parent directory.* Windows uses blackslash `\` to separate the components of a path. But backslash has special meaning in Python: it's an escape character, used to introduce special characters. So for example `\n` means newline. You have to use `\\` to indicate an actual backslash.<jupyter_code>my_data = pandas.read_csv('..\\data\\affordability_index.csv')<jupyter_output><empty_output><jupyter_text>What data type is `my_data`?<jupyter_code>print(type(my_data))<jupyter_output><empty_output><jupyter_text>It's a 'DataFrame'. That's pandas talk for a two-dimensional table of data. What can we find out about the data? Let's start by counting the rows and columns, which we do with a method called `shape`:<jupyter_code>print(my_data.shape)<jupyter_output><empty_output><jupyter_text>The output shows number of records and number of attributes. Have a look at the spreadsheet. Do those numbers look right?Let's see the column names.<jupyter_code>print(my_data.columns)<jupyter_output><empty_output><jupyter_text>All very well, but what do those columns mean? Fortunately Laurence has provided metadata when he published the dataset on CityData. Go back to [that page](https://citydata.be.unsw.edu.au/layers/geonode%3Aaffordability_index) and click the **Attributes** tab below the map to find out what the column names mean. Next let's read the first few rows with the `head` method. This is especially useful when dealing with a large dataset where you don't want to view the whole thing. By default it prints the first 5 rows of the dataframe.<jupyter_code>my_data.head()<jupyter_output><empty_output><jupyter_text>It looks like just the first 5 columns are numeric data. Let's check that with the `dtypes` property:<jupyter_code>my_data.dtypes<jupyter_output><empty_output><jupyter_text>Now we can use the `describe` method to run some statistics on the numeric columns:<jupyter_code>my_data.describe()<jupyter_output><empty_output><jupyter_text>Can you find the following from the statistics above?* The average affordability in 2005* The median affordability in 2015* The worst decline in affordability in this period* The best improvement in affordability We can also use Boolean expressions to find rows matching certain conditions.Which suburbs have more than 50% affordability in 2015?<jupyter_code>my_data[my_data["N_15_100k"] > 50]<jupyter_output><empty_output><jupyter_text>How many rows match that condition? Use `shape` in the cell below to find out. How would we find the most affordable suburbs in 2015?We saw that the `describe` method presents a range of statistics including maximum.Can you guess how to just return the maximum value from one column? Yes it's the `max` function:<jupyter_code>max(my_data["N_15_100k"])<jupyter_output><empty_output><jupyter_text>Let's make a variable to hold that maximum value:<jupyter_code>max_afford_2015 = max(my_data["N_15_100k"])<jupyter_output><empty_output><jupyter_text>Now we can list the rows that match that value:<jupyter_code>my_data[my_data["N_15_100k"] == max_afford_2015]<jupyter_output><empty_output><jupyter_text>Just the one! To get just the suburb name:<jupyter_code>my_data["Sub_indic"][my_data["N_15_100k"] == max_afford_2015]<jupyter_output><empty_output><jupyter_text>And to return the *value* 'Budgewoi' rather than the object (cell) containing that value, append the `values` property and find the first (*zero-th*) value:<jupyter_code>my_data["Sub_indic"][my_data["N_15_100k"] == max_afford_2015].values[0]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09292035398230089, 'fraction_numerical': 0.03350189633375474, 'mean_word_length': 3.4796149490373725, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 59, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '230028', 'license': 'MIT', 'n_tokens_mistral': 2392, 'n_tokens_neox': 2088, 'n_words': 888, 'path': '1. Data analysis with pandas.ipynb', 'repo_name': 'SimoneZLeao/Programmable-Cities-Block-1-Jupyter-notebooks', 'size': '7917'}