<jupyter_start><jupyter_text>Introduction to Jupyter notebooks and Pandas  1. Opening and Navigating the IPython NotebookWe will start today with the interactive environment that we will be using in the tutorial: the [Jupyter Notebook](http://jupyter.org).We will walk through the following steps together:1. (If you don't yet have anaconda) Download [miniconda](https://conda.io/miniconda.html) (be sure to get Version 3.7) and install it on your system (hopefully you have done this before coming to class)2. Use the ``conda`` command-line tool to create a virtual environment with all the python tools we need:    Update ``conda``'s listing of packages for your system:    >```    conda update conda    conda env create -f binder/environment.yml    conda activate dssg2019    jupyter labextension install @jupyter-widgets/jupyterlab-manager \                                 @pyviz/jupyterlab_pyviz \                                 jupyter-matplotlib \                                 jupyter-leaflet    ```   3. Use git to clone our repo (the "2019" tag):     >```    git clone --branch 2019 https://github.com/uwescience/dssg-pandas-sql-tutorial    ```   4. Navigate to the directory containing the tutorial material    >```    cd dssg-pandas-sql-tutorial    ```    You should see a number of files in the directory (0-Jupyter-notebooks.ipynb, 1-intro-geopandas.ipynb...):    >```    ls    ```5. Type ``jupyter notebook`` in the terminal to start the notebook    >```    jupyter lab    ```    If everything has worked correctly, it should automatically launch your default browser   6. Click on ``0-Jupyter-numpy.ipynb`` to open the notebook containing the content for this lecture.With that, you're set up to use the Jupyter notebook on your own computer!If it didn't work, go ahead and open an interactive notebook on mybinder.org (https://mybinder.org/ ):* https://mybinder.org/v2/gh/uwescience/dssg-pandas-sql-tutorial/2019?urlpath=lab  About Jupyter- Combines the features of an IPython terminal with a fancy web interface- Every notebook has one python instance running with it- Also supports markdown and html embedding- Easy to insert $\LaTeX$ formulas- Be ware of VIM-style keyboard shortcuts  Python refresher- Components with the same capabilities are of the same *type*.   - For example, the numbers 2 and 200 are both integers.  - This is called "duck typing" (if it looks like a duck, quacks like a duck...)  - A type is defined recursively. Some examples.  - A list is a collection of objects that can be indexed by position.  - A list of integers contains an integer at each position.  - A type has a set of supported operations. For example:  - Integers can be added  - Strings can be concatented  - A table can find the name of its columns    - In python, members (components and operations) are indicated by a '.'  - If `a` is a list, the `a.append(1)` adds `1` to the list.<jupyter_code>a = 1
# Python lists can store data of different types at the same time
a_list = [1, 'a', [1,2]]
# What if we forgot what functions a list has?
a_list.pop?
a_list.append(2)
1
a_list
1
# What are the full set of functions available for our list
a_list.
# Count how many times the integer '1' occurs
a_list.count(1)<jupyter_output><empty_output><jupyter_text>The IPython kernel/client model<jupyter_code>%connect_info<jupyter_output>{
  "shell_port": 57896,
  "iopub_port": 57897,
  "stdin_port": 57898,
  "control_port": 57899,
  "hb_port": 57900,
  "ip": "127.0.0.1",
  "key": "d35b7ce7-38529646d0d2edcc15288ed9",
  "transport": "tcp",
  "signature_scheme": "hmac-sha256",
  "kernel_name": ""
}

Paste the above JSON into a file, and connect with:
    $> jupyter <app> --existing <file>
or, if you are local, you can connect with just:
    $> jupyter <app> --existing kernel-71ec3129-90c4-4637-af36-85e0771b93d4.json
or even just:
    $> jupyter <app> --existing
if this is the most recent Jupyter kernel you have started.<jupyter_text>We can connect automatically a Qt Console to the currently running kernel with the %qtconsole magic, or by typing ipython console --existing  in any terminal:<jupyter_code>%qtconsole magic<jupyter_output><empty_output><jupyter_text>Python's Data Science EcosystemThere are also many often-used third-party modules that are core tools for doing data science with Python.Some of the most important ones are: [``numpy``](http://numpy.org/): Numerical PythonNumpy is short for "Numerical Python", and contains tools for efficient manipulation of arrays of data.If you have used other computational tools like IDL or MatLab, Numpy should feel very familiar. [``scipy``](http://scipy.org/): Scientific PythonScipy is short for "Scientific Python", and contains a wide range of functionality for accomplishing common scientific tasks, such as optimization/minimization, numerical integration, interpolation, and much more. [``pandas``](http://pandas.pydata.org/): Labeled Data Manipulation in PythonPandas is short for "Panel Data", and contains tools for doing more advanced manipulation of labeled data in Python, in particular with a columnar data structure called a *Data Frame*.If you've used the [R](http://rstats.org) statistical language , much of the functionality in Pandas should feel very familiar. [``matplotlib``](http://matplotlib.org): Visualization in PythonMatplotlib started out as a Matlab plotting clone in Python, and has grown from there in the 15 years since its creation. It is the most popular data visualization tool currently in the Python data world (though other recent packages are starting to encroach on its monopoly).<jupyter_code>import pandas as pd

pd.read_csv?
df = pd.read_csv('data/Places_Full.csv')
df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08627931396569828, 'fraction_numerical': 0.021701085054252712, 'mean_word_length': 4.062001771479185, 'pattern_counts': {'":': 10, '<': 20, '<?xml version=': 0, '>': 28, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '368327', 'license': 'Apache-2.0', 'n_tokens_mistral': 1769, 'n_tokens_neox': 1658, 'n_words': 710, 'path': '1-Intro-Jupyter-notebooks.ipynb', 'repo_name': 'uwescience/dssg-pandas-sql-tutorial', 'size': '5714'}