<jupyter_start><jupyter_text>Importing yout data To import your data you should use pandas. It is also possible to load one of the classics geostatistical datasets, such as Walker Lake and Jura, form our datasets module.![pandas](figs/pandas1.png)pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. The two primary data structures of pandas, ``Series`` (1-dimensional) and ``DataFrame`` (2-dimensional), handle the vast majority of typical use cases in finance, statistics, social science, and many areas of engineering. Pandas is built on top of NumPy and is intended to integrate well within a scientific computing environment with many other 3rd party libraries.To import Pandas:<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>For additional information on Pandas, see the [documentation](https://pandas.pydata.org/pandas-docs/stable/)  Getting Data In/Out Pandas is packed with a set of IO functions that returns a pandas object. Reader functions are accessed like ``pd.read_fileformat()`` while writer functions are accessed like ``DataFrame.to_fileformat()``. Format Type	|  Data Description       |  Reader	          |Writer         |------------|-------------------------|-------------------|---------------|text	    |    CSV	              |   read_csv	      |  to_csv       |text	    |    JSON	              |   read_json	      |  to_json      |text	    |    HTML	              |   read_html	      |  to_html      | text	    |    Local clipboard	  |   read_clipboard  |  to_clipboard |binary	    |    MS Excel	          |   read_excel	  |  to_excel     |binary	    |    HDF5 Format	      |   read_hdf	      |  to_hdf       |binary	    |    Feather Format       |   read_feather	  |  to_feather   | binary	    |    Parquet Format       |   read_parquet	  |  to_parquet   |binary	    |    Msgpack	          |   read_msgpack	  |  to_msgpack   |binary	    |    Stata	              |   read_stata	  |  to_stata     |binary	    |    SAS	              |   read_sas        |	              |binary	    |    Python Pickle Format |   read_pickle	  |  to_pickle    |SQL	        |    SQL	              |   read_sql	      |  to_sql       |SQL	        |    Google Big Query	  |   read_gbq	      |  to_gbq       | Importing a ``.csv`` dataset:<jupyter_code>path = "data/"
file = "walker.csv"

data = pd.read_csv(path + file, sep=",", na_values=-999)<jupyter_output><empty_output><jupyter_text>Viewing Data ``df.head(n)`` shows the first n rows (if nothing is passed n=5) of the DataFrame while ``df.tails(n)`` shows the last n lines.<jupyter_code>data.head()<jupyter_output><empty_output><jupyter_text>Filtering Accesing variables collumns:<jupyter_code>data[["U", "V"]].head()<jupyter_output><empty_output><jupyter_text>Accessing DataFrame where variable V is bigger than 640:<jupyter_code>df_filter = data['V'] > 640

data[df_filter].head()<jupyter_output><empty_output><jupyter_text>Accessing DataFrame where variable T is equal to 1:<jupyter_code>df_filter = data["T"] == 1

data[df_filter].head()<jupyter_output><empty_output><jupyter_text>``describe()`` shows a quick statistic summary of your data:<jupyter_code>data[["U", "V"]].describe()<jupyter_output><empty_output><jupyter_text>``corr()`` shows the correlation matrix:<jupyter_code>data[["U", "V"]].corr()<jupyter_output><empty_output><jupyter_text>Selection Selecting a single column, which yields a ``Series``:<jupyter_code>V_variable = data.V

#which is the same as:
V_variable = data["V"]
type(V_variable)<jupyter_output><empty_output><jupyter_text>To select values:[df.at](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.at.html) can only access a single value at a time.[df.loc](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.loc.html) can select multiple rows and/or columns.<jupyter_code>data.loc[[3, 4, 5]] #indexes 3, 4 and 5 for all columns
data.at[4, "V"] #index 4 for variable V<jupyter_output><empty_output><jupyter_text>Setting First, let`s create a capped U variable:<jupyter_code>import numpy as np
U_cap = np.where(data["U"] > 2535, 2535, data["U"])<jupyter_output><empty_output><jupyter_text>Now let`s create a new collumn data U capped in the DataFrame<jupyter_code>data["U capped"] = U_cap
data.tail()<jupyter_output><empty_output><jupyter_text>``df.loc`` and ``df.at`` can be use to set values too<jupyter_code>data.at[4, "V"] = 0
data.at[4, "V"]<jupyter_output><empty_output><jupyter_text>Missing data To drop any rows that have missing data use ``df.dropna()``. This is specially usefull to filter an isotopic dataset.<jupyter_code>data.dropna().head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12033620336203361, 'fraction_numerical': 0.007380073800738007, 'mean_word_length': 3.5683520599250937, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 59, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '190689', 'license': 'MIT', 'n_tokens_mistral': 1632, 'n_tokens_neox': 1573, 'n_words': 428, 'path': 'docs/source/pandas.ipynb', 'repo_name': 'maneetzaveri/pyLPM', 'size': '4878'}