<jupyter_start><jupyter_text>Introduction to pandasBased on tutorials by <NAME> https://github.com/dshahid380/Data-analysis-with-pandas and exercises by <NAME> https://github.com/guipsamora/pandas_exercises  What is pandas ? ***pandas*** is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.Python has long been great for data munging and preparation, but less so for data analysis and modeling. pandas helps fill this gap, enabling you to carry out your entire data analysis workflow in Python without having to switch to a more domain specific language like **R**.[pandas website] External Information* Pandas website:  https://pandas.pydata.org/* Pandas user guide: http://pandas.pydata.org/pandas-docs/stable/user_guide/index.html* Pandas API documentation: http://pandas.pydata.org/pandas-docs/stable/reference/index.html* VERY USEFULL: Pandas Cheat Sheet: https://github.com/pandas-dev/pandas/blob/master/doc/cheatsheet/Pandas_Cheat_Sheet.pdf  Getting startedTo use pandas you need to first import the pandas module in your program<jupyter_code>import pandas as pd #naming convention for pandas is pd<jupyter_output><empty_output><jupyter_text>Reading CSV and Excel sheets:  d=pd.read_csv("path"): * pd.read_csv() is the function to read the CSV(Comma separated values) file from your computer. * In the function you have to pass "path" of the CSV file under quote. * Store the dataframe in any variable,here i stored it in variable "d". * read_csv() function makes the CSV file into dataframe so that you can access it just like a disctionary. d=pd.read_excel("path") :  * It is same as the read_csv() but it reads  excel sheet or file. Importing Weather data<jupyter_code>d=pd.read_csv('../DATA/weather.csv') 
# returning dataframe object
d
#printing dataframe d in table format<jupyter_output><empty_output><jupyter_text>First interaction with the data  How many rows are  in my DataFrame ?<jupyter_code>print(len(d))
# there are 96453 rows<jupyter_output>96453<jupyter_text>Getting the first n-rows of dataset  * To see the first five rows call head() function with dataframe object.       for example-         d.head()    * If you want to view first n-rows        d.head(n)<jupyter_code>d.head() # first five rows
d.head(9) # to print first n=9 rows<jupyter_output><empty_output><jupyter_text>Getting the last n-rows of dataset  * Call the tail() function       d.tail()  It will show only last five rows of dataframe d.   * If you want to see last n-rows -       d.tail(n)<jupyter_code>d.tail() #last five rows
d.tail(9) # last 9 rows<jupyter_output><empty_output><jupyter_text>Simple Slicing in a DataFrame * Slicing works very similar to Numpy  * Suppose you want to get 10 rows of the dataframe ranging from row 20 to 30.     d[20:31]<jupyter_code>d[20:31]
#slicing with step size
d[20:30:2]
#inverse index
d[-15:-10]<jupyter_output><empty_output><jupyter_text>BUT: accesing a single element is different!<jupyter_code>d[3]
#use iloc instead
d.iloc[3]
#check with head
d.head()<jupyter_output><empty_output><jupyter_text>Accesing the particular column of dataframe : * You can access the particular column of the dataframe just my mentioning its name under quote with dataframe d.    for example you want to access Humidity column e.i,          d['Humidity']<jupyter_code>d['Humidity'].head(10) 
#Here i have applied head() function just to see only first 10 values because humidity 
#column has too many values<jupyter_output><empty_output><jupyter_text>Finding min, max and average of a column:  * min() : To find minimum of a column  * max() : To find maximum of a column   * mean() : To find average of a column   Find minimum and maximum of Humidity column-<jupyter_code>d['Humidity'].min() 
#Printing minimum value of Humidity
d['Humidity'].max()
#Print maximum value Humidity
d['Humidity'].mean()
#Print average value Humidty<jupyter_output><empty_output><jupyter_text>Conditional statements  * d["your column"]["your condition"]     It will return all the values in which your condition holds true. Examples :   * Find temp when Humidity is minimum       d['Temperature(C)'][d['Humidity']==d['Humidity'].min() ]       * Find temp when Humidity is maximum        d['Temperature(C)'][d['Humidity']==d['Humidity'].max() ]<jupyter_code>d['Temperature (C)'][d['Humidity']==d['Humidity'].min()]
 d['Temperature (C)'][d['Humidity']==d['Humidity'].max()]<jupyter_output><empty_output><jupyter_text>In the same way you can apply various condition and analyse it hidden features  Replacing NaN with specific value * Here i am replacing all NaN value with 0.<jupyter_code>d.fillna(0,inplace=True)<jupyter_output><empty_output><jupyter_text>Visualization with PandasPandas also has build in visualization and plotting methods<jupyter_code># example simple plot
import matplotlib.pyplot as plt
%matplotlib inline
d['Temperature (C)'].plot()

# histogram
d['Temperature (C)'].plot.hist()

# boxplot over all numerical variables
plt.figure(figsize=(20,10)) #pandas uses plt, so we can use plt methods to manipulate the output
d.boxplot()

#scatter plot
d.plot.scatter('Temperature (C)','Humidity',figsize=(10,5))
#finaly a realy helpfull tool to get a first look at data - takes some time to compute on full data
output=pd.plotting.scatter_matrix(d[1:100],figsize=(20,15))<jupyter_output>/home/keuper/anaconda3/lib/python3.7/site-packages/pandas/plotting/_misc.py:108: UserWarning: Attempting to set identical left==right results
in singular transformations; automatically expanding.
left=0.0, right=0.0
  ax.set_xlim(boundaries_list[j])
/home/keuper/anaconda3/lib/python3.7/site-packages/pandas/plotting/_misc.py:109: UserWarning: Attempting to set identical bottom==top results
in singular transformations; automatically expanding.
bottom=0.0, top=0.0
  ax.set_ylim(boundaries_list[i])
/home/keuper/anaconda3/lib/python3.7/site-packages/pandas/plotting/_misc.py:100: UserWarning: Attempting to set identical left==right results
in singular transformations; automatically expanding.
left=0.0, right=0.0
  ax.set_xlim(boundaries_list[i])
{'fraction_non_alphanumeric': 0.10426154847104749, 'fraction_numerical': 0.01512687052700065, 'mean_word_length': 5.472631578947368, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '416096', 'license': 'MIT', 'n_tokens_mistral': 2018, 'n_tokens_neox': 1926, 'n_words': 636, 'path': 'Day_1_Intro/Full_Python_Tutorial_(optinal)/08_Pandas-Intro.ipynb', 'repo_name': 'keuperj/ki-sommer', 'size': '6154'}