<jupyter_start><jupyter_text>![Astrofisica Computacional](../logo.png) --- 05. Introduction to `Pandas`<NAME> (<EMAIL>)---  About this notebookIn this notebook we present an introduction to the package `pandas`---<jupyter_code>%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib as ml
import sys

plt = ml.pyplot # matplotlib.pyplot as plt
# Defines the figure size for all the notebook
ml.rcParams['figure.figsize'] = (10.0, 5.0) 


rng = np.random.default_rng(413)  # Initialise our random number generator<jupyter_output><empty_output><jupyter_text>The `Dataframe` in `Pandas`<jupyter_code>data = [1, 2, 3, 4]
pd.DataFrame(data)
# Some data about planets
# mass in [Earth masses]
# orbital period in [Earth years]
# mean distance tu Sun in [au]

data = {'name': ['Earth', 'Mars', 'Jupiter'],
        'mass': [1., 0.107, 317.8],
        'period': [1., 1.88, 11.86],
        'distance': [1., 1.5, 5.2]}
df = pd.DataFrame(data)
df
type(df)<jupyter_output><empty_output><jupyter_text>Accesing Features  (`Series`)<jupyter_code>df['mass']
type(df['mass'])  
df['mass'] * 5.972e24
np.log(df['distance'])
df[['mass','distance']]<jupyter_output><empty_output><jupyter_text>Generating some series using random numbers<jupyter_code>s = pd.Series(rng.integers(0, 10, 10))
s<jupyter_output><empty_output><jupyter_text>We can sort the series. Note that the indices are kept the same as in the original series.<jupyter_code>sorted_s = s.sort_values()
sorted_s<jupyter_output><empty_output><jupyter_text>If we multiply series s by the sorted series, the product is obtained using the indices.<jupyter_code>s * s.sort_values()<jupyter_output><empty_output><jupyter_text>It is also possible to reset the indices after sorting the values:<jupyter_code>new_sorted_s = s.sort_values().reset_index(drop=True)
new_sorted_s
s * new_sorted_s<jupyter_output><empty_output><jupyter_text>Examining a `DataFrame`<jupyter_code>df
df.dtypes
df.columns
df.shape<jupyter_output><empty_output><jupyter_text>Looking into the data<jupyter_code>df.head(2) # First entries
df.tail(2) # Last entries
df.describe()<jupyter_output><empty_output><jupyter_text>Indexing and SlicingThere are different ways to index/slice data in pandas.<jupyter_code>df[0:1]<jupyter_output><empty_output><jupyter_text>There are two interesting methods:<jupyter_code>df.loc
df.iloc<jupyter_output><empty_output><jupyter_text>Using `.loc[]`This one treats the input as label or "row-name".<jupyter_code>df.loc[2] # Index of row
df['name'].loc[2]<jupyter_output><empty_output><jupyter_text>However, index \[-1\] is not defined as one may expect...<jupyter_code>df.loc[-1] # Note the ValueError<jupyter_output><empty_output><jupyter_text>We can accessing multiple rows and/or columns<jupyter_code>df.loc[[0, 2], ['name', 'period']]
df.loc[1:3, ['name']]<jupyter_output><empty_output><jupyter_text>Using `.iloc[]`<jupyter_code>df.iloc[1]  # Index of row
df.iloc[-1] # This time the index [-1] works as expected<jupyter_output><empty_output><jupyter_text>--- Reading a File  We will use, as an example, a catalog of all official named stars collected by the IAU in .csv format available athttps://www.kaggle.com/ecotner/named-stars This catalog contains 330 stars and the following columns:* **Name**: Proper name adopted by IAU Working Group on Star Names (WGSN).* **Designation**: Fiducial designation (in order of preference and availability):HR, GJ, HD, HIP, PSR. (see SIMBAD; http://simbad.u-strasbg.fr/simbad/).* **ID**: ID column is either (in order of preference and availability)Bayer greek letter (using SIMBAD databases; e.g. "alf" = alpha; see table http://simbad.u-strasbg.fr/Pages/guide/chA.htx), Flamsteed numbers (both following The Bright Star Catalogue [BSC], 5th Revised Ed. (Hoffleit & Warren 1991), or variable star designations.* **Con**: Constellation (3-letter IAU abbrev.; following "The IAU Style Manual", Wilkins 1989, Table 11).* ****: The "" after the constellation column is the component ID to Washington Double Star (WDS) catalog multiple system, if needed (e.g. Proxima Centauri = alf Cen "C"), but may be left blank ("-") where the primary, by visual brightness, is unambiguous (blank = A) or if the other WDS catalogue entries are obviously unphysical (i.e. the star appears to have no known physical stellar companions). Component IDs are also given where the star does not have a WDS entry (i.e. is not a resolved multiple), but the star is either known to be a spectroscopic binary or astrometric binary in the literature (usually a "A" or "Aa" is given in those cases, e.g. Wezen). Many thanks to <NAME> (IAU Commission G1 Binary and Multiple Star Systems) for help with vetting the WDS designations and IDs.* **WDS_J**: WDS designation for multiple systems (or candidate multiple systems), the official double star catalogue of IAU Commission 26 (now G1).* **Vmag**: Johnson V magnitude. For multiple stars, the V magnitude of the component is listed where available or calculable, however in some cases (e.g. some spectroscopic binaries, astrometric binaries), the unresolved magnitude is listed. Magnitudes are preferentially from The Hipparcos and Tycho catalogues (ESA 1997) and WDS.* **HIP**: HIP designations from The Hipparcos and Tycho Catalogues (ESA 1997).* **HD**: HD designations, usually cross-identified from SIMBAD, BSC, HIP catalogues.* **RA(J2000)**: Right Ascension, ICRS (epoch 2000.0)* **Dec(J2000)**: Declination, ICRS (epoch 2000.0)Columns 10 and 11 are calculated with Vizier for epoch 2000.0, usually using positions and proper motions from the revised Hipparcos catalogue ("Hipparcos, the New Reduction of the Raw Data" (van Leeuwen 2007 A&A, 474, 653), Gaia DR1 (Gaia Collaboration 2016) or Tycho-2 catalogue (Hog et al. 2000, A&A, 355, L27).* **Date**: Date approved by WGSN. Names marked approved "2015-12-15" are exoplanet host star names - a mix of common names and names adopted via the NameExoWorlds contest - reviewed and adopted by the Executive Committee WG Public Naming of Planets and Planetary Satellites, and recognized by WGSN (although the posted date pre-dates WGSN).<jupyter_code>df = pd.read_csv('IAU-CSN.csv')
df<jupyter_output><empty_output><jupyter_text>--- Group data We may group the data acording to some criteria. For example, 'Constellation':<jupyter_code>df.groupby(['Con']).groups<jupyter_output><empty_output><jupyter_text>If we want to get some specific group:<jupyter_code>Cyg_df = df.groupby(['Con']).get_group('Cyg')
Cyg_df<jupyter_output><empty_output><jupyter_text>Now, we will group the data according to the 'Vmag' feature<jupyter_code>df.groupby(['Vmag']).groups<jupyter_output><empty_output><jupyter_text>Note that there is one sample with a non-numerical value: item \[175\] has 'Vmag' = '-'.<jupyter_code>df.groupby(['Vmag']).get_group('-')
df.groupby(['Vmag']).get_group('1.25')<jupyter_output><empty_output><jupyter_text>--- Types of data in the dataframeThe data type of each column is obtained by<jupyter_code>df.dtypes<jupyter_output><empty_output><jupyter_text>Note that RA and Dec are the only columns with numerical values. The other columns are identified as 'object'. If we want to transform the data type of column 'Vmag' we need to get rid of the row \[175\] which has a non-numerical value. hence, we use the `.drop()` method.<jupyter_code>df = df.drop(175, axis=0)<jupyter_output><empty_output><jupyter_text>and now we transform the dtype of the 'Vmag' column to float:<jupyter_code>df['Vmag'] = df['Vmag'].astype(float)<jupyter_output><empty_output><jupyter_text>Note that the data type has indeed changed:<jupyter_code>df.dtypes<jupyter_output><empty_output><jupyter_text>A simplified way to acomplish the same result is using the `.to_numeric()` method, getting rid of errors using the option 'coerce'.<jupyter_code>df = pd.read_csv('IAU-CSN.csv') # We read again the original df

df['Vmag'] = pd.to_numeric(df['Vmag'],errors='coerce').astype(float)

df.dtypes<jupyter_output><empty_output><jupyter_text>---Lets now group the data by Constellation and find the maximum value of 'Vmag' in each one,<jupyter_code>df.groupby('Con')["Vmag"].max()<jupyter_output><empty_output><jupyter_text>Plotting Using the method `.plot()` is possible to plot all the numerical columns in the dataframe.<jupyter_code>df.plot();<jupyter_output><empty_output><jupyter_text>A histogram with the data is easily done:<jupyter_code>df.hist(); # The semi-colon does not show some extra information about the plot<jupyter_output><empty_output><jupyter_text>We can adjust some arguments in the function to concentrate in only one of the columns,<jupyter_code>df.hist(column='Vmag', figsize=(10,7));<jupyter_output><empty_output><jupyter_text>Using `matplotlib.pyplot` it is possible to customize the plot even more<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline

ax = df.hist(column='Vmag', figsize=(10,7))
ax = ax[0]
for x in ax:
    x.set_title('')
    x.set_xlabel('Visual Apparent Magnitude')
    x.set_ylabel('Count')
plt.show()<jupyter_output><empty_output><jupyter_text>Now, we make a scatter plot to visualise the stars in the sky according to their RA and Dec coordinates.<jupyter_code>fig, ax = plt.subplots(figsize=(10,7))
ax.scatter(df['RA(J2000)'], df['Dec(J2000)'], marker='.')
ax.set_xlabel('RA (J2000)')
ax.set_ylabel('Dec (J2000)')
plt.show()<jupyter_output><empty_output><jupyter_text>Memory Usage of DataFrames<jupyter_code>df.info(memory_usage="deep")<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 330 entries, 0 to 329
Data columns (total 13 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   Unnamed: 0   330 non-null    int64  
 1   Name         330 non-null    object 
 2   Designation  330 non-null    object 
 3   ID           330 non-null    object 
 4   Con          330 non-null    object 
 5   #            330 non-null    object 
 6   WDS_J        330 non-null    object 
 7   Vmag         329 non-null    float64
 8   HIP          330 non-null    object 
 9   HD           330 non-null    object 
 10  RA(J2000)    330 non-null    float64
 11  Dec(J2000)   330 non-null    float64
 12  Date         330 non-null    object 
dtypes: float64(3), int64(1), object(9)
memory usage: 191.7 KB<jupyter_text>Subsampling<jupyter_code>subsample = df.sample(frac=0.1, random_state=413) 
# frac = percentage of the original set
remaining = df.drop(subsample.index)
# .index extract the indices of a dataset
subsample
remaining<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11770298535843317, 'fraction_numerical': 0.027096406160867084, 'mean_word_length': 5.2837514934289125, 'pattern_counts': {'":': 0, '<': 136, '<?xml version=': 0, '>': 136, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '412644', 'license': 'MIT', 'n_tokens_mistral': 3761, 'n_tokens_neox': 3477, 'n_words': 1098, 'path': '02. Pandas/05. Introduction to Pandas.ipynb', 'repo_name': 'ashcat2005/ComputationalAstrophysics2021', 'size': '10549'}