<jupyter_start><jupyter_text>Pandas is a Library that helps data scientist to prepare transform and present the data in a more appealing and structured way, it is flexible, very capable and easy to use. It can manage the whole data preprocessing workflow without the need of jumping from a tool to another.Pandas uses data frames, a tabular data structure that organizes the data in labeled rows (indexes) and columns.Pandas is based on numpy, and can be used as a compliment for matplotlib, sci-kit, and other tool used for data science.This tutorial will show some basic functionalities and applications for Pandas Library. 1. Import libraries needed to manipulate data.For this example we will use pandas, numpy and matplotlib, but because their names are large, and not that easy to handle, we will put an alias for each one, and we will use popular aliases as follows:<jupyter_code>import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
from sklearn.utils import shuffle<jupyter_output><empty_output><jupyter_text>2. Basic Functions Series: One-Dimensional labeled array, which is able to hold any type of data.In the next example is easy to see how a series must be defined.<jupyter_code>serie_1 = pd.Series([2,3,4,5,6], index=['a','b','c','d','e'])<jupyter_output><empty_output><jupyter_text>The output will be a table, where the first column is the Index and the next one is the data stored in there.<jupyter_code>serie_1<jupyter_output><empty_output><jupyter_text>Some manipulation can be done to this structures, in order to be able to obtain certain information. Get one element<jupyter_code>serie_1['c']<jupyter_output><empty_output><jupyter_text>Set an index of a series to a value, for example set index c that belongs to a Series Serie_1 to 11<jupyter_code>serie_1['c'] = 11 # c is originally set to 4 then to 11
serie_1['c']<jupyter_output><empty_output><jupyter_text>Use boolean operations for indexing.<jupyter_code>serie_1[(serie_1 > 1)]
serie_1[(serie_1 >= 3) & (serie_1 <11)]<jupyter_output><empty_output><jupyter_text>Drop a value from a row, here we can specify how many rows you want to remove from the series, just using a comma to separate each element.<jupyter_code>serie_1.drop(['a','b'])<jupyter_output><empty_output><jupyter_text>Dataframe: Two-dimensional labeled data structure that is capable of hold different types of data per column. To simplify things we will first define the data stored in the dataframe, for this example we will use 4 models of electricc guitars, and some of their technical specifications.<jupyter_code>data = {'Guitar models' : ['Stratocaster', 'RGD', ' KM7 ', 'Les Paul'], 
        'scale length' : ['standard 25.5"', 'baritone 26.5"','baritone 26.5"', 'short 24.75" ' ],
        'number of strings' :['6', '6', 7, '6' ] }<jupyter_output><empty_output><jupyter_text>then proceed to create the data frame as follows:<jupyter_code>dataframe = pd.DataFrame(data, columns = ['Guitar models', 'scale length', 'number of strings'])
dataframe<jupyter_output><empty_output><jupyter_text>there some different ways to select specific values in dataframes, by position, by label, and both. By position<jupyter_code>dataframe.iloc[[1], [1]]<jupyter_output><empty_output><jupyter_text>If only one parameter is used it will retrieve the information corresponding to that parameter<jupyter_code>dataframe.iloc[2]<jupyter_output><empty_output><jupyter_text>By label<jupyter_code>dataframe.loc[[1], ['scale length']]<jupyter_output><empty_output><jupyter_text>3. Real life datasetOne of the most useful functionalities of Pandas, is to have the possibility of importing datasets from several document formats, The following example will explore how these Pandas work in a real life data set.For these example we will use a LEGO dataset.<jupyter_code>df = pd.read_csv(r'C:\Users\dengo\Downloads\lego-sets1\lego_sets.csv')

print (df)<jupyter_output>ages  list_price  num_reviews  piece_count  play_star_rating  \
0      6-12     29.9900          2.0        277.0               4.0   
1      6-12     19.9900          2.0        168.0               4.0   
2      6-12     12.9900         11.0         74.0               4.3   
3       12+     99.9900         23.0       1032.0               3.6   
4       12+     79.9900         14.0        744.0               3.2   
5       12+     59.9900          7.0        597.0               3.7   
6       12+     59.9900         37.0        598.0               3.7   
7       12+     49.9900         24.0        780.0               4.4   
8       12+     39.9900         23.0        468.0               3.6   
9       12+     39.9900         11.0        444.0               3.6   
10      12+     39.9900         14.0        386.0               4.1   
11      12+     34.9900         53.0        321.0               3.2   
12      12+     29.9900          7.0        361.0               4.2   
13    [...]<jupyter_text>We can obtain the amount of lines  and columns in the dataset. this information can help us later to determine percentage of the data that will be used for training process and the rest for the testing process.<jupyter_code>df.shape
 print("Total samples:", len(df))<jupyter_output>Total samples: 12261<jupyter_text>Lest say we are interested to make a comparison between the amount of pieces per set, and the list price, the id of the product is included for reference.<jupyter_code># dataframe piece count vs price
df1 =  df[['piece_count', 'list_price' , 'prod_id']]
print(df1)<jupyter_output>piece_count  list_price  prod_id
0            277.0     29.9900  75823.0
1            168.0     19.9900  75822.0
2             74.0     12.9900  75821.0
3           1032.0     99.9900  21030.0
4            744.0     79.9900  21035.0
5            597.0     59.9900  21039.0
6            598.0     59.9900  21028.0
7            780.0     49.9900  21029.0
8            468.0     39.9900  21034.0
9            444.0     39.9900  21033.0
10           386.0     39.9900  21036.0
11           321.0     34.9900  21019.0
12           361.0     29.9900  21032.0
13           847.0    159.9900  17101.0
14           708.0     29.9900  41597.0
15           234.0     19.9900  41614.0
16           160.0     19.9900  41613.0
17           149.0      9.9900  41609.0
18           141.0      9.9900  41608.0
19           101.0      9.9900  41604.0
20           105.0      9.9900  41605.0
21           113.0      9.9900  41606.0
22           136.0      9.9900  41607.0
23            91.0      9.9900  41485.0
[...]<jupyter_text>We can calculate the median of our data.<jupyter_code>mean_price =  df1['list_price'].mean()
print(mean_price)<jupyter_output>65.14199840959138<jupyter_text>Minimun and maximun values for dataset df1.<jupyter_code>df1.min()
df1.max()<jupyter_output><empty_output><jupyter_text>We can verify if there is any null in the dataset.<jupyter_code>df1.isnull().values.any()<jupyter_output><empty_output><jupyter_text>If necesary, it is possible to shuffle data in dataframes.<jupyter_code>df2 =  df[['piece_count', 'list_price' , 'prod_id']]
df2 = shuffle(df2)
print(df2)<jupyter_output>piece_count  list_price    prod_id
4745          63.0     18.2878    10737.0
4540          53.0      7.3078    60156.0
11282        728.0     75.3971    31069.0
8080         622.0     85.3878    41314.0
9148          36.0     24.3878    10816.0
7759        7541.0    909.9860    75192.0
2135         191.0     48.7878    10744.0
7781         140.0     25.1860    42058.0
11501        160.0     34.7971    76076.0
10134         78.0      6.0878    10706.0
11232        387.0     52.1971    60175.0
562           67.0      9.9900    70339.0
12100        312.0     36.5878    70608.0
3026         601.0    311.9922    31313.0
10074        888.0    219.5878    60052.0
3787         149.0     12.1380    41609.0
1061         149.0     15.1924    41190.0
10469         69.0     12.1878    70634.0
328           28.0     29.9900    10608.0
4884         101.0     36.5878    75535.0
1171           1.0    250.7924    45500.0
3890           8.0      4.9980    71020.0
7673          73.0     13.9860    [...]<jupyter_text>It is possible to take advantage of other libraries, in this case plot the information to make it more appealing.<jupyter_code>df2.plot.scatter(x='piece_count', y='list_price')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0836734693877551, 'fraction_numerical': 0.14273709483793517, 'mean_word_length': 2.353864734299517, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 78, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '73176', 'license': 'MIT', 'n_tokens_mistral': 3343, 'n_tokens_neox': 2769, 'n_words': 886, 'path': 'Inv-Corta-1.ipynb', 'repo_name': 'imperialfelp7/Inv-Corta-1', 'size': '8330'}