<jupyter_start><jupyter_text>In this notebook we are going to analyse the data scrapped from jumia.cm<jupyter_code># importing modules
import numpy as np
import pandas as pd
import re 
%matplotlib inline
import matplotlib.pyplot as plt
import time
# loading the scrapped data
df = pd.read_csv('./jumia_cm_smartphones_scraped_on_Sun_May_26_21:47:08_2019.csv')
# Getting somary statistic about our data
print(df.info())<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 910 entries, 0 to 909
Data columns (total 13 columns):
name                    910 non-null object
seller                  910 non-null object
price                   910 non-null float64
number-of-items-sold    863 non-null float64
ram                     796 non-null float64
rom                     824 non-null float64
screen-size             768 non-null float64
color                   910 non-null object
front-camera            797 non-null float64
rear-camera             797 non-null float64
number_of_sims          910 non-null int64
date                    910 non-null object
os                      872 non-null object
dtypes: float64(7), int64(1), object(5)
memory usage: 92.5+ KB
None<jupyter_text>Description of the data analysisFrom the above result, we clearly see that our data have 910 rows. Also we clearly see that some columns entries are missing. we can see that we have the following: 1. names has 910 entries so no missing data2. seller has 910 entries so no missing data3. price has 910 entries so no missing data4. number-of-items-sold  has 863 so we have missing data here5. ram has 796 so we have missing data here6. rom has 824 so we have missing data here7. screen-size has 768 so we have missing data here8. color has 910 so no missing data here9. front-camera has 797 so we have missing data here10. rear-camera has 797 so we have missing data here11. number-of-sim has 910 so we have  no missing data here12. date has no missing data and it worth noting that we filled the date property after scrape data on the said item13. os has 872 entries so we have missing data here.The pandas info methode is really good if we want to identifies missing data from our dataset.  CLEANING OUR DATASET We have a messy data that we should see how to clean it<jupyter_code># We can use pandas dropna method to derop all rows with NAN values
#droping rows with missing values
#without_missing_values = df.dropna()
#getting the description of the data without missing values
#without_missing_values.describe()
# print(without_missing_values['os'].astype('category').value_counts())
# pd.value_counts(df['os']=='ios')
# os = df['os'].astype('category').value_counts()
#price = df['price'].astype('category').value_counts()
#sold = df['number-of-items-sold'].astype('category').value_counts()
#print(sold)
#print(os)
#print(price)
# bins=['android', 'ios']
# #pd.cut(df, bins)
#below= (df['price']<50000).value_counts()
#print(below)
#print(df['number-of-items-sold ']).max()
''' Getting the summary of the dataset'''
summary = df.describe()
print(summary)

''' From the summary statistic we clearly see that the lower price is 5000xaf around 8.6usd wish is not right so lets
check if actually it is a phone. We will do this by subsetting our dataset even below 15000xaf(17.2usd)'''

least_item= df.loc[df['price']<=15000]
print(least_item)

'''We clearly see that  Blitzwolf BW-BTS1 Sport Bluetooth Earphone Head, SP4 Senior - 32Mo - 4Mo RAM - 2,2" - Blanc,
 Chargeur Pour Samsung Galaxy - Noir are not smart phones so we should drop them '''
df=df.drop([351, 612, 908], axis=0)
''' Lets drop the columns that has missing values so that we can have good data to work with
we can achieve this by using the dropna pandas method'''

smartphones = df.dropna()


''' Getting a new summary of the data'''
print('-----------------------------SUMMARY STATISTICS--------------------------------------------')
smartphones.describe()


# print('--------------------------CUMMULATIVE SUM-----------------------------------------------------')
# print(df[.cumsum())<jupyter_output>-----------------------------SUMMARY STATISTICS--------------------------------------------<jupyter_text>DETECTING AND REMOVING OUTLIERSAn outlier is a data point that differs significantly from other observations we can easily look at the max of the numeric values 1. From the summary statistic above we can clearly see that the max prise is too high  so wan can investigate  After the investigation below to see wich phone is sold at 22100000 we see that its  'Galaxy S6 Edge - 32Go HDD - RAM 3Go' which is not possible so we drop it  2. we can find outliers in the rom section since no phone till now sell on jumia.cm has a rom of 1616GO so we can check and fix since we can clearly see that the real value is on the item name(the difficulty to get it from the name is due to the inconsistency of the item name )<jupyter_code>print(smartphones.loc[smartphones['price']>2000000])
print('--------------------------- outliers------------------------------------------------')
''' Checking the outliers in the rom section'''
print(smartphones.loc[smartphones['rom']>258])

'''We drop the item that is supposed to cost 2210000'''
smartphones.drop(451, 0, inplace=True)

''' After detecting it we can fix it'''
smartphones.at[229, 'rom']=8
smartphones.at[229, 'ram']=1
smartphones.at[339, 'rom']=126
smartphones.at[339, 'ram']=4
smartphones.at[611, 'rom']=32
smartphones.at[613, 'rom']=8<jupyter_output>name   seller      price  \
451  Galaxy S6 Edge - 32Go HDD - RAM 3Go - 5,7" - O...  Samsung  2210000.0   

     number-of-items-sold  ram   rom  screen-size color  front-camera  \
451                  49.0  3.0  64.0          5.7    or           8.0   

     rear-camera  number_of_sims                      date       os  
451         12.0               1  Sun May 26 21:33:11 2019  android  
--------------------------- outliers------------------------------------------------
                                                  name   seller     price  \
229  Galaxy J2 Core Dual Sim - 8Go HDD + 1Go RAM - ...  Samsung   56800.0   
339  Redmi Note 7 Dual SIM - 128Go - 4Go Ram HDD - ...    Xaomi  200500.0   
611                        iPhone 7 Plus 32Go HDD - Or    Apple  513600.0   
613            A2 XL Dual SIM 8Go HDD - Noir Et Argent  Alcatel   45000.0   

     number-of-items-sold  ram     rom  screen-size           color  \
229           [...]<jupyter_text>Our dataset is clean to a certain extend so we can analyse it now let's create a new column computing the price of all items sold<jupyter_code>''' Having rename the dataset we are working with as smartphones
 Here we compute the '''
smartphones['total-price']=smartphones['price'] *  smartphones[ 'number-of-items-sold']
smartphones.head()<jupyter_output>/home/devherve/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy<jupyter_text>GETTING SOME INSIGHT FROM OUR DATA1. Bellow we clearly see most sold phones on the platform and their prices     We have 7 phones that the number of sales is above 100000    2. Samsung  has 296 varieties of equipments on the platform ,Apple 104, Tecno 60, Huawei 44, ITEL 44, Xiaomi          37,Infinix 36, Lenovo 17, X-tigi 16,LG 13 and the rest bellow 103. for the total numbers of items sold, Samsung is leading with 461970.0 items sold, Huawei 236752.0, Xiaomi          209343.0, Infinix 156974.0, ITEL  133346.0 and LG 101717.0  and the rest below 1000004. from the analysis we clearly see that sumsung phones have been sold for 81947570000xaf(140736756.72usd), apple      44062290000xaf( 75672576.85usd) etc... as the result of all this, we can see that jumia.cm has sold more than at at total of 236153558184xaf( 405570120.83 usd)<jupyter_code>'''Most sale smartphones on jumia at the date of Sun May 26 21:24:44 2019 '''
most_sell = smartphones['number-of-items-sold'].max()
print('the maximum number of items sold is  {0}'.format(most_sell))
'''Lets get the most sold items '''
most_sold_items= smartphones.loc[smartphones['number-of-items-sold']==most_sell]
print(most_sold_items)

#print(smartphones.loc[smartphones['number-of-items-sold'].idxmax()])

''' making a good print of it'''
print('-------------Below most sold items and their sellers-------------------------------')
print(most_sold_items[['name', 'seller', 'price', 'number-of-items-sold']])

'''numbers of items own by a specific seller'''
sellers = smartphones['seller'].astype('category').value_counts()
print('-------------------numbers of items own by a specific seller---------------------------------')
print(sellers)

''' best seller with highest number of items sold'''
best = smartphones['number-of-items-sold'].groupby(smartphones['seller']).sum()

print('-----------------------number of equipments sold by each seller---------------------')
print(best.sort_values(ascending=False))


''' amount generated by each marque on the websites'''
totalCashByBrand= smartphones['total-price'].groupby(smartphones['seller']).sum()
print(totalCashByBrand.sort_values(ascending=False))
'''cash generated by the items in our dataset'''
total = smartphones['total-price'].sum()
totalSold=smartphones['number-of-items-sold'].sum()
print('---------------------the cash generated in total is -------------------------')
print('Jumia.cm has generated {0}xaf by selling {1} phones we have in our dataset'.format(total, totalSold))<jupyter_output>---------------------the cash generated in total is -------------------------
Jumia.cm has generated 236153558184.0xaf by selling phones we have in our dataset
{'fraction_non_alphanumeric': 0.14713064713064714, 'fraction_numerical': 0.054436304436304435, 'mean_word_length': 2.82005441119316, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '523202', 'license': 'MIT', 'n_tokens_mistral': 3233, 'n_tokens_neox': 2772, 'n_words': 1117, 'path': 'analysing_craped_data.ipynb', 'repo_name': 'dherve19/scraping-jumia-for-data-science', 'size': '9828'}