<jupyter_start><jupyter_text>I have use the above script 5 time to get the data in crunches as per my internet speed and computational speed. I have scrapped data 5 time and store the data in5 dataframes and then finally concate then to a big data.* I have change the link every time i run (give the url of next page before which i have scraped the data)<jupyter_code>df1 = get_jobs(180)
df1
df1.to_csv('df1.csv', index = False)
df2 = get_jobs(200)
df2
df2.to_csv('df2.csv', index = False)
df3 = get_jobs(200)
df3
df3.to_csv('df3.csv', index = False)
#url = 'https://www.glassdoor.co.in/Job/data-scientist-jobs-SRCH_KO0,14_IP21.htm'

df4 = get_jobs(200)
df4
df4.to_csv('df4.csv', index = False)
df5 = get_jobs(200)
df5
df5.to_csv('df5.csv', index = False)
import pandas as pd
df1=pd.read_csv("df1.csv")
df1.head()
df2=pd.read_csv("df2.csv")
df2.head()
df3=pd.read_csv("df3.csv")
df3.head()
df4=pd.read_csv("df4.csv")
df4.head()
df5=pd.read_csv("df5.csv")
df5.head()
original_data=pd.concat([df1,df2,df3,df4,df5],axis=0,ignore_index=True)
original_data.shape
df1.shape
df2.shape
original_data.head(10)
original_data.to_csv('original_data.csv', index = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13900589721988205, 'fraction_numerical': 0.05728727885425442, 'mean_word_length': 5.75, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '528993', 'license': 'MIT', 'n_tokens_mistral': 514, 'n_tokens_neox': 485, 'n_words': 118, 'path': '1-GlassDoor Data scrapper for india.ipynb', 'repo_name': 'tushar176/Data-Science-Salary-Analysis', 'size': '1187'}