<jupyter_start><jupyter_text><jupyter_code>from google.colab import drive
drive.mount('/content/drive')
ls /content/drive/My\ Drive/TDI\ Datasets/
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
data = pd.read_csv("/content/drive/My Drive/TDI Datasets/temp_datalab_records_job_listings.csv", nrows=1000000 )<jupyter_output><empty_output><jupyter_text>Let's see how the data looks like ...<jupyter_code>data.head()
data.describe()<jupyter_output><empty_output><jupyter_text>**I can see a lot of NaNs in the dataset. First see how many NaNs each column has.**<jupyter_code>data.isna().sum()<jupyter_output><empty_output><jupyter_text>**Findings**1) All of the last six columns are NaNs. 2) Most of the 'brand', 'category', and 'number of openings' are NaN.  Finding the number total number of countries in dataset and the number of job openings in each country.<jupyter_code>data['country'].value_counts()
#plotting the number of jobs n top 20 countries in the world
data['country'].value_counts().head(20).plot.bar(figsize=(10,5))<jupyter_output><empty_output><jupyter_text>**Conclusion**In this dataset the number of job openings in USA is far more than any other nation. Lets explore the jobs in USA only.<jupyter_code>dataset = data.copy()
dataset = dataset[dataset.country == 'USA']
pd.set_option('display.max_colwidth', 1000)
pd.set_option('display.max_rows', 1000)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)
# dataset.describe()
#Finding unique values in this dataset 
dataset.nunique()<jupyter_output><empty_output><jupyter_text>**Let's see what are the most popular cities in terms of job opportunities. We will limit the plot to top 20 regions in USA.**<jupyter_code>pd.DataFrame(dataset['region'].value_counts()).head(20).plot.bar(figsize = (15,5))<jupyter_output><empty_output><jupyter_text>**Looks like the California is the highest contributor to job market.  Now I want to dive more deeper in each region (top 5 actually) and see what are the job trends in each of them**<jupyter_code>cali = dataset[dataset.region == 'CA']
tx = dataset[dataset.region == 'TX']
fl = dataset[dataset.region == 'FL']
wa = dataset[dataset.region == 'WA']
ny = dataset[dataset.region == 'NY']

cali['category'].value_counts().head(20).plot.bar(figsize = (15,5))<jupyter_output><empty_output><jupyter_text>**Looks like there are more jobs in Sales and Engineering than in Finances and Manufacturing in California. What about the other regions like Texas, Florida, New York ?**<jupyter_code>#Top 20 job categories in Texas 
tx['category'].value_counts().head(20).plot.bar(figsize = (15,5))
#Top 20 job categories in Florida
fl['category'].value_counts().head(20).plot.bar(figsize = (15,5))
#Top 20 job categories in New York
ny['category'].value_counts().head(20).plot.bar(figsize = (15,5))<jupyter_output><empty_output><jupyter_text>**Conclusion**Florida and Texas has more jobs in Stores and Retail Sales while in New York Operations, regular Part-Time and Sales are more dominant  **What are the most common job titles in all these regions**<jupyter_code>#Top 20 job Job Titles in California
cali['title'].value_counts().head(20).plot.bar(figsize = (15,5))
#Top 20 job Job Titles in New York
ny['title'].value_counts().head(20).plot.bar(figsize = (15,5))<jupyter_output><empty_output><jupyter_text>**Let's Explore the trend in job postings in California, Texas and New York. What months are these jobs posted the most.**<jupyter_code>#Month-wise job postings in California
pd.to_datetime(cali['posted_date']).dt.month.value_counts().sort_index().plot.bar(figsize = (15,5))
#Month-wise job postings in Texas
pd.to_datetime(tx['posted_date']).dt.month.value_counts().sort_index().plot.bar(figsize = (15,5))
#Month-wise job postings in New York
pd.to_datetime(ny['posted_date']).dt.month.value_counts().sort_index().plot.bar(figsize = (15,5))<jupyter_output><empty_output><jupyter_text>**Colclusion****No matter how does the job titles and categories vary in California and New York, they all follow the same trend while rolling out jobs**  What are the companies in Califronia, New York and Florida that hire the most?<jupyter_code>#Top 20 Companies Titles in California
cali['brand'].value_counts().head(20).plot.bar(figsize = (15,5))
#Top 20 Companies Titles in NY
ny['brand'].value_counts().head(20).plot.bar(figsize = (15,5))
#Top 20 Companies Titles in Florida
fl['brand'].value_counts().head(20).plot.bar(figsize = (15,5))<jupyter_output><empty_output><jupyter_text>**Hmm.. All of these places have different mass recruiters. What about the whole USA ? Who recruits the most in whole America ?**<jupyter_code>dataset['brand'].value_counts().head(20).plot.bar(figsize = (15,5))<jupyter_output><empty_output><jupyter_text>**And the winner in 'Most Recruitments Across All States' is : Amazon**  Where should I go if I want to find a job in Software engineering or Software Development ?<jupyter_code>dataset.dropna(subset=['title'], axis=0, inplace=True)
dataset[(dataset['title'].str.contains('Software')) | (dataset['title'].str.contains('software'))]['region'].value_counts().head(20).plot.bar(figsize = (15,5))<jupyter_output><empty_output><jupyter_text>**Conclusion**Washington ? and not California ? Thats something surprising, isn't it ?Despite the huge overall job postings, California is behind Washington. I should move to Washington for a role in Softwares.  Where should I go if I want to find a job in Hardware?<jupyter_code>dataset[(dataset['title'].str.contains('Hardware')) | (dataset['title'].str.contains('hardware'))]['region'].value_counts().head(20).plot.bar(figsize = (15,5))<jupyter_output><empty_output><jupyter_text>**Conclusion**As expected, the most hardware related jobs are in California and CA leads with a huge margin.   Where should I go if I want to find a job in Legal?<jupyter_code>dataset[(dataset['title'].str.contains('Legal')) | (dataset['title'].str.contains('legal'))]['region'].value_counts().head(20).plot.bar(figsize = (15,5))<jupyter_output><empty_output><jupyter_text>**Conclusion**I always had the presumption that all the Legal jobs must be in New York. I guess I am too much influenced with 'Suits'. California leads the Legal battle, followed by New York.  I am more interested in Data Science and Machine Learning. Where are the jobs in this field ?<jupyter_code>dataset[(dataset['title'].str.contains('Data Science')) | ((dataset['title'].str.contains('Machine Learning')))]['region'].value_counts().head(20).plot.pie(figsize = (10,10))<jupyter_output><empty_output><jupyter_text>**California, Massachusetts, New York.** **Noted!** Who hires the most for Data Science and Machine Learning jobs?<jupyter_code>dataset[(dataset['title'].str.contains('Data Science')) | ((dataset['title'].str.contains('Machine Learning')))]['brand'].value_counts().head(10).plot.pie(figsize = (10,10)) #bar(figsize = (15,5))<jupyter_output><empty_output><jupyter_text>**Conclusion**Okay, that was unexpected. Where is google and Facebook ?Amazon is hiring all the Data Scientists and Machine Learning Engineers ? I wonder what are they upto.<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13791660856226468, 'fraction_numerical': 0.020220331892344165, 'mean_word_length': 7.01340782122905, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 73, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '342985', 'license': 'MIT', 'n_tokens_mistral': 2351, 'n_tokens_neox': 2228, 'n_words': 653, 'path': 'TDI_job_listing.ipynb', 'repo_name': 'parasgulati8/Data-Analysis', 'size': '7171'}