<jupyter_start><jupyter_text>The first step is to get the web-page url from the seek jobs website where the job includes "data analyst". The job search for this page is limited to all of Australia Next, the number of pages that the search included needed to be obtained so that we know how many pages to loop through. It was found that there were 22 listings per page, and thus the numebr of jobs was divided by 22. After this it was noticed that the job advertisements just duplicate. Thus a decision was made to round up the number of pages/22 and then delete duplicate entries later.<jupyter_code># Retrieve page with the requests module

#it was noticed that the seek page had 22 jobs listed per page. Thus in order to obtain the number of pages, first need to obtain the number of pages:
#Number of jobs listed at the top of the page, the following code obtains this:
job_soup=bs(jobs_html, 'html.parser')
job_number=job_soup.find('span', class_="_3FrNV7v _3PZrylH _2heRYaN").text

number=job_number.strip('jobs found')

number_1=float(number)
print(number_1)

number_pages=math.ceil(number_1/22)
print(number_pages)

time.sleep(2)<jupyter_output>220.0
10<jupyter_text>The code in the excerpt below saves the job title, location and company for data analyst jobs throughout Australia into lists. This was done through using looping and webscraping.<jupyter_code>#getting the browser html
text_list=[]
job_location=[]
job_title1=[]
job_company=[]
job_classification=[]
x=range(1,number_pages+1)
for n in x:
    seek_url=f"https://www.seek.com.au/jobs/in-All-Australia?keywords=%22data%20%20analyst%22&page={n}"
    browser.visit(seek_url)
    seek_html=browser.html  
    seek_soup=bs(seek_html, 'html.parser')
    all_text=seek_soup.find('div', class_='_3MPUOLE')
    job_title=all_text.find_all('span', class_='_3FrNV7v _2IOW3OW HfVIlOd _2heRYaN E6m4BZb')
    extra_class=all_text.find_all('span', class_="Eadjc1o")
    job_characteristics=all_text.find_all('span', class_="_3FrNV7v _3PZrylH E6m4BZb")

    #converting each of the elements to text 
    
    #modifying the job location element and tidying up this element
    for element in extra_class:
        element1=element.text
        if "location:" in element1:
            element2=element1.lstrip('location: ')
            job_location.append(element2) 
    
    job_title=all_text.find_all('span', class_='_3FrNV7v _2IOW3OW HfVIlOd _2heRYaN E6m4BZb') 
    
    #editing both the job title and job characterstics
    for item in job_title:
        job1=item.text
        job_title1.append(job1)
    
    for company in job_characteristics:
        company1=company.text
        if company1[:3]=='at ':
            company2=company1.lstrip('at ')
            job_company.append(company2) 
    
    for job_class in extra_class:
        classification1=job_class.text
        if "classification:" in classification1:
            classification2=classification1.lstrip('classification: ')
            job_classification.append(classification2) 
    
    time.sleep(3)
            
browser.quit()<jupyter_output><empty_output><jupyter_text>The lists were then combined in a pandas dataframe<jupyter_code>#zip two lists together 

Job_df = pd.DataFrame(
    {'Job Title': job_title1,
     'Job Location': job_location,
     'Job Company': job_company,
     'Job Industry': job_classification
    })

Job_df.head(20)<jupyter_output><empty_output><jupyter_text>Duplicates were then removed from the dataset<jupyter_code>#removing duplicates if all three columns are the same: 

Job_df2=Job_df.drop_duplicates()<jupyter_output><empty_output><jupyter_text>In order to use the Leaflet library, the longitude and latitude co-ordinates needed to be obtained for each job, This will be done using the Google Places API. First, a search string column was added to the dataframe. This Search Engins column will be used as a parameter in the Google Places API.<jupyter_code>#setting up search to use in the API call 
Job_df2["Search Engine"] = Job_df2["Job Company"].astype(str) + " " + Job_df2["Job Location"].astype(str)
Job_df2.head()<jupyter_output><empty_output><jupyter_text>The below code adds Lat and Lng columns to the dataframe to set up the dataframe further for the Google API<jupyter_code>#adding latitude and longitude columns
Job_df2['Lat'] = ""
Job_df2['Lng']=""<jupyter_output>/Users/dhanyamaheswaran/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  
/Users/dhanyamaheswaran/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:3: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  This is separate from the ipykernel package so we can avoid doing imports until<jupyter_text>The following code loops through all the search engine column and outputs a Lat and Lng from the Google Places API. Try and Except is used so that if there is an error, the code will not automatically stop.<jupyter_code>for index, row in Job_df2.iterrows():
    
    search_string=row["Search Engine"]
    
    try:
        url = f'https://maps.googleapis.com/maps/api/place/findplacefromtext/json?input={search_string}&inputtype=textquery&fields=photos,formatted_address,name,rating,opening_hours,geometry&key={gkey}'
        # Run request
        response = requests.get(url)
        search_geo = response.json()
        
    
        Job_df2.loc[index, 'Lat'] = search_geo["candidates"][0]['geometry']['location']['lat']
        Job_df2.loc[index, "Lng"]=search_geo["candidates"][0]['geometry']['location']['lng']

    
    #in the case of a JSONDecode Error, this ensures the code will keep running
    except (KeyError, IndexError, ValueError):
        Job_df2.loc[index, 'Lat'] = "No Coordinates"
        Job_df2.loc[index, 'Lng'] = "No Coordinates"

Job_df2.head(15)<jupyter_output><empty_output><jupyter_text>In order to double check, the entries where lat and lng coordinates were not found were just displayed.<jupyter_code>#check the ones that have no co-ordinates
Job_df2.loc[Job_df2['Lat'] == "No Coordinates"]<jupyter_output><empty_output><jupyter_text>In order to data-clean, all Jobs where no coordinates were found was then deleted from the dataframe and it was exported as a csv<jupyter_code>#Then delete the ones with no co-ordinates
final_Job_df=Job_df2.loc[Job_df2['Lat'] != "No Coordinates"]
#Finally export it to a csv
final_Job_df.to_csv("data/Final_job_df.csv")
df2=pd.read_csv("data/Final_job_df.csv")
#looking at al lthe unique values and mapping them to a state 
df2["Job Location"].unique()

#replacing with states for the location values 
# create ChainMap
L1 = ['Adelaide']
d1 = dict.fromkeys(L1, 'SA')

L2 = ['Gold Coast', "Brisbane", 'Northern QLD', 'Cairns & Far North']
d2 = dict.fromkeys(L2, 'QLD')

L3 = ['Sydney', "Southern Highlands & Tablelands", 'Newcastle, Maitland & Hunter', 'Wollongong, Illawarra & South Coast', 'Blue Mountains & Central West']
d3 = dict.fromkeys(L3, 'NSW')

L4 = ['Melbourne', "South West Coast VIC"]
d4 = dict.fromkeys(L4, 'VIC')

L5 = ['Bunbury & South West', "Perth"]
d5 = dict.fromkeys(L5, 'WA')

L6 = ['Hobart', "Launceston & North East"]
d6 = dict.fromkeys(L6, 'TAS')

L7 = ['Alice Springs & Central Australia']
d7 = dict.fromkeys(L7, 'NT')

L8 = ['ACT']
d8 = dict.fromkeys(L8, 'ACT')


d = {**d1, **d2, **d3, **d4, **d5, **d6, **d7, **d8}
print (d)


df2['State'] = df2['Job Location'].map(d)
df2.head(15)
#count by location the gauge chart 
#Location in NSW
#by sub-category for each state
df2=df2.rename(columns={"Job Industry": "Industry"})

df2["Industry"].unique()

Industry2=df2.Industry.map({'Sales' : "Sales", 'Marketing & Communications': 'Marketing', 'Information & Communication Technology' : "ICT",
                 'Government & Defence' : "Govt", 'Banking & Financial Services': "Banking",'Science & Technology': "Science",'Consulting & Strategy' : "Consulting", 
                  "Insurance & Superannuation" : "Insurance", "Administration & Office Support" : "Admin", "Accounting": "Accounting", 
                 "Healthcare & Medical":"Healthcare", "Education & Training" : "Educ", "Human Resources & Recruitment": "HR", "Real Estate & Property": "Real Estate", "Manufacturing, Transport & Logistics":"Transport", "Hospitality & Tourism": "Hospitality", 
                 "Community Services & Development":"Community", "Engineering":"Engineering", "Mining, Resources & Energy":"Mining" })
df2['Job Industry 2'] = Industry2

grouped_state = df2.groupby('State')
grouped_state_1=grouped_state["Job Title"].count()

grouped_state_1
list1=[]


#dropping all NA values
df2=df2.dropna()
grouped_count_state=pd.DataFrame(grouped_state_1)
grouped_state_1=grouped_state_1.to_dict()
list1.append(grouped_state_1)

print(list1)
grouped_state_2 = df2.groupby(['State','Job Industry 2'])

grouped_state_industry=grouped_state_2["Job Title"].count()
grouped_state_industry=pd.DataFrame(grouped_state_industry)
from collections import defaultdict
results = defaultdict(lambda: defaultdict(dict))

for index, value in grouped_state_industry.itertuples():
    for i, key in enumerate(index):
        if i == 0:
            nested = results[key]
        elif i == len(index) - 1:
            nested[key] = value
        else:
            nested = nested[key]


 
list1.append(results)
grouped_state_3 = df2.groupby(['State','Job Location'])
grouped_state_location=grouped_state_3["Job Title"].count()
grouped_state_location=pd.DataFrame(grouped_state_location)
from collections import defaultdict
results2 = defaultdict(lambda: defaultdict(dict))

for index, value in grouped_state_location.itertuples():
    for i, key in enumerate(index):
        if i == 0:
            nested = results2[key]
        elif i == len(index) - 1:
            nested[key] = value
        else:
            nested = nested[key]

list1.append(results2)


with open('data/job_data.json', 'w') as fp:
    json.dump(list1, fp)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11206150888995675, 'fraction_numerical': 0.017683805862566074, 'mean_word_length': 4.576634512325831, 'pattern_counts': {'":': 10, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '102634', 'license': 'MIT', 'n_tokens_mistral': 3515, 'n_tokens_neox': 3372, 'n_words': 946, 'path': 'Webscraping/web-scraping-seek.ipynb', 'repo_name': 'EstherBaili/Project2', 'size': '10405'}