<jupyter_start><jupyter_text>I am currently in the process of acquiring my US citizenship. It's a long process without a set schedule. I found myself frequently visiting the USCIS status page to see if any update had been posted. After a few iterations I grew tired and rather than waste time going to the website, enter my case number, navigate to the status page, and check if there were any changes, I decided to automate these steps with a python script.In this post I detail the steps of the script, which does the status check for me and shoots me an e-mail when detecting a change. I then show how I made it into a cron job so that it would run automatically on a schedule of my choosing.Here I leverage [SELENIUM](http://www.seleniumhq.org/projects/webdriver/), [BEAUTIFULSOUP](https://www.crummy.com/software/BeautifulSoup/), and [SMTPLIB](https://docs.python.org/3/library/smtplib.html) and [PYTHON-CRONTAB](https://pypi.python.org/pypi/python-crontab). Note that building my automaton requires scouting the pages it needs to navigate to identify the specific elements it needs to interact with; in this case only a handful.<jupyter_code>from selenium import webdriver
import os
from bs4 import BeautifulSoup as Soup
from shutil import which
import sys, IPython, platform
print(f'Setup: {platform.machine()} running {" ".join(platform.linux_distribution()[:2])}')
print(f'Python: {sys.version[:5]}\nIPython: {IPython.__version__}')<jupyter_output>Setup: x86_64 running debian stretch/sid
Python: 3.6.3
IPython: 6.1.0<jupyter_text>The last step I went through in my process is an interview with a case officer. Thus the phrase that if altered on my status page should indicate a a change in my status is "we scheduled an interview". Let's first store that in a token...<jupyter_code>token = 'we scheduled an interview'<jupyter_output><empty_output><jupyter_text>Here, I use [phantomjs](http://phantomjs.org/) to avoid the browser window. **phantompath** gives the location of phantomjs on my computer. If you prefer chromedriver [<NAME> (the Flask guy) has a recipe](https://blog.miguelgrinberg.com/post/using-headless-chrome-with-selenium) for the cost of a couple extra lines.<jupyter_code>phantompath = which('phantomjs')
browser = webdriver.PhantomJS(executable_path=phantompath)<jupyter_output><empty_output><jupyter_text>Navigate to the USCIS landing page...<jupyter_code>browser.get('https://egov.uscis.gov/casestatus/landing.do')<jupyter_output><empty_output><jupyter_text>... find the input box to enter my case number, stored in the variable case_num...<jupyter_code>caseNo = browser.find_element_by_id('receipt_number')
caseNo.send_keys(case_num)<jupyter_output><empty_output><jupyter_text>... find the button to initiate the lookup and click...<jupyter_code>srchButtn = browser.find_element_by_name('initCaseSearch')
srchButtn.click()<jupyter_output><empty_output><jupyter_text>... now we're on the status page, get the page source and parse it with BeautifulSoup<jupyter_code>pageSrc = browser.page_source
pageSoup = Soup(pageSrc, 'lxml')<jupyter_output><empty_output><jupyter_text>... from inspecting the page source, I know that the text I'm looking for is under tag "p"...<jupyter_code>targetSection = pageSoup.find('p')<jupyter_output><empty_output><jupyter_text>... and is the first item yielded by the child of targetSection (a generator)...<jupyter_code>targeText = list(targetSection.children)[0]<jupyter_output><empty_output><jupyter_text>... sampling some of the text returns...<jupyter_code>targeText[20:100]<jupyter_output><empty_output><jupyter_text>... well, the status still reflects the interview, so nothing happens after this and the process goes back to sleep. However, the rest of the code shows the next steps in the case of change. Namely* craft message including the new status text* intantiate an SMTP object* call the SMTP object's sendmail method to send the email from/to the appropriate sender/receiver pair, in my case they are one and the same...<jupyter_code>if token not in targeText:
    message = f'USCIS CASE UPDATE: \n {targeText}'
    try:
        smtpObj = smtplib.SMTP('localhost')
        smtpObj.sendmail(from_, to, message)
    except SMTPException:
        print("Problem sending email")<jupyter_output><empty_output><jupyter_text>The full source code is available at the end of the page. I saved this code as **CheckCaseStatus.py**, and the snippets below show how I add it to my cron tasks. First import CronTab and instantiate a cron object.<jupyter_code>from crontab import CronTab

myCron = CronTab(user=True)<jupyter_output><empty_output><jupyter_text>I'm the only user on my computer so setting user=True is sufficient. On a multi-user setup, an actual username should be specified. Now add running the CheckCaseStatus.py as a new job to the cron instance...<jupyter_code>runpath = os.path.join('~/DEV/MyCronJobs/CheckCaseStatus.py')
job = myCron.new(command='python CheckCaseStatus.py')<jupyter_output><empty_output><jupyter_text>... set the running schedule - check for a change every 5 days - and save...<jupyter_code>job.day.every(5)
myCron.write()
# CheckCaseStatus.py
from selenium import webdriver
import os
from bs4 import BeautifulSoup as Soup
import smtplib
from shutil import which

def check_uscis_page(caseNum, phantompath):
    """
    opens uscis webpage in the background (phantomjs)
    navigates to case status page (requires case number)
    """
    browser = webdriver.PhantomJS(executable_path=phantompath)
    browser.get('https://egov.uscis.gov/casestatus/landing.do')
    caseNo = browser.find_element_by_id('receipt_number')
    caseNo.send_keys(caseNum)
    srchButtn = browser.find_element_by_name('initCaseSearch')
    srchButtn.click()
    pageSrc = browser.pageSource
    pageSoup = Soup(pageSrc, 'lxml')
    browser.close()
    targetSection = pageSoup.find('p')
    return list(targetSection.children)[0]

def check_update(targeText, token):
    if token in targeText:
        return False
    else:
        return True

def shoot_email(text, from_, to):
    from_ = from_
    to = [to]
    message = f'USCIS CASE UPDATE: \n {text}'
    try:
        smtpObj = smtplib.SMTP('localhost')
        smtpObj.sendmail(from_, to, message)
    except SMTPException:
        print("Problem sending email")

if __name__ == '__main__':
    myEmail = os.getenv('MYGMAIL')
    case = os.getenv('USCIS_CASE_NO')
    phantompath = which('phantomjs')   
    retrievedText = check_uscis_page(case, phantompath)
    if  check_update(retrievedText, 'we scheduled an interview'):
        shoot_email(retrievedText, myEmail, myEmail) 
# CronScheduler.py
from crontab import CronTab

myCron = CronTab(user=True)
job = myCron.new(command='python CheckCaseStatus.py')
job.day.every(5)
myCron.write()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0963572267920094, 'fraction_numerical': 0.003672150411280846, 'mean_word_length': 5.063223508459483, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '616443', 'license': 'MIT', 'n_tokens_mistral': 2194, 'n_tokens_neox': 2099, 'n_words': 686, 'path': 'posts/automatically-checking-changes-in-my-citizenship-case-status.ipynb', 'repo_name': 'madHatter106/DataScienceCorner', 'size': '6817'}