<jupyter_start><jupyter_text>Section 232, Webscrapping with PythonThis Jupyter notebook is an exploration of webscrapping the US Dpt of Commerce website to collect information about the Tariff exemption requests. This exploration is made following a feasbility inquiry from <NAME> and <NAME>.<jupyter_code>#Base URL of the U.S. Department of Commerce, Section 232 Steel and Aluminum website
burl = 'https://232app.azurewebsites.net/'

#URL for Details for an exclusion, you need to concatenate the Request ID to the right
durl = burl + '/Forms/ExclusionRequestItem/'

#Lambda function providing the URL for the details of a given Request ID
drurl = (lambda id: durl + str(id))

#Request ID for unit testing, 42561 is not arbitrated, 22561 has attachment
ridut = 22561

#Check that the lambda perform as expected
print(drurl(ridut))<jupyter_output>https://232app.azurewebsites.net//Forms/ExclusionRequestItem/22561<jupyter_text>The lambda function return the expected result, the URL match the pattern of the website.  Obtain the data for a given Request ID<jupyter_code>#Import required libraries
import requests
from bs4 import BeautifulSoup

#r is the request sent to the website
r = requests.get(drurl(ridut))

#Cannot use the json() method as the answer is not a json but an html
#json_data = r.json()
#for key, value in json_data.items():
#    print(key + ': ' + value)

#Display the content of the answer
rhtml = r.text                 #Request html answer
rsoup = BeautifulSoup(rhtml)   #Create soup for the individual request
print(rsoup.prettify())        #Print the prettify html of the detail request<jupyter_output>$(elem).html(bisFormLabels[product]["labels"][index]);
        if (bisFormLabels[product]["help"][index]) {
            $(elem).after("<div class='bisformdynamichelp'> <span data-balloon='size: 2x' data-balloon-pos='up' class='db color-inherit link hover-cyan' data-placement='right' title='" + bisFormLabels[product]["help"][index] + "'><i class='fa fa-info-circle'  aria-hidden='true'></i></span></div>");
        }
    });
    let loadingDiv = $('.loading-div');
    if (loadingDiv.length) {
        loadingDiv.hide();
    }
    let help = $(".bisformdynamichelp>span");
    if (help.length) {
        help.tooltip({ viewport: { selector: '.bisform', padding: 0 } });
    }
}

function GetOR(d, callback) {

    var id = d[0];
    var tblId = "or_" + id + "_";
    var template = "<div class='tree-level'>"
        + "<table width='100%' id='" + tblId + "'>"
        + "<thead>"
        + "<td class='spacer'></td>"
        + "<td colspan='4'>Objection Rebuttals</td>"
        + "</thead>"
    [...]<jupyter_text>The above is the html that define the Detail webpage of the individual exclusion request.  Exploration of the associated html<jupyter_code>#General information and function to manipulate the html using beautifulsoup
print('Tag: ' + str(rsoup.tag))
print('Name: ' + str(rsoup.name))
print('Title string: ' + rsoup.title.string)
print('Title Parent Name: ' + rsoup.title.parent.name)
print('p: ', str(rsoup.p))
print('p[`class`]: ', str(rsoup.p['class'][0]))
print('\n' + 'First link: ')
print(rsoup.a)
#Print list of link on the webpage
print('\n' + 'All links target:')
for link in rsoup.find_all('a'):
    print(link.get('href'))
#Print text of the webpage
print('\n' + 'Text:')
print(rsoup.get_text())

print('\n' + 'soup.input: ')
print(rsoup.input)   #just return one tag, could use in a loop/generator context, using .next_element

print('\n' + 'Use find to target a specific tag')
mytag = rsoup.find(id="BIS232Request_Product")
print('id: ' + mytag['id'])        #id in this context is the identification of the tag, not the Request ID
print('title: ' + mytag['title'])  #title, name and label are possible text to be used for the columns name 
print('name: ' + mytag['name'])
print('value: ' + mytag['value'])  #is the content of the field itself
#can perform a test using mytag.has_attr('xyz')
print('xyz: ' + mytag['xyz'])      #this attribute does not exist, calling it generate an error and not a Blank

##Lookin to access the label of the id as this is the caption on the form. It is under the 'for=' class.
#mytag = rsoup.find(for="BIS232Request_Product")  #error, probably because "for" is a keyword
#print('label: ' + mytag['label'])<jupyter_output>Tag: None
Name: [document]
Title string: Exclusion Request 22561
Title Parent Name: head
p:  <p class="section-text">Section 232 Steel and Aluminum</p>
p[`class`]:  section-text

First link: 
<a class="active" href="https://www.commerce.gov/" title="Home"><img alt="Department of Commerce" class="site-logo m-3" src="https://www.commerce.gov/themes/commerce/assets/img/doc_logo.png"/></a>

All links target:
https://www.commerce.gov/
https://www.commerce.gov/
/22561
https://www.commerce.gov/page/section-232-investigations
/Identity/Account/Login
/Index
#

mailto:<EMAIL>?subject=Aluminum exclusion request inquiry
tel:2024824757
mailto:<EMAIL>?subject=Steel exclusion request inquiry
tel:2024825642
https://www.commerce.gov/
https://www.commerce.gov/
https://www.commerce.gov/about/policies/accessibility
https://www.commerce.gov/about/policies/privacy

Text:





Exclusion Request 22561






















U.S. Department of Commerce
Section 232 Steel and Aluminum[...]<jupyter_text>The hyperlink provided the attached document is not identifiable, just the "/22561"The values of the fields do not surface through get_text method, only the labels and extended captions.calling an attribute that does not exist in a tag return an error Searching for the value in the webpageBy looking at the html it appears that the field and value are all under the input tag<jupyter_code>import pandas as pd
import datetime

#lit = rsoup.find_all('input')     #list input tag, but we are not interested in tags w/o value
def has_value(ltag):               #function allow for more refined list of tags to consider
    return ltag.has_attr('value') and ltag.name == 'input' 

startt = datetime.datetime.now()   #calculate run time for one

lit = rsoup.find_all(has_value)    #list of tags matching the above function

##For readability, build a dictionary with only the usefull attributes of each tags. 
## First attempt, error as the designated keys do not exist in each of the tags we are looping on
#for t in lit:           #Loop the list of input tag
#    print(t['name'])
#    print(t['title'])
#    print(t['value'])

## Second attempt, similar error, crash when asking for keys that are not present
#myattr = ['id'] #, 'name', 'title', 'label', 'value']    #List of attributes to keep
#for t in lit:            #Loop the list of input tag
#    dat = t.attrs # dictionary of attibutes of that tag
#    trad = { k: dat[k] for k in myattr }  #tag reduced attribute dictionary #using dictionary comprehension 
#    print(trad)

## Third attempt,
#myattr = ['id', 'name', 'title', 'label', 'value']    #List of attributes to keep
#for t in lit:            #Loop the list of input tag
#    dat = t.attrs        # dictionary of attibutes of that tag
#    #the next instruction error
#    mylattr = myattr[dat.keys()]  #Idea is to have a mylattr contains the intersection of myattr and the tag attribute
#    trad = { k: dat[k] for k in mylattr }  #tag reduced attribute dictionary #using dictionary comprehension
#    print(trad)   

## Fourth attempt, return a data frame that only contains the selected tags
#myattr = ('id', 'name', 'title', 'label', 'value')    #List of attributes to keep
#mydf = pd.DataFrame({'id': [], 'name': [], 'title': [], 'label': [], 'value': []} ) #forcing the desired columns
#mydf.columns = myattr     #not required
#for t in lit:            #Loop the list of input tag
#    dat = t.attrs        # dictionary of attibutes of that tag
#    trad = dict((k,v) for k, v in dat.items() if k in myattr)  #tag reduced attribute dictionary 
#    mydf = mydf.append(trad, ignore_index=True)

## Generalized version
mydf = pd.DataFrame()
for t in lit:            #Loop the list of input tag
    dat = t.attrs        # dictionary of attibutes of that tag
    mydf = mydf.append(dat, ignore_index=True)

endt = datetime.datetime.now()
print('Run duration: ', endt-startt)
    
#All the attributes of all tags of the detail webpage are now in the data frame
print(mydf.head())
print(mydf.info())

#saving to csv to allow visualiation in Excel
mydf.to_csv('mydf.csv') #save at the local directory of the jupyter notebook
#mydf.to_csv(r'C:\Users\Raphael.Louvrier\OneDrive - Vallourec\3 Personal Development\MyPython\Section232\mydf.csv')  #with absolute path, need the little r<jupyter_output>Run duration:  0:00:01.447139
                   class data-toggle data-val  \
0  [form-control-static]     tooltip     true   
1  [form-control-static]         NaN     true   
2  [form-control-static]         NaN     true   
3  [form-control-static]         NaN      NaN   
4         [form-control]         NaN     true   

                                   data-val-required  \
0  The Please select product type field is required.   
1  The Identify the class of product for which th...   
2                   The HTSUSCode field is required.   
3                                                NaN   
4  The Full Organization Legal Name field is requ...   

                                                  id  \
0                              BIS232Request_Product   
1                  BIS232Request_JSONData_MetalClass   
2                            BIS232Request_HTSUSCode   
3         BIS232Request_JSONData_PreviouslyGrantedER   
4  BIS232Request_JSONData_RequestingOrg_OrgLegalName   

 [...]<jupyter_text>This analysis was made for given Request ID. It is not certain that the form is identical for all type and kind of request. Running a loop over a greater number of consecutive request and build a dataframe<jupyter_code>startt = datetime.datetime.now()   #calculate run time
print('Progress ', end='')

mydf = pd.DataFrame()               #initialize the dataframe
lr = list(range(46557, 46560))      #list of requests --- Pls adjust ---
for ridut in lr:
    r = requests.get(drurl(ridut))  #r is the request sent to the website
    rhtml = r.text                  #Request html answer
    rsoup = BeautifulSoup(rhtml)    #Create soup for the individual request
    lit = rsoup.find_all(has_value) #list of tags matching the selection function
    for t in lit:                   #Loop the list of input tag
        dat = t.attrs               #Dictionary of attibutes of that tag
        dat['request_id'] = str(ridut) #add the request id to the dictionary
        mydf = mydf.append(dat, ignore_index=True) #append to the dataframe
    print('.', end='')              #progress bar one dot per request processed
print()                             #go to new line

#All the attributes of all tags of all the detail webpages from lr are now in the data frame
print(mydf.info())

#saving to csv
mydf.to_csv(r'mydfexplore.csv') #save at the local directory of the jupyter notebook

endt = datetime.datetime.now()
print('Run duration: ', endt-startt)<jupyter_output>Progress ...
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 906 entries, 0 to 905
Data columns (total 26 columns):
class                     702 non-null object
data-toggle               3 non-null object
data-val                  528 non-null object
data-val-required         501 non-null object
id                        897 non-null object
name                      900 non-null object
readonly                  867 non-null object
request_id                906 non-null object
title                     561 non-null object
type                      906 non-null object
value                     906 non-null object
data-val-length           6 non-null object
data-val-length-max       6 non-null object
data-val-regex            309 non-null object
data-val-regex-pattern    309 non-null object
data-val-phone            15 non-null object
data-val-email            15 non-null object
data-val-url              12 non-null object
onchange                  12 non-null object
rows              [...]<jupyter_text>run was made for 60 requests from 46500 to 46559. There are still 26 columns. We will assume that there are only 26 different tags. For convenience, the exploration of the tags will be done in Excel using the csv file generated.  The following tags will be discarded:data-toggle, too few valuesrows, too few valuesaria-describedby, too few valueschecked, too few values, equivalent to value =TRUEplaceholder, too few value, not applicable, see titleonchange, too few values, not applicabledata-val-phone, error messagedata-val-email, error messagedata-val-url, too few valuesdata-val-regex, error messagedata-val-regex-pattern, error messagestyle, not applicableonclick, not applicabledata-val-required, error messagereadonly, emptydata-val-length, error messagedata-val-length-max, error messageid, name provide more and cleaner designation than id, therefore droping iddata-val, after further analysis, not requiredtype, after further analysis, not required, button and submit to be excluded, equivalent to filter out blank namearia-hidden, after further analysis, not requiredclass, cannot use the class to filter out keys as the use is inconsistent in the ProductClassification section, = ['btn', 'btn-info', 'backbutton']  and  ['btn', 'btn-primary', 'submitButton']  to be excluded, equivalent to filter out blank name The following tags will be kept: -request_id, will be the key of the final export-name, only the one starting with BIS232Request-title-valueNext, decide proper column heading for result. This will use id, name, title and possibly the label attribute as well. name provide more and cleaner result than id, therefore droping idtitle are nicer than name, but some are missingThe following name, do not have title, need to insertProduct              - Product MetalClass           - Metal ClassRequestingOrg.State  - Requesting Organization StateID                   - Request IDname is also required for the ProductClassification section, cannot dropvalue is value but there are key and value for the array section of the form to be taken care of.<jupyter_code>import re                          #for regular expression

def fkeeptag(ltag):                #function allow for more refined list of tags to consider
    bn = True                      #boolean for name, by default the tag is included
    if ltag.has_attr('name'):      #to avoid error need to check is the name attribute exist for that tag
        lname = ltag['name']
        #exclude name = __RequestVerificationToken, and empty, equivalent only keep name starting with BIS232Request
        #if lname == '' or lname == '__RequestVerificationToken':
        #    bn = False
        bn = not(lname == '' or lname == '__RequestVerificationToken')    
        #We will use regular expression ^ match begin of string, $ match at end of string, (|) or
        #Exlude name in *ChemicalComposition[*].Key, ProductClassification[0].Key, *ProductDimensions[*].Key,
        #*ToughnessJoules[*].Key, *ToughnessShear[*].Key, *ToughnessTemperature[*].Key
        #*ProductProcessing.ProcessesPercentage[*].Key, *ProductProcessing.Processes[*].Key, 
        lrex = None != (re.search( r'^BIS232Request.JSONData.(ProductClassification|ChemicalComposition|ProductDimensions|ProductStrength|ProductProcessing.ProcessesPercentage|ProductProcessing.Processes|ToughnessJoules|ToughnessShear|ToughnessTemperature).*Key$', lname, flags=0))
        #if lreg: bn = False #the name attribute match the above exclusion pattern
    else:                           #exclude tag without name attribute
        bn = False                  #no name attribute, exclude
    return ltag.has_attr('value') and ltag.name == 'input' and bn and not(lrex)

def fkeepprodclass(ltag):           #function allow for more refined list of tags to consider
    if ltag.has_attr('name'):      #to avoid error need to check is the name attribute exist for that tag
        lrex = None != (re.search( r'^BIS232Request.JSONData.ProductClassification.*Key$', ltag['name'], flags=0)) #lrex true if prodclass
    else:                           #exclude tag without name attribute
        lrex = False                
    return lrex

startt = datetime.datetime.now()    #calculate run time
print('Progress ', end='')

mydf = pd.DataFrame()               #initialize the dataframe
lr = list(range(46558, 46560))      #list of requests --- Pls adjust ---

#one time fill in the ProductClassification dictionary for title
r = requests.get(drurl(lr[0]))  #r is the request sent to the website
rhtml = r.text                  #Request html answer
rsoup = BeautifulSoup(rhtml)    #Create soup for the individual request    
dpc = {}                        #dictionary for title for product classificaiton value
lit = rsoup.find_all(fkeepprodclass)  #list of tags matching the selection function
for t in lit:                   #Loop the list of input tag for ProductClassification
    dat = t.attrs               #Dictionary of attibutes of that tag
    for ls in ('BIS232Request.', 'JSONData.'): #for readability sake replace BIS232Request. and JSONData. with '' 
        dat['name'] = dat['name'].replace(ls, '', 1)
    if   dat['value'] != '': ltrg = dat['value']  #if not empty use value of ProductClassification[i].Key 
    elif dat['title'] != '': ltrg = dat['title']  #else use title of ProductClassification[i].Key 
    else: ltrg = ''
    #save the title for ProductClassification[i].Value in the dpc dictionary
    dpc[dat['name'].replace('.Key', '.Value', 1)] = ltrg

#loop on all the request in lr
for ridut in lr:
    r = requests.get(drurl(ridut))  #r is the request sent to the website
    rhtml = r.text                  #Request html answer
    rsoup = BeautifulSoup(rhtml)    #Create soup for the individual request

    #the dpc creation could be inserted here if the title change base on the type of request
        
    lit = rsoup.find_all(fkeeptag)  #list of tags matching the selection function
    # <----error, the prodclass.key are still in the lit at this stage, should not, using the bskip to get rid of them
            
    for t in lit:                   #Loop the list of input tag
        rdat = {}                   #Reduce dictionary of attributes
        bskip = False               #to skip '.Key' insertion in the result
        dat = t.attrs               #Dictionary of attibutes of that tag
        for ls in ('BIS232Request.', 'JSONData.'): #for readability sake replace BIS232Request. and JSONData. with '' 
            dat['name'] = dat['name'].replace(ls, '', 1)
        lname = dat['name']
        #set title to following for the given name
        #Product -> Product, MetalClass -> Metal Class, RequestingOrg.State -> Requesting Organization State, ID -> Request ID
        if   lname == 'Product':             dat['title'] = 'Product'
        elif lname == 'MetalClass':          dat['title'] = 'Metal Class'
        elif lname == 'RequestingOrg.State': dat['title'] = 'Requesting Organization State'
        elif lname == 'ID':                  dat['title'] = 'Request ID'
        
        #If the tag is a ProductClassification[*].Value, call the title for the dpc dictionary 
        if lname[:22] == 'ProductClassification[' and lname[-7:] == '].Value':
            dat['title'] = dpc[lname]
        elif lname[:22] == 'ProductClassification[': 
            bskip = True
        
        #mistake in the title of the ToughnessShear[*].Value, replace Joule by Shear
        if lname[:15] == 'ToughnessShear[' and lname[-7:] == '].Value':
            dat['title'] = dat['title'].replace('Joule', 'Shear', 1)
        
        #mistake in the title of the bendadility % instead of mm
        if lname[:37] == 'ProductProcessing.Processes[0].Value.':
            dat['title'] = dat['title'].replace('Percentage', 'mm', 1)
        
        #fall back, use the name if no title is available
        if t.has_attr('title'):
            if dat['title'] == '': 
                dat['title'] = 'name: ' + str(dat['name'])
        else:
            dat['title'] = 'name: ' + str(dat['name'])
            
        #fill in reduced dictionary
        rdat['request_id'] = str(ridut) #add the request id to the dictionary
        rdat['title'] = dat['title']
        rdat['value'] = dat['value']
        if not(bskip):
            mydf = mydf.append(rdat, ignore_index=True) #append to the dataframe
    print('.', end='')              #progress bar one dot per request processed
print()                             #go to new line

#title, name and value for all fields the detail webpages from lr are now in the data frame
print(mydf.info())

#saving to csv, need to export and check for empty title
#once we are good on the title then we will drop the name and denormalize
mydf.to_csv(r'mydfcleaner.csv') #save at the local directory of the jupyter notebook

endt = datetime.datetime.now()
print('Run duration:', endt-startt, 'for', len(lr), 'requests')<jupyter_output>Progress ..
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 426 entries, 0 to 425
Data columns (total 3 columns):
request_id    426 non-null object
title         426 non-null object
value         426 non-null object
dtypes: object(3)
memory usage: 10.1+ KB
None
Run duration: 0:00:04.347347 for 2 requests<jupyter_text>A detail review of the webpage of the detail request and the export from above shows thatwe miss a few information as- Sumission date, done by adding p tags- Public Status, done by adding p tags- multiple comments fields, done by adding the textarea's tags- Ports, ?- Country of import, ?- presence of attachments, ?- presence of decision, done section is only present when there is a decision memoleading to addition to the code<jupyter_code>import re                          #for regular expression

def fkeeptag(ltag):                #function allow for more refined list of tags to consider
    bn = True                      #boolean for name, by default the tag is included
    if ltag.has_attr('name'):      #to avoid error need to check is the name attribute exist for that tag
        lname = ltag['name']
        #exclude name = __RequestVerificationToken, and empty, equivalent only keep name starting with BIS232Request
        #if lname == '' or lname == '__RequestVerificationToken':
        #    bn = False
        bn = not(lname == '' or lname == '__RequestVerificationToken')    
        #We will use regular expression ^ match begin of string, $ match at end of string, (|) or
        #Exlude name in *ChemicalComposition[*].Key, ProductClassification[0].Key, *ProductDimensions[*].Key,
        #*ToughnessJoules[*].Key, *ToughnessShear[*].Key, *ToughnessTemperature[*].Key
        #*ProductProcessing.ProcessesPercentage[*].Key, *ProductProcessing.Processes[*].Key, 
        lrex = None != (re.search( r'^BIS232Request.JSONData.(ProductClassification|ChemicalComposition|ProductDimensions|ProductStrength|ProductProcessing.ProcessesPercentage|ProductProcessing.Processes|ToughnessJoules|ToughnessShear|ToughnessTemperature).*Key$', lname, flags=0))
        #if lreg: bn = False #the name attribute match the above exclusion pattern
    else:                           #exclude tag without name attribute
        bn = False                  #no name attribute, exclude
    return (( ltag.has_attr('value') and ltag.name == 'input' ) or ltag.name == 'textarea') and bn and not(lrex)

def fkeepprodclass(ltag):           #function allow for more refined list of tags to consider
    if ltag.has_attr('name'):      #to avoid error need to check is the name attribute exist for that tag
        lrex = None != (re.search( r'^BIS232Request.JSONData.ProductClassification.*Key$', ltag['name'], flags=0)) #lrex true if prodclass
    else:                           #exclude tag without name attribute
        lrex = False                
    return lrex

def fattach(ltag):           #function allow for more refined list of tags to consider
    bn = False
    if ltag.has_attr('id'):      #to avoid error need to check is the name attribute exist for that tag
        bn = (ltag['id'] == 'ERAttachment' or ltag['id'] == 'DMAttachment')              
    return bn

startt = datetime.datetime.now()    #calculate run time
print('Progress ', end='')

mydf = pd.DataFrame()               #initialize the dataframe
mytbl = pd.DataFrame()               #initialize the dataframe
lr = list(range(46558, 46560))      #list of requests --- Pls adjust ---
lr = (22561, 33333, 44444, 46558)     
#one time fill in the ProductClassification dictionary for title
r = requests.get(drurl(lr[0]))  #r is the request sent to the website
rhtml = r.text                  #Request html answer
rsoup = BeautifulSoup(rhtml)    #Create soup for the individual request    
dpc = {}                        #dictionary for title for product classificaiton value
lit = rsoup.find_all(fkeepprodclass)  #list of tags matching the selection function
for t in lit:                   #Loop the list of input tag for ProductClassification
    dat = t.attrs               #Dictionary of attibutes of that tag
    for ls in ('BIS232Request.', 'JSONData.'): #for readability sake replace BIS232Request. and JSONData. with '' 
        dat['name'] = dat['name'].replace(ls, '', 1)
    if   dat['value'] != '': ltrg = dat['value']  #if not empty use value of ProductClassification[i].Key 
    elif dat['title'] != '': ltrg = dat['title']  #else use title of ProductClassification[i].Key 
    else: ltrg = ''
    #save the title for ProductClassification[i].Value in the dpc dictionary
    dpc[dat['name'].replace('.Key', '.Value', 1)] = ltrg

#loop on all the request in lr
for ridut in lr:
    r = requests.get(drurl(ridut))  #r is the request sent to the website
    rhtml = r.text                  #Request html answer
    rsoup = BeautifulSoup(rhtml)    #Create soup for the individual request

    myd = {}                        #pivoted version
    myd['request_id'] = str(ridut)
    #find ports
    #mt = rsoup.find(name='BIS232Request.JSONData.ExclusionExplanation.ShipmentQtyEstimate') #BIS232Request.JSONData.Ports[0] #
    #print('Ports:',mt)
    
    #the dpc creation could be inserted here if the title change base on the type of request

    #Acquiring p tags for Submission date and Public Status
    lit = rsoup.find_all('p')
    for t in lit:                                       #Loop the list of p tag 
        if t.text[:17] == 'Submission Date: ':          #exemple:Submission Date: 12/13/2019
            rdat = {}                                   #Reduce dictionary of attributes
            rdat['request_id'] = str(ridut)             #add the request id to the dictionary
            rdat['title'] = 'Submission Date'           #fill in reduced dictionary
            rdat['value']= t.text[17:]
            mydf = mydf.append(rdat, ignore_index=True) #append to the dataframe
        if t.text[:15] == 'Public Status: ':            #exemple:Public Status: Pending-Objection Window Open
            rdat = {}                                   #Reduce dictionary of attributes
            rdat['request_id'] = str(ridut)             #add the request id to the dictionary
            rdat['title'] = 'Public Status'             #fill in reduced dictionary
            rdat['value']= t.text[15:]
            mydf = mydf.append(rdat, ignore_index=True) #append to the dataframe
            
    lit = rsoup.find_all(fkeeptag)  #list of tags matching the selection function
    # <----error, the prodclass.key are still in the lit at this stage, should not, using the bskip to get rid of them
            
    for t in lit:                   #Loop the list of input tag
        rdat = {}                   #Reduce dictionary of attributes
        bskip = False               #to skip '.Key' insertion in the result
        dat = t.attrs               #Dictionary of attibutes of that tag
        for ls in ('BIS232Request.', 'JSONData.'): #for readability sake replace BIS232Request. and JSONData. with '' 
            dat['name'] = dat['name'].replace(ls, '', 1)
        lname = dat['name']
        
        #set title to following for the given name
        #Product -> Product, MetalClass -> Metal Class, RequestingOrg.State -> Requesting Organization State, ID -> Request ID
        if   lname == 'Product':             dat['title'] = 'Product'
        elif lname == 'MetalClass':          dat['title'] = 'Metal Class'
        elif lname == 'RequestingOrg.State': dat['title'] = 'Requesting Organization State'
        elif lname == 'ID':                  dat['title'] = 'Request ID'
        
        #If the tag is a ProductClassification[*].Value, call the title for the dpc dictionary 
        if lname[:22] == 'ProductClassification[' and lname[-7:] == '].Value':
            dat['title'] = dpc[lname]
        elif lname[:22] == 'ProductClassification[': 
            bskip = True
        
        #mistake in the title of the ToughnessShear[*].Value, replace Joule by Shear
        if lname[:15] == 'ToughnessShear[' and lname[-7:] == '].Value':
            dat['title'] = dat['title'].replace('Joule', 'Shear', 1)
        
        #mistake in the title of the bendadility % instead of mm
        if lname[:37] == 'ProductProcessing.Processes[0].Value.':
            dat['title'] = dat['title'].replace('Percentage', 'mm', 1)
        
        #fall back, use the name if no title is available
        if t.has_attr('title'):
            if dat['title'] == '': 
                dat['title'] = 'name: ' + str(dat['name'])
        else:
            dat['title'] = 'name: ' + str(dat['name'])
        
        #for textarea, there is no value, fill in with text
        if t.name == 'textarea': 
            dat['value'] = t.text
            for sc in ('\r','\n','\t'): 
                dat['value'] = dat['value'].replace(sc,"")
        
        #fill in reduced dictionary
        rdat['request_id'] = str(ridut) #add the request id to the dictionary
        rdat['title'] = dat['title']
        rdat['value'] = dat['value']
        if not(bskip):
            mydf = mydf.append(rdat, ignore_index=True) #append to the dataframe
    
    #add links for attachment and decision memo
    lit = rsoup.find_all(fattach)
    bisdm = False #boolean is there a decision memo
    for t in lit:                               #Loop the list of link tag
        if t.has_attr('id'):   
            if t['id'] == 'DMAttachment':       #the section BIS Decision memo is not present when no DM 
                bisdm = True 
            #if t['id'] == 'ERAttachment':        
                #cannot find any difference between 22561 with ER and 44558 w/o ER
    
    rdat = {}                       #Reduce dictionary of attributes
    rdat['request_id'] = str(ridut) #add the request id to the dictionary
    rdat['title']= 'BIS Decision Memo'
    rdat['value']= bisdm
    mydf = mydf.append(rdat, ignore_index=True) #append to the dataframe 
    myd[rdat['title']] = rdat['value']
    rdat['title']= 'Attachment'
    rdat['value']= ''               #dont know for the attachment
    mydf = mydf.append(rdat, ignore_index=True) #append to the dataframe

    print('.', end='')              #progress bar one dot per request processed
print()                             #go to new line

#title, name and value for all fields the detail webpages from lr are now in the data frame
print(mydf.info())

#saving to csv, need to export and check for empty title
mydf.to_csv(r'mydfcleaner.csv') #save at the local directory of the jupyter notebook

endt = datetime.datetime.now()
print('Run duration:', endt-startt, 'for', len(lr), 'requests')

#Pivoting the resulting dataframe
#print('columns:')
#print(mydf.columns.values.tolist())
#mydf.drop_duplicates(subset=('request_id', 'title'))
#print(mydf.pivot(index='request_id', columns='title', values='value')) #error if use index?, columns are by alphabetical order<jupyter_output>Progress ....
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1016 entries, 0 to 1015
Data columns (total 3 columns):
request_id    1016 non-null object
title         1016 non-null object
value         1016 non-null object
dtypes: object(3)
memory usage: 23.9+ KB
None
Run duration: 0:00:10.859996 for 4 requests<jupyter_text>We now need to turn the data into a usable table with index set to Request ID.Consolidate the various import and function call from all the above code<jupyter_code>import requests
from bs4 import BeautifulSoup
import pandas as pd
import datetime
import re                          #for regular expression

myfile = 'mytbl.csv'
burl = 'https://232app.azurewebsites.net/'   #Base URL of the U.S. Department of Commerce, Section 232 Steel and Aluminum website
durl = burl + '/Forms/ExclusionRequestItem/' #URL for Details for an exclusion, you need to concatenate the Request ID to the right
drurl = (lambda i: durl + str(i))            #Lambda function providing the URL for the details of a given Request ID

def fkeeptag(ltag):                #function allow for more refined list of tags to consider
    bn = True                      #boolean for name, by default the tag is included
    if ltag.has_attr('name'):      #to avoid error need to check is the name attribute exist for that tag
        lname = ltag['name']
        #exclude name = __RequestVerificationToken, and empty, equivalent only keep name starting with BIS232Request
        #if lname == '' or lname == '__RequestVerificationToken':
        #    bn = False
        bn = not(lname == '' or lname == '__RequestVerificationToken')    
        #We will use regular expression ^ match begin of string, $ match at end of string, (|) or
        #Exlude name in *ChemicalComposition[*].Key, ProductClassification[0].Key, *ProductDimensions[*].Key,
        #*ToughnessJoules[*].Key, *ToughnessShear[*].Key, *ToughnessTemperature[*].Key
        #*ProductProcessing.ProcessesPercentage[*].Key, *ProductProcessing.Processes[*].Key, 
        lrex = None != (re.search( r'^BIS232Request.JSONData.(ProductClassification|ChemicalComposition|ProductDimensions|ProductStrength|ProductProcessing.ProcessesPercentage|ProductProcessing.Processes|ToughnessJoules|ToughnessShear|ToughnessTemperature).*Key$', lname, flags=0))
        lexceptionkey = ('BIS232Request.JSONData.ChemicalComposition[27].Key', 'BIS232Request.JSONData.ProductStrength[2].Key', 'BIS232Request.JSONData.ToughnessTemperature[2].Key')
        if lname in lexceptionkey:
            lrex = False
    else:                           #exclude tag without name attribute
        bn = False                  #no name attribute, exclude
    return (( ltag.has_attr('value') and ltag.name == 'input' ) or ltag.name == 'textarea') and bn and not(lrex)

def fprodclass(ltag):           #function allow for more refined list of tags to consider
    if ltag.has_attr('name'):      #to avoid error need to check is the name attribute exist for that tag
        lrex = None != (re.search( r'^BIS232Request.JSONData.ProductClassification.*Key$', ltag['name'], flags=0)) #lrex true if prodclass
    else:                           #exclude tag without name attribute
        lrex = False                
    return lrex

def fattach(ltag):           #function allow for more refined list of tags to consider
    bn = False
    if ltag.has_attr('id'):      #to avoid error need to check is the name attribute exist for that tag
        bn = (ltag['id'] == 'ERAttachment' or ltag['id'] == 'DMAttachment')              
    return bn

def cleantext(linput):           #function allow for more refined list of tags to consider
    ls = str(linput)
    #if ls.find(r'\ ') >= 0:      #to avoid error need to check is the name attribute exist for that tag         
    lss = re.sub(r"[\r\n\t]", '', ls).strip()
        #for sc in ('\r','\n','\t', '\f', '\v'): 
        #    ls = ls.replace(sc,"")
    if ls == lss:
        output = linput
    else:
        output = lss
    return output

startt = datetime.datetime.now()    #calculate run time
listrequest = list(range(46525, 46565))      #list of requests --- Pls adjust ---   
#lr = (22561, 33333, 44444, 46558) 

listrequest.sort()
try:
    mytbl = pd.read_csv(myfile)              #load previously extarcted csv
    listrdone = list(mytbl2['Request ID'])   #list of request id already in the file
    listrdone.sort()
except:                                      #if there is no such file
    listrdone = []                           #there is no request already done

# lr = listrequest - listrdone
lr = [rid for rid in listrequest if rid not in listrdone] #using list comprehension

print('Progress ', end='')           #Progress bar

mytbl = pd.DataFrame()               #reset the dataframe, so it only contains new entries

#one time fill in the ProductClassification dictionary for title
r = requests.get(drurl(lr[0]))  #r is the request sent to the website
rhtml = r.text                  #Request html answer
rsoup = BeautifulSoup(rhtml)    #Create soup for the individual request    
dpc = {}                        #dictionary for title for product classificaiton value
lit = rsoup.find_all(fprodclass)  #list of tags matching the selection function
for t in lit:                   #Loop the list of input tag for ProductClassification
    dat = t.attrs               #Dictionary of attibutes of that tag
    for ls in ('BIS232Request.', 'JSONData.'): #for readability sake replace BIS232Request. and JSONData. with '' 
        dat['name'] = dat['name'].replace(ls, '', 1)
    if   dat['value'] != '': ltrg = dat['value']  #if not empty use value of ProductClassification[i].Key 
    elif dat['title'] != '': ltrg = dat['title']  #else use title of ProductClassification[i].Key 
    else: ltrg = ''
    #save the title for ProductClassification[i].Value in the dpc dictionary
    dpc[dat['name'].replace('.Key', '.Value', 1)] = ltrg
    
#loop on all the request in lr
for ridut in lr:
    r = requests.get(drurl(ridut))  #r is the request sent to the website
    rhtml = r.text                  #Request html answer
    rsoup = BeautifulSoup(rhtml)    #Create soup for the individual request

    myd = {}                        #pivoted version
    myd['Request ID'] = str(ridut)
    #find ports
    #mt = rsoup.find(name='BIS232Request.JSONData.ExclusionExplanation.ShipmentQtyEstimate') #BIS232Request.JSONData.Ports[0] #
    #print('Ports:',mt)
    
    #the dpc creation could be inserted here if the title change base on the type of request

    #Acquiring p tags for Submission date and Public Status
    lit = rsoup.find_all('p')
    for t in lit:                                       #Loop the list of p tag 
        if t.text[:17] == 'Submission Date: ':          #exemple:Submission Date: 12/13/2019
            myd['Submission Date'] = t.text[17:]
        if t.text[:15] == 'Public Status: ':            #exemple:Public Status: Pending-Objection Window Open
            myd['Public Status'] = t.text[15:]
            
    lit = rsoup.find_all(fkeeptag)  #list of tags matching the selection function
    # <----error, the prodclass.key are still in the lit at this stage, should not, using the bskip to get rid of them
            
    for t in lit:                   #Loop the list of input tag
        bskip = False               #to skip '.Key' insertion in the result
        dat = t.attrs               #Dictionary of attibutes of that tag
        for ls in ('BIS232Request.', 'JSONData.'): #for readability sake replace BIS232Request. and JSONData. with '' 
            dat['name'] = dat['name'].replace(ls, '', 1)
        lname = dat['name']
        
        #set title to following for the given name
        #Product -> Product, MetalClass -> Metal Class, RequestingOrg.State -> Requesting Organization State, ID -> Request ID
        if   lname == 'Product':             dat['title'] = 'Product'
        elif lname == 'MetalClass':          dat['title'] = 'Metal Class'
        elif lname == 'RequestingOrg.State': dat['title'] = 'Requesting Organization State'
        elif lname == 'ID':                  
            dat['title'] = 'Request ID'
            dat['value'] = str(ridut)
        
        #If the tag is a ProductClassification[*].Value, call the title for the dpc dictionary 
        if lname[:22] == 'ProductClassification[' and lname[-7:] == '].Value':
            dat['title'] = dpc[lname]
        elif lname[:22] == 'ProductClassification[': 
            bskip = True
        
        #mistake in the title of the ToughnessShear[*].Value, replace Joule by Shear
        if lname[:15] == 'ToughnessShear[' and lname[-7:] == '].Value':
            dat['title'] = dat['title'].replace('Joule', 'Shear', 1)
        
        #mistake in the title of the bendadility % instead of mm
        if lname[:37] == 'ProductProcessing.Processes[0].Value.':
            dat['title'] = dat['title'].replace('Percentage', 'mm', 1)
        
        #fall back, use the name if no title is available
        if t.has_attr('title'):
            if dat['title'] == '': 
                dat['title'] = 'name: ' + str(dat['name'])
        else:
            dat['title'] = 'name: ' + str(dat['name'])
        
        #for textarea, there is no value, fill in with text
        if t.name == 'textarea': 
            dat['value'] = cleantext(t.text)
        
        #fill result
        if not(bskip):
            myd[dat['title']] = cleantext(dat['value'])
    
    #add links for attachment and decision memo
    lit = rsoup.find_all(fattach)
    bisdm = False #boolean is there a decision memo
    for t in lit:                               #Loop the list of link tag
        if t.has_attr('id'):   
            if t['id'] == 'DMAttachment':       #the section BIS Decision memo is not present when no DM 
                bisdm = True 
            #if t['id'] == 'ERAttachment':        
                #cannot find any difference between 22561 with ER and 44558 w/o ER
    
    myd['BIS Decision Memo'] = bisdm
    myd['Attachment'] = '' #dont know for the attachment
    
    mytbl = mytbl.append(myd, ignore_index=True) #column sorted by alphabetical order
    print('.', end='')              #progress bar one dot per request processed
print()                             #go to new line

#change the orders of the columns of the df
cols = list(myd.keys())
mytbl = mytbl[cols]
mytbl.set_index('Request ID', inplace=True)
print(mytbl.info())
mytbl.to_csv(myfile, mode='a', header=False) #save at the local directory of the jupyter notebook, 'a' for append
endt = datetime.datetime.now()
print('Run duration:', endt-startt, 'for', len(lr), 'requests')<jupyter_output><empty_output><jupyter_text>load the csv, get the Request ID column in a list, remove the already processed Request ID from lrappend to the csv the new lines every now and then.<jupyter_code>import requests                               #for querying the webite
from bs4 import BeautifulSoup                 #for reading and parsing the html
import pandas as pd                           #for dataframe manipulation
import datetime                               #for performance measurement
import re                                     #for regular expression

listrequest = list(range(35000, 40000))       #list of requests --- Pls adjust min and max ---  
myfile = 'myrequests35000-39999.csv'                     #file name saved at the local directory of the jupyter notebook
burl = 'https://232app.azurewebsites.net/'    #Base URL of the U.S. Department of Commerce, Section 232 website
eurl = 'Forms/ExclusionRequestItem/'          #URL extension for Details for an exclusion
drurl = (lambda i: burl + eurl + str(i))      #Lambda function providing the full URL for the details of a given Request ID

def finput(ltag):                             #function to select the input tag base on above analysis
    bn = True                                 #boolean for name, true keep, false discard 
    if ltag.has_attr('name'):                 #check if the name attribute exist for that tag
        lname = ltag['name']                  #save to local name for convenience
        bn = not(lname == '' or lname == '__RequestVerificationToken') #exclude those names
        #We will use regular expression, ^ match begin of string, $ match at end of string, (|) or
        #Exlude name in *ChemicalComposition[*].Key, ProductClassification[0].Key, *ProductDimensions[*].Key, *ToughnessJoules[*].Key, *ToughnessShear[*].Key, *ToughnessTemperature[*].Key, *ProductProcessing.ProcessesPercentage[*].Key, *ProductProcessing.Processes[*].Key 
        #lrex true match the exclusion pattern, false does not match the exclusion regex
        lrex = None != (re.search( r'^BIS232Request.JSONData.(ProductClassification|ChemicalComposition|ProductDimensions|ProductStrength|ProductProcessing.ProcessesPercentage|ProductProcessing.Processes|ToughnessJoules|ToughnessShear|ToughnessTemperature).*Key$', lname, flags=0))
        #the following keys are excluded by the regex but are actually required
        lexceptionkey = ('BIS232Request.JSONData.ChemicalComposition[27].Key', 'BIS232Request.JSONData.ProductStrength[2].Key', 'BIS232Request.JSONData.ToughnessTemperature[2].Key')
        if lname in lexceptionkey:
            lrex = False                      #update the lrex to include the exception for keys
    else:                                     #exclude tag without name attribute
        bn = False
    #include the textarea tags as they contains the comments    
    return (( ltag.has_attr('value') and ltag.name == 'input' ) or ltag.name == 'textarea') and bn and not(lrex)

def fprodclass(ltag):                         #function to select ProductClassification, used for dictionary of caption
    if ltag.has_attr('name'):                 #to avoid error need to check is the name attribute exist for that tag
        #lrex, true if it is a BIS232Request.JSONData.ProductClassification*Key
        lrex = None != (re.search( r'^BIS232Request.JSONData.ProductClassification.*Key$', ltag['name'], flags=0)) 
    else:                                     #exclude tag without name attribute
        lrex = False                
    return lrex

def fattach(ltag):                            #function to select link for attachment
    if ltag.has_attr('id'):                   #to avoid error need to check is the id attribute exist for that tag
        ba = (ltag['id'] == 'ERAttachment' or ltag['id'] == 'DMAttachment')              
    else:                                     #discard the tag without id
        ba = False 
    return ba

def cleantext(linput):                        #function to clean string and commments from special characters
    ls = str(linput)                          #convert input to a local string
    lcs = re.sub(r"[\r\n\t]", '', ls).strip() #use regex to replace carriage return, line break, tab..., strip extra spaces
    if ls == lcs:                             #if the above did not change the input
        output = linput                       #return the input to preserve the original format
    else:                                     #if the above changes the input 
        output = lcs                          #then return the cleaned string
    return output

startt = datetime.datetime.now()              #Performance management, start of run time

listrequest.sort()                            #Need to be sorted for substraction via list comprehension
try:
    mydf = pd.read_csv(myfile)                #load previously extracted csv
    listrdone = list(mydf['Request ID'])      #list of request id already in the file
    listrdone.sort()                          #Need to be sorted for substraction via list comprehension
    listrdone = list( dict.fromkeys(listrdone) ) #remove duplicate
    csvheader = False                         #as the file already exist, we will not add the header
except:                                       #if there is no such file
    listrdone = []                            #there is no request already done
    csvheader = True                          #as we create the file, we include the header

#using list comprehension, create list of request to extract, lr = listrequest - listrdone
lr = [rid for rid in listrequest if rid not in listrdone] 
if lr == []:                                  #error message if there is no request id to extract
    print('All Request ID are already downloaded')
else:
    print(len(lr), 'requests to extract')
    print(lr)
    print('Starting', end='')                 #Progress bar
    
#one time process to fill in the ProductClassification dictionary for title from the .Key
r = requests.get(drurl(22561))                #r is the request sent to the website, 22561 is as good as any other
rhtml = r.text                                #Request html answer
rsoup = BeautifulSoup(rhtml)                  #Create soup    
dpc = {}                                      #initiate dictionary for title for product classification value
lit = rsoup.find_all(fprodclass)              #list of tags matching the selection function
for t in lit:                                 #Loop the list of input tag for ProductClassification
    dat = t.attrs                             #Dictionary of attibutes of that tag
    for ls in ('BIS232Request.', 'JSONData.'):   #for readability sake replace BIS232Request. and JSONData. with '' 
        dat['name'] = dat['name'].replace(ls, '', 1)
    if   dat['value'] != '': ls = dat['value']   #most of the time, the caption is in value of ProductClassification[i].Key 
    elif dat['title'] != '': ls = dat['title']   #else it is in title of ProductClassification[i].Key 
    else                   : ls = ''
    #save the title for ProductClassification[i].Value in the dpc dictionary
    dpc[dat['name'].replace('.Key', '.Value', 1)] = ls

isave = 0                                     #i save every 10 requests
icount = 0                                    #i count for progress bar
mydf = pd.DataFrame()                         #reset the dataframe, so it only contains new entries

for ridut in lr:                              #loop on all the request to download from lr
    try:                                      #error management
        r = requests.get(drurl(ridut))        #r is the request sent to the website
    except:
        continue                              #resume at next iteration
    rhtml = r.text                            #Request html answer
    rsoup = BeautifulSoup(rhtml)              #Create soup for the individual request
    myd = {}                                  #initiate working dictionary for this request
    myd['Request ID'] = str(ridut)            #identify the request
    
    #fail retrieve ports BIS232Request.JSONData.Ports[0]
    #fail retrieve commercial name
    
    lit = rsoup.find_all('p')                 #Acquiring p tags for Submission date and Public Status
    for t in lit:                             #Loop the list of p tag, there are only a few
        if t.text[:17] == 'Submission Date: ':      #exemple:Submission Date: 12/13/2019
            myd['Submission Date'] = t.text[17:]    #save to request dictionary
        if t.text[:15] == 'Public Status: ':        #exemple:Public Status: Pending-Objection Window Open
            myd['Public Status'] = t.text[15:]      #save to request dictionary
            
    lit = rsoup.find_all(finput)              #list of tags matching the selection function          
    for t in lit:                             #Loop the list of input tag
        dat = t.attrs                         #Dictionary of attibutes of that tag
        
        for ls in ('BIS232Request.', 'JSONData.'):  #for readability sake replace BIS232Request. and JSONData. with '' 
            dat['name'] = dat['name'].replace(ls, '', 1)
        lname = dat['name']                   #save to local name for convenience
        
        #the prodclass.key are still in the lit at this stage, should not, using the bskip to get rid of them
        bskip = False                         #to skip '.Key' insertion in the result, default is false
        #If the tag is a ProductClassification[*].Value, call the title for the dpc dictionary 
        if lname[:22] == 'ProductClassification[' and lname[-7:] == '].Value':
            dat['title'] = dpc[lname]         #dictionary for product classification provide the title for that value
        elif lname[:22] == 'ProductClassification[': #the current tag is a classificatin key, not value
            bskip = True                      #skip insertion in result 
        
        #force the title for the following names as they are exceptions to the global pattern
        if   lname == 'Product':             dat['title'] = 'Product'
        elif lname == 'MetalClass':          dat['title'] = 'Metal Class'
        elif lname == 'RequestingOrg.State': dat['title'] = 'Requesting Organization State'
        elif lname == 'ID':                  
            dat['title'] = 'Request ID'
            dat['value'] = str(ridut)
        
        #fix mistake in the title of the ToughnessShear[*].Value, replace Joule with Shear
        if lname[:15] == 'ToughnessShear[' and lname[-7:] == '].Value':
            dat['title'] = dat['title'].replace('Joule', 'Shear', 1)
        
        #fix mistake in the title of the bendadility %, replace with mm
        if lname[:37] == 'ProductProcessing.Processes[0].Value.':
            dat['title'] = dat['title'].replace('Percentage', 'mm', 1)
        
        #fall back, use the name if no title is available, there should not be any
        if t.has_attr('title'):
            if dat['title'] == '': 
                dat['title'] = 'name: ' + str(dat['name'])
        else:
            dat['title'] = 'name: ' + str(dat['name'])
        
        #for textarea, there is no value attribute, the comment is found with text
        if t.name == 'textarea': 
            dat['value'] = cleantext(t.text)
        
        #save the tag to the request dictionary
        if not(bskip):
            myd[dat['title']] = cleantext(dat['value'])
    
    
    lit = rsoup.find_all(fattach)             #acquire tag for attachment and decision memo
    bisdm = False                             #boolean is there a decision memo
    for t in lit:                             #Loop the list of attachment tags
        if t.has_attr('id'):   
            if t['id'] == 'DMAttachment':     #the section BIS Decision memo is not present when no DM 
                bisdm = True                  #so if there is a section, there is a DM
            #if t['id'] == 'ERAttachment':        
                #cannot find any difference between 22561 with ER and 44558 w/o ER
    
    myd['BIS Decision Memo'] = bisdm          #save boolean for DM to request dictionary
    myd['Attachment'] = ''                    #dont know for the attachment
    
    mydf = mydf.append(myd, ignore_index=True) #Add the request dictionary to the dataframe
    isave = isave + 1                         #increase count for next save
    if isave >= 10:                           #it double processing time to save at each iteration, therefore once/10 times 
        #by default column sorted by alphabetical order
        print('indesname:', mydf.index.name)
        if mydf.index.name != 'Request ID':       #if not already done, set request as the index of the df
            mydf.set_index('Request ID', inplace=True)  
            cols = list(myd.keys())[1:]           #request dictionary has them in the correct order
            mydf = mydf[cols]                     #change the orders of the columns of the df
        try:
            mydf.to_csv(myfile, mode='a', header=csvheader, low_memory=False) #save at the end of target file, 'a' for append
            if csvheader: csvheader = False   #only include header once 
            mydf = pd.DataFrame()             #reset the dataframe, so it only contains new entries 
            endt = datetime.datetime.now()    #to calculate and display run time
            icount = icount + isave           #increase total count of request processed
            print('\r', end='')               #clear progress bar
            print(str(icount), 'requests saved in', endt-startt, end='') #update progress bar
        except:         
            print(mydf)
            print('fail to save', end='')     #update progress bar, mydf is likely not cleared, try at next loop
        isave = 0                         #reset save counter
        
if not(mydf.empty):                           #Save for the finals entries
    #by default column sorted by alphabetical order
    if mydf.index.name != 'Request ID':       #if not already done, set request as the index of the df
        mydf.set_index('Request ID', inplace=True)  
        cols = list(myd.keys())               #request dictionary has them in the correct order
        mydf = mydf[cols]                     #change the orders of the columns of the df
    mydf.to_csv(myfile, mode='a', header=csvheader, low_memory=False) #save at the end of target file, 'a' for append
    endt = datetime.datetime.now()            #to calculate and display run time
    print('\r', end='')                       #clear progress bar
    print(str(icount+isave), 'requests saved in', endt-startt, end='') #update progress bar
import requests                               #for querying the webite
from bs4 import BeautifulSoup                 #for reading and parsing the html
import pandas as pd                           #for dataframe manipulation
import datetime                               #for performance measurement
import re                                     #for regular expression

listrequest = list(range(35000, 40000))       #list of requests --- Pls adjust min and max ---  
myfile = 'myrequests35000-39999.csv'                     #file name saved at the local directory of the jupyter notebook
burl = 'https://232app.azurewebsites.net/'    #Base URL of the U.S. Department of Commerce, Section 232 website
eurl = 'Forms/ExclusionRequestItem/'          #URL extension for Details for an exclusion
drurl = (lambda i: burl + eurl + str(i))      #Lambda function providing the full URL for the details of a given Request ID

def fprodclass(ltag):                         #function to select ProductClassification, used for dictionary of caption
    if ltag.has_attr('name'):                 #to avoid error need to check is the name attribute exist for that tag
        #lrex, true if it is a BIS232Request.JSONData.ProductClassification*Key
        lrex = None != (re.search( r'^BIS232Request.JSONData.ProductClassification.*Key$', ltag['name'], flags=0)) 
    else:                                     #exclude tag without name attribute
        lrex = False                
    return lrex

def fattach(ltag):                            #function to select link for attachment
    if ltag.has_attr('type'):                   #to avoid error need to check is the id attribute exist for that tag
        ba = ( ltag['type'] == 'hidden' )              
    else:                                     #discard the tag without id
        ba = False 
    return ba

mydf = pd.DataFrame()                         #reset the dataframe, so it only contains new entries

for ridut in ['25663']:                              #loop on all the request to download from lr
    try:                                      #error management
        r = requests.get(drurl(ridut))        #r is the request sent to the website
    except:
        continue                              #resume at next iteration
    rhtml = r.text                            #Request html answer
    rsoup = BeautifulSoup(rhtml)              #Create soup for the individual request
    myd = {}                                  #initiate working dictionary for this request
    myd['Request ID'] = str(ridut)            #identify the request
    
    lit = rsoup.find_all('p')                 #Acquiring p tags for Submission date and Public Status
    for t in lit:                             #Loop the list of p tag, there are only a few
        print('Name: ', t.name, 'Text:', t.text, 'Tag:',t)
        if t.text[:17] == 'Submission Date: ':      #exemple:Submission Date: 12/13/2019
            myd['Submission Date'] = t.text[17:]    #save to request dictionary
        if t.text[:15] == 'Public Status: ':        #exemple:Public Status: Pending-Objection Window Open
            myd['Public Status'] = t.text[15:]      #save to request dictionary    
    
    lit = rsoup.find_all(fattach)             #acquire tag for attachment and decision memo
    bisdm = False                             #boolean is there a decision memo
    for t in lit:                             #Loop the list of attachment tags
        print('Name: ', t.name, 'id:', t['id'], 'Tag:',t)
        if t.has_attr('id'):   
            if t['id'] == 'DMAttachment':     #the section BIS Decision memo is not present when no DM 
                bisdm = True                  #so if there is a section, there is a DM
            #if t['id'] == 'ERAttachment':        
                #cannot find any difference between 22561 with ER and 44558 w/o ER
    
   
t.attrs
t.text
t.name
rsoup.find_all('title')[0]
rsoup<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09894798553887242, 'fraction_numerical': 0.012637201576393186, 'mean_word_length': 2.384051581615783, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 71, 'https://': 14, 'lorem ipsum': 0, 'www.': 9, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 65, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '147843', 'license': 'Unlicense', 'n_tokens_mistral': 17651, 'n_tokens_neox': 16677, 'n_words': 6301, 'path': 'Section232.ipynb', 'repo_name': 'Raphael9999/scraping-section-232', 'size': '61450'}