Q:

Getting content of specific cells from table

I am trying to parse patent data from, e.g., this page. Final output should be a CSV file with one row per patent, and (assignee, family ID, and filing date) on each row. 
I am using BeautifulSoup and I am fine retrieving most of the information and writing the CSV file. 
My problem is that I noticed that the structure of the tables changes over time; not all fields are there all the time. E.g. In the example given, there is no family ID. 
Hence, I cannot assign each cell to a specific variable (like here). Depending on the number of fields reported, the number of rows/columns also changes. 
I would like to write the code so that it is flexible enough to do the following:
If header == "Assignee", then get text from that cell. Otherwise, leave it blank. 
If header == "Family ID", then get text from that cell. Otherwise, leave it blank. 
So that the final output would be similar to:
Assignee, Family ID, filing date
"Potomac Aviation",         , "June 11, 2002"
"Anonymous Co", 40432687, "June 5, 2016"
etc. 

This code is what got me closest but I'm still far from where I want to end up.  
fourth_table=table.find_next("table")
header_1 = fourth_table.find('th')
if header_1.get_text() == "Inventors:":
   inventors=fourth_table.find('td').get_text()

header1=fourth_table.th
header_2 = header1.find_next('th')
cell1=fourth_table.td
cell2=cell1.find_next('td')
if header_2.get_text() == "Applicant:":
   applicant= cell2.find('td').get_text()

It is clearly very wordy; I will try to make code more efficient once I make sure I understand how every single bit works. 
Edit: This is an alternative that I think gets me much closer. However, although it works all right for "Assignee", python returns "None" for the print(family_id) line. I've checked spelling.
 fourth_table=table.find_next("table")
 assignee=fourth_table.find(text="Assignee:").findNext('td').get_text().replace("\n","").strip()
#family_id=fourth_table.find(text="Family ID:").findNext('td').get_text().replace("\n","").strip()
 family_id=fourth_table.find(text="Family ID:")
 print(family_id)

Apologies if I a missing something obvious. TIA!

A:

It is a bit more involved, you really need to use a regex to find the rows you want as there are line breaks in the text so you cannot just specify text="Family ID": etc.., this will get all the links from the site and write the three rows of interest if available, if the row is nto there the csv will have an empty entry for the missing data:
import csv
import re

def get_nxt_link(soup):
    nxt = soup.select_one("img[src=/netaicon/PTO/nextdoc.gif]")
    if nxt:
        return urljoin(base, nxt.parent["href"])
    return False

start = 'http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&co1=AND&d=PTXT&s1=boston&s2=2005$.PD.&OS=boston%20AND%20ISD/2005&RS=boston%20AND%20ISD/2005'

base = "http://patft.uspto.gov/"
def get_text(soup, txt):
    for t in txt:
        _tag = soup.find("th", text=t)
        yield _tag.parent.td.text.replace("\n", "") if _tag else ""

with open("out.csv", "w") as o:
    wr = csv.writer(o)
    soup = BeautifulSoup(requests.get(start).content, "lxml")
    table = soup.find("th", text="Assignee:").find_previous("table")
    nxt = get_nxt_link(soup)
    wr.writerow(["Assignee", "Family ID", "filing date"])
    text = [re.compile("Assignee:"), re.compile("Family ID:"), re.compile("Filed:")]
    wr.writerow(tuple(get_text(table, text)))
    while nxt:
        soup = BeautifulSoup(requests.get(nxt).content, "lxml")
        table = soup.find("th", text="Assignee:").find_previous("table")
        wr.writerow(tuple(get_text(table, text)))
        nxt = get_nxt_link(soup)

Running on the first few urls we get:
Assignee,Family ID,filing date
"Potomac Aviation Technology Corp. (Boston, MA)",,"June 11, 2002"
"Microsoft Corp. (Redmond, WA)",33563549,"December 31, 1998"
"Teradyne, Inc. (Boston, MA)",32029308,"September 27, 2002"
"First Data Corporation (Greenwood Village, CO)",22696181,"December 5, 2001"
"Micron Technology, Inc. (Boise, ID)",35482789,"July 8, 2003"
"Digital River, Inc. (Eden Prairie, MN)",38226542,"December 11, 2000"
"Oracle International Corporation (Redwood Shores, CA)",35482753,"October 1, 2002"
"The United States of America as represented by the Secretary of the Navy (Washington, DC)",35482734,"October 6, 2003"
Garmin Ltd. (KY),21836040,"November 19, 2004"
"Lucent Technologies Inc. (Murray Hill, NJ)",25439780,"July 30, 2001"
"The Chamberlain Group, Inc. (Elmhurst, IL)",23942291,"October 17, 2001"
"Inplane Photonics, Inc. (South Plainfield, NJ)",32824042,"February 7, 2003"
"Motorola, Inc (Horsham, PA)",22314958,"November 17, 2000"
"Xerox Corporation (Stamford, CT)",35482631,"July 7, 2004"
"General Hospital Corporation (Boston, MA)",35480219,"October 16, 2002"

