The following spider will grab some driver links, OS version, and the name.All the info is in a table class, but some pages might be a little different in the location and Number of cells in each row.I can't post the HTML page, but the main problem I'm facing now is that sometimes the number of the row I need is in cell 0 or 1. I know it should be simple to make the spider check but I just can't think of a way to make it into the current spider.import bs4def bkc_spider(path, start, stop):    BKC Spider - grabs driver names, links and OS version from BKC releases    ver_lst = ['Win7', 'Win8.1', 'Win10', 'WinT', 'WinN']    file = open(path, 'rb')    drivers = {}    rng_lst = [str(x) for x in range(start, stop)]    soup = bs4.BeautifulSoup(file)    table = soup.findAll('table')[0]    for row in table.findAll('tr'):        cells = row.findAll('td')        if len(cells) > 4:            cell_num = cells[1].get_text(strip=True)            if any(cell_num == n for n in rng_lst):                drv_name = cells[2].get_text(strip=True)                drivers[drv_name] = {'links': []}                for pTag in cells[4].findAll('p'):                    try:                        link = pTag.findNext('a')                        link_os = pTag.find('span', {'class': 'MsoHyperlink'}).span.get_text(strip=True)                        if any(link_os == ver for ver in ver_lst):                            drivers[drv_name]['links'].append(link['href'] +  |  + link_os)                        else:                            drivers[drv_name]['links'].append(link['href'])                        continue                    except AttributeError:                        drivers[drv_name]['links'].append(link['href'])                        continue    for x in drivers:        print(x)        for y in drivers[x]:            print(drivers[x][y])        print('\n')    return drivershtml_file = rBKC.htmbkc_spider(html_file, 6, 31)
BeautifulSoup web spider for driver links
python;python 3.x;web scraping;beautifulsoup