I've written some script in python using lxml library which is capable of dealing with a webpage with complicatedly layed-out next page links. My scraper is able to parse all the next page links without going to the next page and hardcoding any number to the lastmost link and scrape the required fields flawlessly. Although I tried to do the whole thing specklessly, any improvement on this will be highly appreciable. Here is what I've tried with:import requestsfrom lxml import htmlmain_link = https://www.yify-torrent.org/search/1080p/base_link = https://www.yify-torrent.orgdef get_links(item_link):    response = requests.get(item_link).text    tree = html.fromstring(response)    last_page = tree.cssselect('div.pager a:contains(Last)')[0].attrib[href].split(/)[-2].replace('t-','')    links = [item_link +t-{0}/.format(page) for page in range(int(last_page) +1)]    for link in links:        process_docs(link)def process_docs(nextpage_link):    response = requests.get(nextpage_link).text    tree = html.fromstring(response)    for items in tree.cssselect('div.mv'):        name = items.cssselect('h3 a')[0].text        try:            genre = items.xpath('.//div[@class=mdif]//li[b=Genre:]/text()')[0] # I don't know If css selector could do this        except IndexError:            genre =         try:            imd = items.xpath('.//div[@class=mdif]//li[b=Rating:]/text()')[0] # Used xpath in lieu of css selector        except IndexError:            imd =         print(name, genre, imd)get_links(main_link)
Scraping required fields exhausting a full website
python;performance;python 3.x;web scraping;lxml
This is quite clean and understandable.I would pinpoint only a couple of things:to avoid code repetition and to generalize getting search result properties, what if you would collect all the properties (genre, quality, size etc) into a dictionary:properties = {    label.text: label.xpath(following-sibling::text())[0]    for label in items.xpath(.//div[@class=mdif]//li/b)}As a bonus, you would not need the try/except part at all.as usual, you may reuse a session to boost your HTTP requests part performanceSome other minor notes:it looks like that you make an extra request to the page number 0, adjust your range if this is the caseitems variable name is probably a bit misleading, at the very least it should be item since you are iterating over search result items one by oneput get_links() call to under the if __name__ == '__main__' to avoid the code being executed if the module is ever importedI would also return or yield from the get_links and process_docs functions instead of printing things out