I've written a script to harvest Name and Phone number from yellowpage Canada under pizza category. The web page doesn't show its full content until scrolled downmost. In my crawler's every cycle, it fetches 40 records. I twitched a little in my loop which I've learnt lately to get the full content. Hope I did it the way it should be.import requestsfrom lxml import htmlBase_url=https://www.yellowpages.ca/search/si/{0}/pizza/Torontowith requests.session() as session:    page_num=1    while True:        response = session.get(Base_url.format(page_num))        if response.status_code==404:            break        tree = html.fromstring(response.text)        for titles in tree.xpath(//div[@itemprop='itemListElement']):            try:                title = titles.xpath(.//h3[@itemprop='name']/a/text())[0]            except IndexError:                title=            try:                phone= titles.xpath(.//h4[@itemprop='telephone']/text())[0]            except IndexError:                phone=            print(title,phone)        page_num+=1
Parser for a lazyloading yellowpage
python;python 3.x;web scraping
Applying PEP8 recommendations about naming and formatting,as well as wrapping the main logic in a main function and calling from within a if __name__ == '__main__': guard,the code would become easier to read and possible to import and test:import requestsfrom lxml import htmlurl_format = https://www.yellowpages.ca/search/si/{0}/pizza/Torontodef main():    with requests.session() as session:        page_num = 1        while True:            response = session.get(url_format.format(page_num))            if response.status_code == 404:                break            tree = html.fromstring(response.text)            for titles in tree.xpath(//div[@itemprop='itemListElement']):                try:                    title = titles.xpath(.//h3[@itemprop='name']/a/text())[0]                except IndexError:                    title =                 try:                    phone = titles.xpath(.//h4[@itemprop='telephone']/text())[0]                except IndexError:                    phone =                 print(title, phone)            page_num += 1if __name__ == '__main__':    main()I'm not a huge fan of using exceptions in situations that are not really exceptions, like here:try:    title = titles.xpath(.//h3[@itemprop='name']/a/text())[0]except IndexError:    title = What if a titles.xpath(...) call would raise an IndexError due to a bug? It would be inadvertently caught and unnoticed.Catching IndexError here is an indirect way of the real intention,which is getting the first title if it exists.It would be better to write the code in a way that expresses the intention directly, for example:for item in tree.xpath(//div[@itemprop='itemListElement']):    title = get_first_or_empty(item, .//h3[@itemprop='name']/a/text())    phone = get_first_or_empty(item, .//h4[@itemprop='telephone']/text())    print(title, phone)Where get_first_or_empty is:def get_first_or_empty(item, xpath):    matches = item.xpath(xpath)    if matches:        return matches[0]    return 