Q:

Content of a page changing at each request

I'm trying to scrape an HTML page from a website that stores various information about french companies. However each time I send a request via pd.read_html (to extract only the tables), I get a different result.
To illustrate my point, you'll find some code to reproduce:
result = []
for i in range(0,10):
    result.extend(pd.read_html('https://www.societe.com/societe/eram-388583239.html', encoding='utf-8',attrs={'id':'rensjur'}))
    time.sleep(5)
print(result)

I expect to get the same table as when I open the link in my browser.

A:

After playing around with it for a few minutes I found changing the user agent helps. My guess is that the website obfuscates the real data when it detects a user agent that is not a web browser.
I am sure there is a more elegant way of doing this, but here is the code I used that pulls the same data every time:
import pandas as pd
import time
import urllib.request as request

results = []
for i in range(0,10):
    url = 'https://www.societe.com/societe/eram-388583239.html'
    opener = request.build_opener()
    opener.addheaders = [('User-agent', 'Mozilla/5.0')]
    response = opener.open(url)
    result = pd.read_html(response.read(), encoding='utf-8',attrs={'id':'rensjur'})
    print(result)
    results.extend(result)
    time.sleep(5)
print(results)

