r = requests.get(url, headers={'User-Agent': '...'})
is_html = content_type_header.split(';', 1)[0].lower().startswith('text/html')
Correctly detect encoding without any guessing when using Beautiful Soup