<jupyter_start><jupyter_text>How random is `r/random`?There's a limit of 0.5 req/s (1 request every 2 seconds) What a good response looks like (status code 302)```$ curl https://www.reddit.com/r/random   302 Found    302 Found  The resource was found at https://www.reddit.com/r/Amd/?utm_campaign=redirect&amp;utm_medium=desktop&amp;utm_source=reddit&amp;utm_name=random_subreddit;you should be redirected automatically. ``` What a bad response looks like (status code 429)```$ curl https://www.reddit.com/r/random      Too Many Requests          body {          font: small verdana, arial, helvetica, sans-serif;          width: 600px;          margin: 0 auto;      }      h1 {          height: 40px;          background: transparent url(//www.redditstatic.com/reddit.com.header.png) no-repeat scroll top right;      }            whoa there, pardner!we're sorry, but you appear to be a bot and we've seen too many requestsfrom you lately. we enforce a hard speed limit on requests that appear to comefrom bots to prevent abuse.if you are not a bot but are spoofing one via your browser's user agentstring: please change your user agent string to avoid seeing this messageagain.please wait 4 second(s) and try again.    as a reminder to developers, we recommend that clients make no    more than one    request every two seconds to avoid seeing this message.  ``` What happensGET --> 302 (redirect) --> 200 (subreddit)I only want the name of the subreddit, so I don't need to follow the redirect.<jupyter_code>import pandas as pd
import requests

from time import sleep
from tqdm import tqdm
from random import random
def parse_http(req):
  """
  Returns the name of the subreddit from a request

  If the status code isn't 302, returns "Error"
  """
  if req.status_code != 302:
    return "Error"
  
  start_idx = req.text.index('/r/') + len('/r/')
  end_idx = req.text.index('?utm_campaign=redirect') - 1

  return req.text[start_idx:end_idx]

  
sites = []
codes = []

headers = {
    'User-Agent': 'Mozilla/5.0'
}

# Works for 10, 100 @ 3 seconds / request
# Works for 10      @ 2 seconds / request
for _ in tqdm(range(1000), ascii=True):
  # Might have to mess with the User-Agent to look less like a bot
  # https://evanhahn.com/python-requests-library-useragent
  # Yeah the User-Agent says it's coming from python requests
  # Changing it fixed everything
  r = requests.get('https://www.reddit.com/r/random', 
                   headers=headers,
                   allow_redirects=False)
  if r.status_code == 429:
    print("Got rate limit error")
  sites.append(parse_http(r))
  codes.append(r.status_code)
  # Jitter the sleep a bit to throw off bot detection
  sleep(2 + random())


#[print(code, site) for code, site in zip(codes, sites)];
for row in list(zip(codes, sites))[-10:]:
  print(row[0], row[1])
df = pd.DataFrame(list(zip(sites, codes)), columns=['subreddit', 'response_code'])
df.head()
df.info()
from time import time
fname = 'reddit_randomness_' + str(int(time())) + '.csv'
df.to_csv(fname,index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10944135903299575, 'fraction_numerical': 0.019601437438745508, 'mean_word_length': 3.543026706231454, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 5, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '7176', 'license': 'MIT', 'n_tokens_mistral': 1004, 'n_tokens_neox': 940, 'n_words': 339, 'path': 'reddit_randomness_scrape.ipynb', 'repo_name': 'dwisdom0/reddit-randomness-notebooks', 'size': '3061'}