<jupyter_start><jupyter_text><jupyter_code>from IPython.display import HTML, Latex
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from ipywidgets import IntProgress<jupyter_output><empty_output><jupyter_text>What's the Optimal Size of the House?---The House of Representatives has stood at 435 members for nearly 100 years as the population of the country has tripled. How much larger should it be to minimize the difference in how many people each Member represents?To find out, let's run a computational experiment. The proof-of-concept for these functions, tested against the official Congressional tally, is available in the `Apportionment.ipynb` file in this repo or on [Google CoLab](https://colab.research.google.com/drive/1bBHxmUGtqs2kIH-myie5_bOZgFaT-hPT?usp=sharing).  Loading the DataCongress reapportions the House every 10 years after the decennial Census figures are released. The [GitHub repo for this demo](https://github.com/TimeMagazineLabs/CongressionalApportionment) has data files from Census.gov for the 2010 and 2020 decennial Census counts as well as the official apportionments so that we can check our work. Here's what the data looks like:<jupyter_code>dataByDecade = {}

for year in range(1910, 2021, 10):
  data = pd.read_csv('https://raw.githubusercontent.com/TimeMagazineLabs/CongressionalApportionment/main/data/apportionment_%s.csv' % year, dtype={'State': 'string', 'Abbr': 'string', 'Reps': 'Int64'})
  data['Per_Rep'] = data['Population'] / data['Reps']
  dataByDecade['y' + str(year)] = data
dataByDecade['y2020'].head()<jupyter_output><empty_output><jupyter_text>As we can easily see, the number of people represented by each Member of Congress has ballooned drastically in the past 100 years<jupyter_code>reps1920 = dataByDecade['y1920'][dataByDecade['y1920']['Reps'] > 0]
reps2020 = dataByDecade['y2020'][dataByDecade['y2020']['Reps'] > 0]

print('In 1920, the %s House Members for %d states represented between %s and %s people' % (
  sum(reps1920['Reps']), reps1920.shape[0], '{0:,}'.format(int(min(reps1920['Per_Rep']))), '{0:,}'.format(int(max(reps1920['Per_Rep'])))
))

print('In 2020, the %s House Members for %d states represented between %s and %s people' % (
  sum(reps2020['Reps']), reps2020.shape[0], '{0:,}'.format(int(min(reps2020['Per_Rep']))), '{0:,}'.format(int(max(reps2020['Per_Rep'])))
))<jupyter_output>In 1920, the 435 House Members for 48 states represented between 77,407 and 360,350 people
In 2020, the 435 House Members for 50 states represented between 542,703 and 990,837 people<jupyter_text>How Apportionment Works: The Equal Proportions MethodSince 1940, Congress [has used the "Equal Proportions Method"](https://www.census.gov/topics/public-sector/congressional-apportionment/about/computing.html) to apportion the 435 House seats to each state.First, per the Constitution, every state gets one seat. For the remaining 385, each state's "priority" is measured by a simple formula that divides its population, as of the most recent decennial Census, by the square root of the product of the number of seats it currently has and that number plus 1 (the [geometric mean](https://mathworld.wolfram.com/GeometricMean.html)):\begin{equation}priority = \frac{P}{\sqrt{n*(n+1)}}\end{equation}For each seat, one at a time, the state with the highest priority is awarded that seat. Then its priority is recalculated and it moves further back in the line. Let's write a Python function to calculate the priority of a state, representated as a row in the table (aka "DataFrame") above. (We'll be adding a `RepsCalculated` column to compare to the official tally.)<jupyter_code>def EqualProportionsMethod(st):
  reps = st["RepsCalculated"]
  priority = st["Population"] / np.sqrt(reps * (reps + 1))
  return priority<jupyter_output><empty_output><jupyter_text>Recalculating the apportionmentsAs covered in detail in the previous Notebook, we can then write a function to apportion any number of seats we like:<jupyter_code>def addNextSeat(df, ORDER=[]):
  indexNext = df['Priority'].idxmax() # The index of the row with the highest priority
  # Add a seat to the state in that row
  df.loc[indexNext,'RepsCalculated'] += 1
  # Recompute the priority for this state
  df.loc[indexNext,'Priority'] = EqualProportionsMethod(df.loc[indexNext])
  # Add the state to the ORDER list
  ORDER.append(df.loc[indexNext,'Abbr'])

def calculateApportionment(popData=dataByDecade['y2020'], ignoreStates=['District of Columbia', 'Puerto Rico'], TOTAL_SEATS=435):
  ORDER_SEATS = []

  data_filtered = popData[~popData['State'].isin(ignoreStates)]
  data_filtered.reset_index(drop=True)

  test_apportionment = data_filtered.copy()
  test_apportionment['RepsCalculated'] = 1
  test_apportionment["Priority"] = test_apportionment.apply(EqualProportionsMethod, 1)

  SEATS_LEFT = TOTAL_SEATS - test_apportionment.shape[0] # This is the number of rows, which will account for any hypothetical states we add
  while SEATS_LEFT > 0:
    addNextSeat(test_apportionment, ORDER_SEATS)
    SEATS_LEFT -= 1

  test_apportionment['Per_RepCalculated'] = test_apportionment['Population'] / test_apportionment['RepsCalculated']

  return test_apportionment, ORDER_SEATS 
test_2020, order_2020 = calculateApportionment()
test_2020.head()<jupyter_output><empty_output><jupyter_text>Just to reiterate, this matches the official apportionment, which the final seat going to Minnesota:<jupyter_code>error = np.abs(test_2020['Reps'] - test_2020['RepsCalculated'])
print('Total difference from actual apportionment: %d' % max(error))
print('Final seat went to %s' % order_2020[-1])<jupyter_output>Total difference from actual apportionment: 0
Final seat went to MN<jupyter_text>Great, it works! But how well does the current apportionment represent every American equally? Since we have the number of residents per Representatives for each state, one way to measure this is the "Coefficient of Variation," a simple statistical method of normalizing the standard deviation by dividing it by the average value:<jupyter_code>def coeff_of_variance(vals):
  return np.std(vals) / np.mean(vals)

coeff_of_variance(test_2020['Per_Rep'])<jupyter_output><empty_output><jupyter_text>Okay, so the current variation is about 0.1, which could be better with more seats. Since we made the number of seats a variable, let's see if it improves if we add 25 seats:<jupyter_code>test_460, order_460 = calculateApportionment(TOTAL_SEATS=460)
print(order_460[-1])
coeff_of_variance(test_460['Per_RepCalculated'])<jupyter_output>MO<jupyter_text>Yep, a House of Representatives of 460 seats would have a lower variance in how equally each American is represented. Is there an optimal number? Let's just try every value from 435 to 1,305, three times that.While we could run the `calculateApportionment` function 970 times, this will take around 10 minutes--believe me, I tried! But that would be wasteful, because each iteration is recomputing the same order of seats and just adding another. So we can save a lot of time by running this once, for 1,035 seats, and then apply the order to however many seats we want, like so:<jupyter_code>apportionment_1305, ORDER_1305 = calculateApportionment(TOTAL_SEATS=435*3)<jupyter_output><empty_output><jupyter_text>Now we just need a function to take a subset of that grand order, up to however many seats we want, and dole them out:<jupyter_code>def computeSeats(popData=dataByDecade['y2020'], ignoreStates=['District of Columbia', 'Puerto Rico'], TOTAL_SEATS=435):
  # Remember to subtract 50 from each state since we apportion one automatically
  orderX = ORDER_1305[:TOTAL_SEATS - 50]

  # Here's a little trick: Assuming that every state gets at least one seat in the 1,305 scenario, we can use `set` to reduce to unique abbreviations:
  abbrs = set(ORDER_1305)
  # print(len(abbrs)) # will be 50

  # Here's another neat trick for tallying the number of duplicate items in a list
  # h/t https://stackoverflow.com/questions/42317134/how-to-get-a-tally-of-each-letter-in-a-string
  # This will assign 1 to each state even if it never shows up in the order (e.g. AK in the 435 scenario)
  seats = { st: 1 + orderX.count(st) for st in abbrs }

  # convert the dictionary to a list of tuples
  seats = [ (st,reps) for st,reps in seats.items() ]

  # then convert it to a dataframe
  df = pd.DataFrame(seats, columns = ['Abbr', 'RepsCalculated'])
  df = df.astype({ 'Abbr': 'string', 'RepsCalculated': 'Int64' })

  # Again make a copy of the population dataframe, then join them
  data_filtered = popData[~popData['State'].isin(ignoreStates)]
  data_filtered.reset_index(drop=True)

  apportionmentX = data_filtered.copy()
  apportionmentX = apportionmentX.set_index("Abbr").join(df.set_index("Abbr"))
  apportionmentX['Per_RepCalculated'] = apportionmentX['Population'] / apportionmentX['RepsCalculated']

  # print(len(orderX), len(ORDER_1035))
  return apportionmentX, orderX<jupyter_output><empty_output><jupyter_text>Again, let's run a sanity test to make sure this method computes correctly for the real apportionment<jupyter_code>test_2020B, order_2020B = computeSeats(TOTAL_SEATS=435)
test_2020B.head()

error = np.abs(test_2020B['Reps'] - test_2020B['RepsCalculated'])
print('Total difference from actual apportionment: %d' % max(error))
print('Final seat went to %s' % order_2020B[-1])<jupyter_output>Total difference from actual apportionment: 0
Final seat went to MN<jupyter_text>Once again, looks good! Let's try it for all sizes:<jupyter_code>SIZES = list(range(435,435*3))
TRIALS = []

progressBar = IntProgress(min=435, max=435*3) # instantiate the bar
display(progressBar) # display the bar

for s in SIZES:
  apportionmentN, orderN = computeSeats(TOTAL_SEATS=s)
  TRIALS.append({
      'size': s,
      'apportionment': apportionmentN,
      'order': orderN,
      'score': coeff_of_variance(apportionmentN['Per_RepCalculated'])
  })
  progressBar.value += 1<jupyter_output><empty_output><jupyter_text>Trust me, that went WAY faster than reculating the apportionment from scratch every time. How did we do?<jupyter_code>SCORES = [x['score'] for x in TRIALS]

fig = plt.figure(figsize=(12, 8), frameon=True);
plt.title("Variance by Size of House")
plt.xlabel('Number of seats')
plt.ylabel('Covariance in People / Representative')
ax = fig.gca();
ax.plot(SIZES, SCORES, 'o', markersize=2)<jupyter_output><empty_output><jupyter_text>This is curious! While the variation, or "score," goes down generally as the size of the House increases, as we'd expect, there are clearly ideal values, as we saw last time [I ran this experiment](https://time.com/5423623/house-representatives-number-seats/). Let's identify them:<jupyter_code>results = pd.DataFrame({
    'size': SIZES,
    'score': SCORES
})
results['delta'] = results.score -  results.score.shift(1)
idealValues = results.sort_values('delta').head(15).sort_values('score')
idealValues
dropPoints = list(idealValues['size'][1:])
dropValues = list(idealValues['score'][1:])

fig = plt.figure(figsize=(12, 8), frameon=True);
plt.title("Variance by Size of House")
plt.xlabel('Number of seats')
plt.ylabel('Covariance in People / Representative')
ax = fig.gca();
ax.plot(SIZES, SCORES, 'o', markersize=2)
ax.plot(dropPoints, dropValues, 'o', markersize=8)<jupyter_output><empty_output><jupyter_text>Let's see if there are any interesting properties, like the smallest number of reps per state, the minimum, maximum and mean number of people per representative, and a few other items:<jupyter_code>testPoints = [x for x in TRIALS if x['size'] in dropPoints]
SCENARIOS = []

for trial in testPoints:
  vals = trial['apportionment']['Per_RepCalculated']
  seats = trial['apportionment']['RepsCalculated'].tolist()

  SCENARIOS.append({
    'size': trial['size'],
    'score': trial['score'],
    'perRepMin': min(vals),
    'perRepMax': max(vals),
    'perRepMean': np.mean(vals),
    'minSeats': min(seats),
    'minSeatCount': seats.count(min(seats)),
    'lastSeat': trial['order'][-1]
  })
df = pd.DataFrame.from_dict(SCENARIOS)
df.sort_values('size')<jupyter_output><empty_output><jupyter_text>As we see, there are many optimal values that get increasingly more equal as the size grows. **The value 917 has a dramatic drop to a covariance below .05**, and increasing the size beyong that doesn't improve the score that much.But if that's too big a House for your liking, even 473 is a huge improvement.<jupyter_code>apportionment917, _ = calculateApportionment(TOTAL_SEATS=917)
apportionment917[['State', 'Population', 'Reps', 'Per_Rep', 'RepsCalculated', 	'Per_RepCalculated']]
apportionment473, _ = calculateApportionment(TOTAL_SEATS=473)
apportionment473[['State', 'Population', 'Reps', 'Per_Rep', 'RepsCalculated', 	'Per_RepCalculated']]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10948848106208513, 'fraction_numerical': 0.03076923076923077, 'mean_word_length': 6.0517621145374445, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 64, 'https://': 7, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '310125', 'license': 'MIT', 'n_tokens_mistral': 4526, 'n_tokens_neox': 4003, 'n_words': 1294, 'path': 'EnlargeTheHouse.ipynb', 'repo_name': 'TimeMagazineLabs/CongressionalApportionment', 'size': '12805'}