<jupyter_start><jupyter_text>![projectimage.jpg](./images/projectbanner.jpg) Demonstration Notebook for Cluster Analysis of the "Views of the Electorate Research Survey" (VOTERS) dataset_a project by [<NAME>](http://kalebnyquist.me)_&nbsp;&nbsp;This project demonstrates a potential application of unsupervised machine learning to help policy entrepreneurs and political scientists better understand one of the contributing factors to contemporary American political dysfunction — that is, the reality that majority support in terms of public opinion does not always translate to desired policy outcomes. For example, most Americans want to see [campaign finance reform](https://www.pewresearch.org/fact-tank/2018/05/08/most-americans-want-to-limit-campaign-spending-say-big-donors-have-greater-political-influence/), [background checks for gun ownership](https://news.gallup.com/poll/1645/guns.aspx), and [reductions in fossil fuel consumption](https://news.gallup.com/poll/248006/americans-support-reducing-fossil-fuel.aspx). However, this majority popular support has resulted in minimal national policy action at best, although there has been some state-level policy action and grassroots action in each of these issue arenas.What this project illustrates is one partial explanation for this phenomenon. With resonances to the [Condorcet paradox](https://en.wikipedia.org/wiki/Condorcet_paradox), where majorities can simultaneously prefer candidates for elected office in a cyclical "rock-paper-scissors" manner, here we assume there are limited opportunities for political representation available to voters (in terms of political parties, candidates for elected office, civil society advocacy organizations, activist movements, etc.) and that voters can coalesece around these opportunities for representation in a quasi-paradoxical manner. The best metaphor for this dilemna is a sort of "mathematical gerrymandering" that, while less nefarious and more accidental than [geographic gerrymandering](https://www.theatlantic.com/ideas/archive/2019/03/how-courts-can-objectively-measure-gerrymandering/585574/), is similar in that contested issues where the political support is highly concentrated within certain networks will tend to lose out to issues where the political support is evenly distributed across networks (at least enough so to form slight majorities per network.To illustrate, consider the simplified model below:![](./images/theorygraph.gif)Pretend that each dot represents a single voter, and that the horizontal axis represents that voter's "concern for economic issues" and the vertical axis represents "concern for social issues". In terms of a simple direct majority, positive support for social issues outweighs negative support for social issues, 17 voters to 13 voters. However, if we consider that each voter has to choose their representation in terms of economic and social issues simultaneously, we find that those voters self-organize into five distinct networks that are stratified in terms of their positive or negative concern for economic issues. Assuming that each network gets an equal amount of representation, and each voter gets a single vote within each network, we see that negative support for social issues wins out, 3 networks to 2 networks.  **Is it possible to measure how significant a factor this phenomenon is in American politics?** The assertion of this project is that __*yes, it is possible*__ and that the machine learning technique of __*cluster analysis is a viable method of doing so*__. The dataset we use for this project is the "Views of the Electorate Research Survey" (or VOTERS) dataset made publicly available by Democracy Fund's [Voter Study Group](https://www.voterstudygroup.org/data). Although not specifically designed for machine learning applications, due to its size and scope of this dataset it is one of the best freely available datasets for machine learning within the domain of political science.What follows is a step-by-step walkthrough of a Python-language program designed to do exactly this form of cluster analysis on VOTERS. Since I am trying to recommend Python code rather than sell 21st century snake oil, there also will be some discussion on the limitations of this analysis given current data capacities alongside the current development status of this code, which may be best understood as a working prototype that invites input rather than a finished product). As you work through this notebook, feel free to (if you are a Python developer) [fork the repository on GitHub](https://github.com/KalebNyquist/voters-clustering) to contribute your own amendments or (if you are a political scientist or policy entrepreneur) send me your comments and suggestions via e-mail to <EMAIL>. &nbsp;&nbsp; Import LibrariesBelow, we import libraries of Python code created specifically for working with the VOTERS dataset.<jupyter_code>import VOTERSdata
import VOTERSguide
import VOTERSvisualizer
import VOTERSclustering<jupyter_output><empty_output><jupyter_text>&nbsp;&nbsp; Data RetrievalThe code below imports the .csv version of the data into a Pandas dataframe suitable for Python. Access to the free 2018 Survey Full Data Set can be requested [here](https://www.voterstudygroup.org/publication/2018-voter-survey-1). Note that to submit a request for the data, you will have to give your name and email.<jupyter_code>voters = VOTERSdata.import_csv_file('data/VOTER_Survey_April18_Release1.csv')<jupyter_output>🗃️ Data imported!<jupyter_text>For intelligibility, the code below scrapes labels of survey variables from the corresponding guide. "Guide to 2018 Survey Data" available either [here](https://www.voterstudygroup.org/publication/2018-voter-survey-1), or when downloading the full data set.<jupyter_code>guide_dict = VOTERSguide.compile_dictionary('data/dfvsg_Guide_May2018_VOTER_Survey.pdf')<jupyter_output>Extracting Text from PDF: 100%|██████████████████████████████████████████████████████| 268/268 [00:15<00:00, 17.46it/s]<jupyter_text>&nbsp;&nbsp; Choose Respondent Group to AnalyzeThe 2018 Survey Full Data Set can be subsetted to one of six groups, based on the year(s) they responded to questions or if they belong to an oversample of Latino voters or an oversample of youth voters. Furthermore, each respondent is weighted according to a algorithm designed by survey administrator YouGov, which matches a respondent to a multi-dimensional strata of age, gender, race and education proportional to the U.S. Census Bureau's American Community Survey and other databases.Here, you can [view a sketch of the respondent groups across the sample](./images/respondentgroups.png). The following code allows you to select a specific subgroup and expand the data according to the sample weights. The tradeoff in the `magnitude` parameter above is precision versus computing time. The higher the magnitude, the more "data points" that are created from a particular respondent; however, each additional respondent exponentially increases the amount of computing time needed to create clusters.<jupyter_code>voters_panel = VOTERSdata.choose_subset(voters, 'panel')
reconstituted_voters_panel = VOTERSdata.apply_weights(voters_panel, 'panel', magnitude=9)<jupyter_output>Expanding data according to weight: 100%|██████████████████████████████████████████| 4705/4705 [07:18<00:00, 10.74it/s]<jupyter_text>&nbsp;&nbsp; Choose Features of InterestVOTERS contains responses collected over many years for a wide variety of questions, ranging from "If you had that chance to do it again, how would you vote in 2016?" (asked in 2017) to "Outside of attending religious services, how often do you pray?" (asked in 2011).Below, the Features Picker can be used to create a version of the VOTERS data subset that includes any combination of questions from various categories, such as: * __Issue Importance__ (regarding issues, e.g. "Climate change" or "Religious liberty")* __Feeling Thermometer__ (towards groups, e.g. "Asians" or "Feminists")* __Favorability__ (towards public figures, e.g. "Ted Cruz" or "<NAME>")* __Institutional Confidence__ (regarding large institutions, e.g., "Supreme Court" or "Big business")<jupyter_code>voters_issues = VOTERSdata.FeaturesPicker(IssueImportance=True, FeelingThermometer=False).process(reconstituted_voters_panel)<jupyter_output>🔵 60 features out of 1074 selected, relevant to: Issue Importance<jupyter_text>⚠️ *In the current version of this data analysis code, only the `IssueImportance` features are guaranteed to be clustered and visualized neatly.* &nbsp;&nbsp; Reduce Number of FeaturesBecause some questions were asked multiple times in different years to the same respondents, there exist a number of features in the data that are made redundant (assuming we are not doing an analysis of changes over time).Here, we condense the data by the most recent value available. Note that if a datapoint is missing for a question asked multiple times, this function imputes the "most recent" as the most recent response: that is, if a question was asked in 2017 did not receive a response from an individual who answered that question in 2016, then the 2016 data is used to impute the missing 2017 value. The assumption here is that political preferences are relatively stable over time.<jupyter_code>voters_issues_condensed = VOTERSdata.condense_by_most_recent_feature(voters_issues)<jupyter_output>📉 Number of features reduced from 60 to 23.<jupyter_text>As a means of representation into what this process achieves, the code below can be uncommented to see a list of the features in the original subset and in the condensed subset.<jupyter_code># print("\n--- Original ---")
# print(voters_issues.columns)
# print("\n--- Condensed ---")
# print(voters_issues_condensed.columns)<jupyter_output><empty_output><jupyter_text>&nbsp;&nbsp; Label Features into Readable EnglishDuring the data retrival stage, we scraped the VOTERS Survey Guide .pdf file to create a means of connecting variable codenames with English readable names. Here, the module below is used to undertake the actual translation.<jupyter_code>voters_issues_condensed.columns = VOTERSguide.feature_labeler(voters_issues_condensed, guide_dict, make_concise=True)<jupyter_output>🏷️ 23 variables labeled.
For example: "imiss_a_2017" → "Crime"<jupyter_text>To see the list of labeled features, uncomment and run the code below.<jupyter_code>#voters_issues.columns<jupyter_output><empty_output><jupyter_text>&nbsp;&nbsp; Manage missing valuesWe can visualize missing values in our data with help of the `missingno` library, below.<jupyter_code>import missingno
missingno.matrix(voters_issues_condensed)<jupyter_output><empty_output><jupyter_text>As is typical for survey data, some respondents did not answer a couple of the questions in our variables of interest. Furthermore, a few respondents hardly answered any questions at all (if any). For the clustering algorithm to work, we need at least some value the computer to work with. The first line of code below removes low-information respondents who did not answer a significant number of questions, while the second line of code imputes missing values in order to retain high-information respondents.<jupyter_code>voters_issues_condensed = VOTERSdata.remove_low_response_respondents(voters_issues_condensed)
voters_issues_imputed = VOTERSdata.impute_nas(voters_issues_condensed)<jupyter_output>Imputing: 100%|████████████████████████████████████████████████████████████████████████| 23/23 [00:38<00:00,  1.69s/it]<jupyter_text>Note that the imputation algorithm, with an emphasis on mode, works best for numerical data on a small scale (for example, Issue Importance, evaluated on a 1-4 scale). Larger scale numerical data would be better decided with an emphasis on mean or median (for example, the "Feeling Thermometer" questions, evaluated on a 1-100 scale). ⚠️ *In the current version of this data analysis code, the latter solution has not yet been engineered.*Now that we have removed low-information respondents and imputed the remaining missing values, we can return to our missing values matrix and see that we are now working with a complete dataset.<jupyter_code>missingno.matrix(voters_issues_imputed)<jupyter_output><empty_output><jupyter_text>&nbsp;&nbsp; Feature AgglmomerationSurvey data are prone to a number of subjectivity biases that can skew the data.The purpose of feature agglmomeration is to reduce bias created by (creators of) the survey instrument itself. That is, if two or three questions are asked about similar topics (eg., "climate change" and the "environnment"), then these topics are condensed into one so the underlying concern is not given a double or triple amount of significance in determining the cluster, relative to a more unique question topic (eg., "religious liberty").<jupyter_code>voters_agglomerated = VOTERSdata.feature_agglomeration(voters_issues_imputed, 15, rounding=False)<jupyter_output>🔹→💠←🔹 23 features agglomerated into 15 hybrid features.<jupyter_text>⚠️ _The number of hybrid features, 15, is arbitrary. One way to improve this project would be to create a comparison clustering metrics for different number of hybrid features, or aggregate results across different number of hybrid features._To see the list of agglmomerated features, uncomment and run the code below.<jupyter_code>#voters_agglomerated.columns<jupyter_output><empty_output><jupyter_text>&nbsp;&nbsp; Correcting for Respondent BiasNext, we remove two forms of bias created by the respondents and their answers to the questions. First, we remove "Abortion" as an issue of concern. Because of the unique nature of this issue, it is unclear if the concern being reported is "Abortion rights" or "Abortion restrictions". (Arguably, "Immigration" has a similar ambiguity, but this does not rise to the same degree as "Abortion", and it does not appear to affect our clusters in the same way abortion does, so we maintain this feature.) Note that, later on, we can still see how the "Abortion" feature is distributed among clusters even if we do not use this feature to construct the clusters.<jupyter_code>voters_agglomerated.drop('Abortion', axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>To make the math easier and more intuitive, the 4-1 scale is converted to a 0-3 scale.<jupyter_code>voters_inverted = -((voters_agglomerated - 1) - 3)
print("🔀 Scale inverted.")<jupyter_output>🔀 Scale inverted.<jupyter_text>Second, many respondents are likely to respond to nearly every question as "very important". In terms of our clustering algorithm, this creates a problem in that the initial clusters that might be created are simply (1) people who are very concerned just about everything and (2) people who are not very concerned about much of anything. This is not as interesting as relative preferences, so to correct for this we create a "proportional" version of the dataset that adds up the "total amount of concern" from each respondent and uses that to scale each specific amount of issue concern.<jupyter_code>voters_proportional = VOTERSdata.importance_scale_proportionizer(voters_inverted)<jupyter_output>⏫ Proportional scale calculated.<jupyter_text>&nbsp; &nbsp; Rank features by mean value (visualization aid)<jupyter_code>#voters_proportional.columns<jupyter_output><empty_output><jupyter_text>To make the graph easier to interpret, features are ranked by their mean total value. Uncomment code above and below to see the difference.<jupyter_code>features_ranked = voters_proportional.mean().sort_values().keys()
voters_proportional = voters_proportional[features_ranked[::1]]
voters_inverted = voters_inverted[features_ranked[::1]]
#voters_proportional.columns<jupyter_output><empty_output><jupyter_text>&nbsp;&nbsp; Visualize entire dataset, pre-clusteringBefore moving to a somewhat "magical" step of clustering our data, this is a good checkpoint for us to take a step back and visualize what our processed dataset now looks like. However, for our visualization to display properly, we first need to repeat some of our previous steps on a copy of the imputed version of the dataset, where rounding is introduced to feature agglomeration function.<jupyter_code>voters_agglomerated_visualization = VOTERSdata.feature_agglomeration(voters_issues_imputed, 15, rounding=True)
voters_agglomerated_visualization.drop('Abortion', axis=1, inplace=True)
voters_inverted_visualization = voters_agglomerated_visualization.applymap(VOTERSdata.importance_scale_inverter)
voters_inverted_visualization = voters_inverted_visualization[features_ranked[::1]]<jupyter_output>🔹→💠←🔹 23 features agglomerated into 15 hybrid features.<jupyter_text>Now that we have a version of the dataset that can be used for visualiztion, we can now plug this into the code below to view the dataset in agglomerated form.<jupyter_code>VOTERSvisualizer.cluster_visualization(voters_proportional, voters_proportional, voters_inverted_visualization, "Entire Dataset")<jupyter_output><empty_output><jupyter_text>To visualize each cluster, the grey bars in the back indicate percentage of respondents who gave a particular answer to a survey question (or, in the case of features combined by agglomeration, the visualization is "rounded" to the nearest answer; i.e., two unimportants and one not very important becomes an unimportant). **Dark grey** bars indicate the percentage of "very important" responses, whereas **light grey** bars indicate the percentage of "unimportant" responses.  &nbsp;&nbsp; Clustering Algorithm All the code prior to this point has been for the purpose of preparing our data to be "clustered", an unsupervised machine learning technique that detects patterns in the data.An easy example of clustering would be to imagine a dataset of U.S. Cities in a two-dimensional space of latitude and longitude. A clustering algorithm might find the following clusters: (Seattle, Portland, Boise), (Minneapolis, Milwaukee, Chicago, Indianapolis), (Los Angeles, San Diego, San Francisco, Las Vegas), (New York, Philadelphia, Washington DC, Boston). In the case of the VOTERS dataset we have modified, the idea is the same, except the number of dimensions is equal to number of agglomerated features. That is: instead of (A) latitude and (B) longitude, we might have something like (A) "The economy, Jobs" (B) "Health Care, Social Security, Medicare", (C) "Crime, Terrorism, Taxes", and so on.The algorithm used here is [Ward's method](https://en.wikipedia.org/wiki/Ward%27s_method), an agglomerative hierarchical clustering technique. Compared to [other clustering methods](https://scikit-learn.org/stable/modules/clustering.html), Ward's has a couple advantages.* It is **hierarchical**. This means that every additional cluster is created as a split between two pre-existing clusters. Therefore, the structure of our clustering does not radically change if, for example, we go from five to six clusters or from ten to seven. Furthermore, because we can chart this hierarchy, we can easily which clusters are most closely related to each other. * It is **comprehensive**. Every datapoint gets assigned a cluster, even if it is an outlier or ambiguously between two clusters. This reflects the reality that fringe or idiosyncratic political opitions do not necessarily discourage political participation: "every vote counts."* It produces clusters of **relatively equal sizes**. While there will likely be some small clusters (particularly if a peculiar pattern of responses are given by a highly weighted respondent or two) and there likely will be a large center cluster or two, for the most part the size of clusters produced will be of a similar scale. That said, it would not be difficult to run a different clustering algorithm at this point. It would just take a few tweaks to the underlying code. All the pre-processing steps that were applied earlier, and all the visualization and analysis code that comes later, should work with most clustering algorithms. &nbsp;&nbsp; Performance Metrics over Number of ClustersFirst, we need to determine the number of clusters we want to work with.The code below takes our data and evaluates the performance of the algorithm across a range of total clusters. For both metrics displayed in the first visualization below, silhouette and Calinski-Harabasz, the higher the number the better. In the second visualization, the greater the distance between "splits" the better.<jupyter_code>silhouette_scores, calinski_harabasz_scores, cal = VOTERSclustering.performance_per_no_clusters(voters_proportional, 15)<jupyter_output>Clustering: 100%|███████████████████████████████████████████████████████████████████| 14/14 [1:46:23<00:00, 455.94s/it]<jupyter_text>While higher Calinski-Harabasz and higher Silhouette scores are desirable, we also are interested in higher number of clusters. That is, the more clusters we have, the more "precise" we can be, even though we risk being less "accurate". One way to determine a threshold number of clusters we might want to work with is by looking for local maxima in the Silhouette score (where the Silhouette score is increasing). The code below is helpful for this purpose.<jupyter_code>VOTERSclustering.metrics_table(calinski_harabasz_scores, silhouette_scores)<jupyter_output><empty_output><jupyter_text>*__In the demonstration version of this notebook, we proceed with 8 clusters, as that is the highrst silhouette score in the range reported above.__* &nbsp;&nbsp; Make Determined Number of Clusters from Agglomerated FeaturesOnce we have determined the number of clusters we want to use, we can now make cluster assignments and a dendrogram given that specific number.<jupyter_code>cluster_assignments = VOTERSclustering.get_cluster_assignments(voters_proportional, 8, dendrogram_generate=True)<jupyter_output>✨ Cluster Assignments obtained.
Now drawing dendrogram plot...<jupyter_text>⚠️ *Note that in order to match individual clusters from the dendrogram above with the visualizations below, this process has to be done by cluster size rather than an id or order number. This admittedly is a limitation in the current usability of the code, and is the consequence of using two different Python modules for cluster assignment and the dendrogram calculations.* **For convenience, I've created [a PDF sketch](./images/dendrogram_cheatsheet.pdf) that arranges the dendrogram with the cluster visualizations to make it easier to see how the demonstration version of the cluster appears.** &nbsp;&nbsp; Make clusters from non-agglomerated featuresWe now create a new copy of the dataset from right before we agglomerated the features. Although we use agglomerated features for purposes of clustering, for interpretation of the clusters it makes sense to keep features separate (that is, saying a respondent considers "Education" to be *very important* and "Poverty" to be *somewhat unimportant* tells us more than if they consider "Education, Poverty" to be *somewhat important*).<jupyter_code>voters_nonagglomerated = voters_issues_imputed.copy()<jupyter_output><empty_output><jupyter_text>We run the new copy of the dataset through the same pre-processing steps as before that correct for respondent bias.<jupyter_code>voters_inverted = voters_nonagglomerated.applymap(VOTERSdata.importance_scale_inverter)
voters_proportional = VOTERSdata.importance_scale_proportionizer(voters_inverted)<jupyter_output>⏫ Proportional scale calculated.<jupyter_text>And, again, we rank features to make our dataset easier to visualize.<jupyter_code>features_ranked = voters_proportional.mean().sort_values().keys()
voters_proportional = voters_proportional[features_ranked[::1]]
voters_inverted = voters_inverted[features_ranked[::1]]<jupyter_output><empty_output><jupyter_text>We now apply our cluster assignments to the non-agglomerated dataset.<jupyter_code>clusters_proportional = VOTERSclustering.assign_clusters(voters_proportional, cluster_assignments)
clusters_absolute = VOTERSclustering.assign_clusters(voters_inverted, cluster_assignments)<jupyter_output><empty_output><jupyter_text>And, now, we can visualize our dataset on a per cluster basis! Note that **red lines** indicate that for that particular feature, the cluster as a whole considered the issue less important than the average person in the sample whereas **blue lines** indicate that the cluster as a whole considers the issue to be more important than the average person.<jupyter_code>VOTERSvisualizer.show_all_clusters(clusters_proportional, voters_proportional, clusters_absolute)<jupyter_output><empty_output><jupyter_text>Notice in particular places where there are some respondents who consider the issue to be of high importance (indicated by **dark grey**) but the cluster as a whole considers the issue to be of less importance (indicated by a long **red line**). Similarly, there are some respondents who consider the issue to be a low importance (indicated by **light grey**) but the cluster as a whole considers the issue to be of greater importance (indicated by a long **blue line**).These "outliers" are significant in the context of our investigation, because they indicate voters whose ability to express a policy preference on a particular issue is crowded out by their other preferences. In terms of efficiency use of resources within a political economy, political institutions (e.g. elected officials, broad-based advocacy organizations, political parties, etc.) will tend towards satisfying the concerns of a cluster as a whole rather than the idiosyncracies of each individual voter.One democratically frustrating outcome this can potentially create is that voters who care deeply for a particular issue ("Animal Rights", to use an example not in the survey) are concentrated (say, 80%) in clusters that do not add up to a majority and spread out among other clusters (say 40%) that do add up to a majority. So even if over 50% of the population cares deeply about Animal Rights, this direct majority might be not be enough to effect policy change in a representative system. &nbsp;&nbsp; Identify possible coalitionsThe above graphs are made available so that an interpreter can describe each cluster by policy profile preference, and the hierarchical dendrogram can be used for identifying which clusters are most closely related to each other. That said, given most parameters, it is unlikely that any particular cluster already represents 50% of the sampled population. The code below aids in interpretation by finding possible coalitions of clusters that add up to 50% of the population.First, we need to calculate cluster sizes (relative to the total dataset).<jupyter_code>cluster_sizes = VOTERSclustering.cluster_sizes_dict(clusters_proportional, voters_proportional) #can also be voters inverted
VOTERSclustering.display_cluster_sizes(cluster_sizes)<jupyter_output>📐 Cluster sizes (in terms of percentage of total) calculated.<jupyter_text>The first analysis that can be done is by issue. Here, we add up the clusters that are "positive" on a particular issue, and also add up the clusters that are "negative" on a particular issue. If concern for an issue was evenly distributed between clusters, we would expect something like a 50/50 split here. However, if above-average support for an issue is concentrated in particular clusters, then the clusters that add up to positive be less than half (and vice versa). The greater this difference, the more potentially frustrating it will be to achieve desired policy outcomes through a representative democratic process on that particular issue.<jupyter_code>VOTERSclustering.issues_by_clustered_advantage(clusters_proportional, voters_proportional, cluster_sizes)<jupyter_output>Religious liberty
+0.82 [1, 2, 4, 5, 7, 8]
-0.18 [3, 6]

Abortion
+0.71 [1, 2, 4, 8]
-0.29 [3, 5, 6, 7]

Climate change
+0.68 [1, 3, 5]
-0.32 [2, 4, 6, 7, 8]

Gender equality
+0.68 [1, 3, 5]
-0.32 [2, 4, 6, 7, 8]

Family and medical leave
+0.68 [1, 3, 5]
-0.32 [2, 4, 6, 7, 8]

Racial equality
+0.68 [1, 3, 5]
-0.32 [2, 4, 6, 7, 8]

The environment
+0.68 [1, 3, 5]
-0.32 [2, 4, 6, 7, 8]

Poverty
+0.68 [1, 3, 5]
-0.32 [2, 4, 6, 7, 8]

Gay rights
+0.57 [1, 3]
-0.43 [2, 4, 5, 6, 7, 8]

Social Security
+0.52 [2, 3, 5, 6, 7, 8]
-0.48 [1, 4]

Education
+0.44 [2, 3, 4, 5]
-0.56 [1, 6, 7, 8]

Terrorism
+0.43 [2, 4, 5, 6, 8]
-0.57 [1, 3, 7]

Crime
+0.43 [2, 4, 5, 6, 8]
-0.57 [1, 3, 7]

Money in politics
+0.4 [2, 3, 6, 8]
-0.6 [1, 4, 5, 7]

Medicare
+0.4 [2, 3, 5, 7]
-0.6 [1, 4, 6, 8]

Health care
+0.4 [3, 4, 5, 6, 7, 8]
-0.6 [1, 2]

Infrastructure investment
+0.38 [2, 3, 4, 6]
-0.62 [1, 5, 7, 8]

The size of government
+0.32 [2, 4, 6, 8]
-0.68 [1, 3, 5, 7]

Immigration
+0.32 [2, 4, 6, 8]
-0.68 [1,[...]<jupyter_text>However, looking at the landscape on an issue-by-issue basis doesn't tell the full story. While thinking in terms of 50% or more support might make sense for passing legislation on a piecemeal deal, it certainly does not indicate if those clusters could work together on a sustained basis (especially, in the American context, the executive branch of government). The following code can be used to see which cluster coalitions are feasible. Here, we search for consensus coalitions which represent half of the population and agree on at least four issues.<jupyter_code>cluster_coalitions = VOTERSclustering.generate_cluster_coalitions(cluster_sizes, maximum = 10, threshold = .5)
VOTERSclustering.find_consensus_coalitions(cluster_coalitions, clusters_proportional, voters_proportional, 
                                           agreements = 4, show_negatives=False)<jupyter_output>(1, 3)
Positive Consensus: ['Gay rights', 'Climate change', 'Gender equality', 'Family and medical leave', 'Racial equality', 'The environment', 'Poverty']
Negative Consensus: ['The size of government', 'Immigration', 'The budget deficit', 'Terrorism', 'Taxes', 'Crime', 'Jobs', 'The economy']
 
(1, 5)
Positive Consensus: ['Climate change', 'Gender equality', 'Family and medical leave', 'Racial equality', 'The environment', 'Religious liberty', 'Poverty']
Negative Consensus: ['The size of government', 'Immigration', 'Money in politics', 'The budget deficit', 'Infrastructure investment', 'Taxes', 'Jobs', 'The economy']
 
(1, 8)
Positive Consensus: ['Abortion', 'Religious liberty']
Negative Consensus: ['Infrastructure investment', 'Education', 'Medicare']
 
(1, 3, 5)
Positive Consensus: ['Climate change', 'Gender equality', 'Family and medical leave', 'Racial equality', 'The environment', 'Poverty']
Negative Consensus: ['The size of government', 'Immigration', 'The budget deficit', 'Taxes[...]<jupyter_text>This code is versatile! Here, it can be used to see which consensus coalitions represent a supermajority (two-thirds) of the population and agree on at least one issue.<jupyter_code>cluster_coalitions = VOTERSclustering.generate_cluster_coalitions(cluster_sizes, maximum = 10, threshold = .67)
VOTERSclustering.find_consensus_coalitions(cluster_coalitions, clusters_proportional, voters_proportional, 
                                           agreements = 1, show_negatives=False)<jupyter_output>(1, 2, 5)
Positive Consensus: ['Religious liberty']
Negative Consensus: []
 
(1, 2, 8)
Positive Consensus: ['Abortion', 'Religious liberty']
Negative Consensus: []
 
(1, 3, 5)
Positive Consensus: ['Climate change', 'Gender equality', 'Family and medical leave', 'Racial equality', 'The environment', 'Poverty']
Negative Consensus: ['The size of government', 'Immigration', 'The budget deficit', 'Taxes', 'Jobs', 'The economy']
 
(1, 2, 4, 5)
Positive Consensus: ['Religious liberty']
Negative Consensus: []
 
(1, 2, 4, 8)
Positive Consensus: ['Abortion', 'Religious liberty']
Negative Consensus: []
 
(1, 2, 5, 7)
Positive Consensus: ['Religious liberty']
Negative Consensus: []
 
(1, 2, 5, 8)
Positive Consensus: ['Religious liberty']
Negative Consensus: []
 
(1, 2, 7, 8)
Positive Consensus: ['Religious liberty']
Negative Consensus: []
 
(1, 3, 5, 7)
Positive Consensus: []
Negative Consensus: ['The size of government', 'Immigration', 'The budget deficit', 'Taxes', 'Jobs', 'The economy']
 
(1, 2[...]<jupyter_text>&nbsp;&nbsp; What else can we know about these clusters?While creating these clusters only required that we use a subset of the data available to us in the VOTERS data, we can use the remainder of data available to us through VOTERS to learn more about each cluster. First, we match our clusters up with the original (post-expansion-by-weight) dataset.<jupyter_code>full_data_clusters = VOTERSclustering.create_full_data_clusters(reconstituted_voters_panel, clusters_absolute)<jupyter_output>🗃️🔺✨ Cluster assignments merged with original dataset, expanded by weight.<jupyter_text>Then, we can use most any variable in the survey to learn more about a particular cluster. Below are results for political party identification in 2017, followed by race, religion, and gender.⚠️ _Note that the dictionary scraper isn't perfect! For example, it is likely to be confused by questions where there are numerals in the answer, for example, a question such as "highest level of education completed" having an answer like "**4**-year degree"_.<jupyter_code>VOTERSdata.investigate_variable_across_clusters(guide_dict, 'pid3_2017', full_data_clusters)
VOTERSdata.investigate_variable_across_clusters(guide_dict, 'race_2017', full_data_clusters)
VOTERSdata.investigate_variable_across_clusters(guide_dict, 'religpew_2017', full_data_clusters)
VOTERSdata.investigate_variable_across_clusters(guide_dict, 'gender_2018', full_data_clusters)<jupyter_output>Are you male or female?

Cluster #1 breakdown:
Female    0.63
Male      0.37
Name: gender_2018, dtype: float64

Cluster #2 breakdown:
Male      0.62
Female    0.38
Name: gender_2018, dtype: float64

Cluster #3 breakdown:
Male      0.57
Female    0.43
Name: gender_2018, dtype: float64

Cluster #4 breakdown:
Male      0.7
Female    0.3
Name: gender_2018, dtype: float64

Cluster #5 breakdown:
Female    0.58
Male      0.42
Name: gender_2018, dtype: float64

Cluster #6 breakdown:
Male      0.68
Female    0.32
Name: gender_2018, dtype: float64

Cluster #7 breakdown:
Female    1.0
Name: gender_2018, dtype: float64

Cluster #8 breakdown:
Male      0.68
Female    0.32
Name: gender_2018, dtype: float64<jupyter_text>Note that if, for reference purposes, you are interested in seeing the breakdown of a particular variable for the entire dataset, the following code is also available.<jupyter_code>VOTERSdata.investigate_variable_by_subset(guide_dict, 'religpew_2017', reconstituted_voters_panel)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07577417680641255, 'fraction_numerical': 0.01977971281932991, 'mean_word_length': 5.8665610770144525, 'pattern_counts': {'":': 0, '<': 132, '<?xml version=': 0, '>': 128, 'https://': 11, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '36', 'dir': 'jupyter-structured-clean-dedup', 'id': '553766', 'license': 'MIT', 'n_tokens_mistral': 10290, 'n_tokens_neox': 9283, 'n_words': 4290, 'path': 'Demonstration Notebook.ipynb', 'repo_name': 'KalebNyquist/voters-clustering', 'size': '34711'}