<jupyter_start><jupyter_text>BipartiteData example<jupyter_code># Add pytwoway to system path, do not run this
# import sys
# sys.path.append('../../..')
# Import the pytwoway package 
# (Make sure you have installed it using pip install pytwoway)
import pytwoway as tw<jupyter_output><empty_output><jupyter_text>Simulate some dataThe package contains functions to simulate data. We use this here to keep things simple.<jupyter_code># For the example, we simulate data
sim_data = tw.SimTwoWay().sim_network()[['wid', 'fid', 'comp', 'year']]
display(sim_data)<jupyter_output><empty_output><jupyter_text>BipartiteDataBipartiteData is a superclass with 3 subclasses:- BipartiteLong- BipartiteLongCollapsed- BipartiteEventStudyThe user interfaces with BipartiteData which converts between the 3 subclasses.<jupyter_code># Our data is in long form (each row gives a single observation)
bd = tw.BipartiteData(sim_data, formatting='long') # Long is default formatting, most data is long
bd.clean_data() # Should always clean data after initializing
# Let's see how our data looks
display(bd.data)<jupyter_output><empty_output><jupyter_text>Converting formats<jupyter_code># While our original data is long, we might want it to be in event study form (each row gives two consecutive observations)
bd.long_to_es()
display(bd.data)
# We can also use event study data to retrive cross section data (cs=1 gives y1 as y1 for both stayers and movers; cs=0 gives y2 as y1 for only movers - this allows (almost) all income data to be accessed from the y1 column. Note that for movers, the last observation for each worker is not available without manipulation as it is shifted to the y2 column. Also note that the y1 row contains duplicates for all mover income, except for the first period.)
display(bd.get_cs())
# Maybe we want to convert back into long form
bd.es_to_long()
display(bd.data)
# Now suppose we want to collapse by employment spells (so any consecutive observations with the same worker in the same firm are collapsed into 1 observation)
bd.long_to_collapsed_long()
display(bd.data)
# We can then check out the event study using collapsed data
bd.collapsed_long_to_es()
display(bd.data)
# We can then go back to collapsed long
bd.es_to_collapsed_long()
display(bd.data)<jupyter_output><empty_output><jupyter_text>Clustering<jupyter_code># Starting over
bd = tw.BipartiteData(sim_data, formatting='long')
bd.clean_data()
display(bd.data)
# We can cluster from any format and clusters stay when reformatting
bd.cluster()
display(bd.data)
bd.long_to_es()
display(bd.data)
bd.es_to_long()
bd.long_to_collapsed_long()
display(bd.data)
bd.collapsed_long_to_es()
display(bd.data)<jupyter_output><empty_output><jupyter_text>Clustering options<jupyter_code># We can cluster on a specific year in long form
bd = tw.BipartiteData(sim_data, formatting='long')
bd.clean_data()
bd.cluster({'year': 1})
display(bd.data)
# We can cluster on movers or stayers in any form
# Note that not all firms are clustered, so some rows have NaNs
bd = tw.BipartiteData(sim_data, formatting='long')
bd.clean_data()
bd.cluster({'stayers_movers': 'stayers'})
display(bd.data)
# We can cluster on movers or stayers in any form
# Set 'dropna': True if you want to drop firms that don't get clustered (note that this fully resets firm and worker ids, since they must be contiguous for the FE/CRE/BLM estimators to work)
bd = tw.BipartiteData(sim_data, formatting='long')
bd.clean_data()
bd.cluster({'stayers_movers': 'stayers', 'dropna': True})
display(bd.data)
# We can cluster on movers or stayers in any form
bd = tw.BipartiteData(sim_data, formatting='long')
bd.clean_data()
bd.cluster({'stayers_movers': 'movers'})
display(bd.data)<jupyter_output><empty_output><jupyter_text>Some extra features<jupyter_code># We can get some summary statistics by printing the BipartiteData object
sim_data = tw.SimTwoWay().sim_network()[['wid', 'fid', 'comp', 'year']]
sim_data = sim_data.rename({'fid': 'f1i'}, axis=1)
bd = tw.BipartiteData(sim_data, col_dict={'fid': 'f1i'}, formatting='long')
bd.clean_data()
display(bd)
# BipartiteData does some nice column inference
# Suppose some of our columns are mislabeled - create a col_dict to clarify which columns to correct
sim_data = tw.SimTwoWay().sim_network()[['wid', 'fid', 'comp', 'year']]
sim_data = sim_data.rename({'fid': 'f1i'}, axis=1)
bd = tw.BipartiteData(sim_data, col_dict={'fid': 'f1i'}, formatting='long')
bd.clean_data()
display(bd.data)
# Suppose we have already clustered - the class determines this automatically
sim_data = tw.SimTwoWay().sim_network()[['wid', 'fid', 'comp', 'year']]
sim_data['j'] = 1
bd = tw.BipartiteData(sim_data, formatting='long')
bd.clean_data()
display(bd.data)
# Now suppose we mislabeled our clusters - the class drops this column
sim_data = tw.SimTwoWay().sim_network()[['wid', 'fid', 'comp', 'year']]
sim_data['j1'] = 1
bd = tw.BipartiteData(sim_data, formatting='long')
bd.clean_data()
display(bd.data)
# We can make sure our columns are included by specifying the corrected labels in col_dict (note that if the variable of interest requires 2 columns, e.g. weight 1 and weight 2 for event study data, BOTH columns are required - if only one column is included, it is automatically dropped.)
sim_data = tw.SimTwoWay().sim_network()[['wid', 'fid', 'comp', 'year']]
sim_data['j1'] = 1
bd = tw.BipartiteData(sim_data, col_dict={'j': 'j1'}, formatting='long')
bd.clean_data()
display(bd.data)
# We may also specify a col_dict that overwrites a column label that would normally be inferred, and inference is skipped
sim_data = tw.SimTwoWay().sim_network()[['wid', 'fid', 'comp', 'year']]
sim_data = sim_data.rename({'fid': 'j'}, axis=1)
bd = tw.BipartiteData(sim_data, col_dict={'fid': 'j'}, formatting='long')
# Note that even though we have a column named j, because we specify it is the 'fid' column it is not inferred as the cluster values
bd.clean_data()
display(bd.data)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11415141849924458, 'fraction_numerical': 0.004868222259526607, 'mean_word_length': 6.195652173913044, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '618594', 'license': 'MIT', 'n_tokens_mistral': 1978, 'n_tokens_neox': 1910, 'n_words': 667, 'path': 'docs/source/notebooks/.ipynb_checkpoints/bipartite_example-checkpoint.ipynb', 'repo_name': 'economoser/pytwoway', 'size': '5957'}