<jupyter_start><jupyter_text>W209 Final Project Data ProcessingWe will be using two sets of data from a current Kaggle competition, “Two Sigma: Using News to Predict Stock Movements.” The first is stock market data provided by Intrinion (from 2007 to present), which contains financial market information such as opening & closing prices, trade volumes, calculated returns, etc. The second data set is news data provided by <NAME>, for the same time period. This data set has information regarding news articles and alerts (article headlines, sentiments, etc.) that have been posted about the companies included in the market data set. The two data sets can be joined using the unique assetCode or assetName (i.e. company name) identifiers as well as the time variable. It is important to note that one company can have more than one assets, which will be taken into account in our visualization. Link to data souce: https://www.kaggle.com/c/two-sigma-financial-news/data    Note: In order to download the full dataset from kaggle we had to write a kernel within the kaggle competition and export the data to our local machines.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import datetime
import time
# reading in csv without index column 0

start = time.time()

market_df = pd.read_csv('data/market_data.csv', index_col=0)
news_df = pd.read_csv('data/news_data.csv', index_col=0)

end = time.time() - start

print("Data import completed in {}s".format(end))
# checking

#market_df.head()
#news_df.head()
# parsing the time columns in both df
start = time.time()

market_df['time'] = pd.to_datetime(market_df['time'])
news_df['time'] = pd.to_datetime(news_df['time'])

end = time.time() - start

print("Date parse completed in {}s".format(end))
# adding a date column at index 0

market_df.insert(0, 'date', market_df['time'].apply(lambda x: x.date()))
news_df.insert(0, 'date', news_df['time'].apply(lambda x: x.date()))
market_df.columns.values
news_df.columns.values
# dropping unused columns

news_drop_cols =['sourceTimestamp', 'firstCreated', 'sourceId', 'headline', 
                 'urgency', 'takeSequence', 'provider', 'audiences', 'bodySize', 
                 'companyCount', 'headlineTag', 'marketCommentary', 'sentenceCount', 
                 'firstMentionSentence', 'noveltyCount12H', 'noveltyCount24H', 
                 'noveltyCount3D', 'noveltyCount5D', 'noveltyCount7D', 'volumeCounts12H',
                 'volumeCounts24H', 'volumeCounts3D', 'volumeCounts5D', 'volumeCounts7D']
news_df.drop(news_drop_cols, axis=1, inplace=True)

assetCode_list = market2016_df['assetCode'].unique()
# cleaning up assetCode in news_df

start = time.time()

# for each unique assetCode in the market dataframe, find and replace in the news dataframe
# keeping the parenthesis '' in the search parameter

for c in assetCode_list:
    mask = news_df['assetCodes'].str.contains('\''+c+'\'', regex=False)
    news_df.loc[mask, 'assetCodes'] = c

# for those lines in news which do not have a match to a unique assetCode in market dataframe
# they will still be in the format {'600115.SS', '0670.HK', 'CEA.N'}
# hence match '{' and remove those lines from the news dataframe

mask_rm = news_df['assetCodes'].str.contains('{', regex=False)
news_df.drop(news2016_df.loc[mask_rm, 'assetCodes'].index, axis=0, inplace=True)

# reset the index for news dataframe
news_df.reset_index(drop=True, inplace=True)

end = time.time() - start

print("Asset Code cleanup completed in {}s".format(end))
# renaming the news dataframe column from 'assetCodes' to 'assetCode'
news_df.rename(columns = {'assetCodes':'assetCode'}, inplace = True)

# merging the two dataframes on data and assetCode
combined_df = market_df.merge(news_df, how='left', on=['date','assetCode'])
# saving to csv

news_df.to_csv('data/news_data_cleaned.csv')
combined_df.to_csv('data/combined_data.csv')
# slicing out only 2016 data for exploration

combined_df['date'] = pd.to_datetime(combined_df['date'])

combined2016_df = combined_df[(combined_df['date'] >= '2016-01-01') & (combined_df['date'] < '2017-01-01')]
# checking

#combined2016_df.head()
#combined2016_df.tail()
combined2016_df.to_csv('data/combined2016_data.csv')
# switching over to Tableau...
# reading in csv without index column 0

start = time.time()

combined_df = pd.read_csv('data/combined_data.csv', index_col=0)
end = time.time() - start

print("Data import completed in {}s".format(end))
list(combined_df)
cd2 = combined_df[combined_df.assetName_x != combined_df.assetName_y]
cd2.groupby(['assetName_y', 'assetName_x', 'assetCode']).count().head(100)
# Cleaning data, correcting assetNames

combined_df.assetName_y.replace(to_replace='ADESA Inc', value='KAR Auction Services Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='ADT Corp', value='ADT Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Advanced Semiconductor Engineering Inc', value='ASE Technology Holding Co Ltd', inplace=True)
combined_df.assetName_y.replace(to_replace='Allergan Inc', value='Allergan plc', inplace=True)
combined_df.assetName_y.replace(to_replace='Ameron International Corp', value='AMN Healthcare Services Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Anheuser Busch Inbev SA', value='Anheuser Busch Inbev NV', inplace=True)
combined_df.assetName_y.replace(to_replace='Anheuser-Busch Companies LLC', value='Anheuser Busch Inbev NV', inplace=True)
combined_df.assetName_y.replace(to_replace='Ashland LLC', value='Ashland Global Holdings Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Atlassian Corporation PLC', value='Atlassian Corp PLC', inplace=True)
combined_df.assetName_y.replace(to_replace='BancorpSouth Inc', value='BancorpSouth Bank', inplace=True)
combined_df.assetName_y.replace(to_replace='CBOE Holdings Inc', value='Cboe Global Markets Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='CenterState Banks Inc', value='CenterState Bank Corp', inplace=True)
combined_df.assetName_y.replace(to_replace='Chubb Corp', value='Chubb Ltd', inplace=True)
combined_df.assetName_y.replace(to_replace='Coca-Cola Enterprises Inc', value='Coca-Cola European Partners PLC', inplace=True)
combined_df.assetName_y.replace(to_replace='Coeur d\'Alene Mines Corp', value='Coeur Mining Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Cogent Communications Group Inc', value='Cogent Communications Holdings Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='CommVault Systems Inc', value='Commvault Systems Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Dana Corp', value='Dana Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Delphi Automotive PLC', value='Delphi Technologies PLC', inplace=True)
combined_df.assetName_y.replace(to_replace='Dennys Corp', value='Denny\'s Corp', inplace=True)
combined_df.assetName_y.replace(to_replace='Discovery Communications Inc', value='Discovery Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Dominion Resources Inc', value='Dominion Energy Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Energy Transfer LP', value='Energy Transfer Partners LP', inplace=True)
combined_df.assetName_y.replace(to_replace='Enterprise GP Holdings LP', value='EP Energy Corp', inplace=True)
combined_df.assetName_y.replace(to_replace='Envision Healthcare Holdings Inc', value='Envision Healthcare Corp', inplace=True)
combined_df.assetName_y.replace(to_replace='Evercore Partners Inc', value='Evercore Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Expedia Inc', value='Expedia Group Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Express Scripts Inc', value='Express Scripts Holding Co', inplace=True)
combined_df.assetName_y.replace(to_replace='FMC Technologies Inc', value='TechnipFMC PLC', inplace=True)
combined_df.assetName_y.replace(to_replace='Genworth Holdings Inc', value='Genworth Financial Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Gerdau Macsteel Inc', value='Quanex Building Products Corp', inplace=True)
combined_df.assetName_y.replace(to_replace='HCA Holdings Inc', value='HCA Healthcare Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Herbalife Ltd', value='Herbalife Nutrition Ltd', inplace=True)
combined_df.assetName_y.replace(to_replace='ITT Corp', value='ITT Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='J2 Global Inc', value='j2 Global Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Jabil Circuit Inc', value='Jabil Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Johnson Controls Inc', value='Johnson Controls International PLC', inplace=True)
combined_df.assetName_y.replace(to_replace='Kinder Morgan Kansas Inc', value='Kinder Morgan Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Knight Transportation Inc', value='Knight-Swift Transportation Holdings Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='LSI Corp', value='Life Storage Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='MGOC Inc', value='Media General Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Metlife Inc', value='MetLife Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Monro Muffler Brake Inc', value='Monro Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Monster Beverage 1990 Corp', value='Monster Beverage Corp', inplace=True)
combined_df.assetName_y.replace(to_replace='Mylan Inc', value='Mylan NV', inplace=True)
combined_df.assetName_y.replace(to_replace='NextEra Energy Inc', value='Nextera Energy Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Nicor Inc', value='Southern Company Gas', inplace=True)
combined_df.assetName_y.replace(to_replace='Open Text Corp Canada', value='Open Text Corp', inplace=True)
combined_df.assetName_y.replace(to_replace='PACCAR Inc', value='Paccar Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Parkway Properties Inc', value='Parkway Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Pentair plc', value='Pentair PLC', inplace=True)
combined_df.assetName_y.replace(to_replace='Potlatch Corp', value='PotlatchDeltic Corp', inplace=True)
combined_df.assetName_y.replace(to_replace='Priceline Group Inc', value='Booking Holdings Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Quintiles Transnational Holdings Inc', value='Quintiles IMS Holdings Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Reliance Steel and Aluminum Co', value='Reliance Steel & Aluminum Co', inplace=True)
combined_df.assetName_y.replace(to_replace='Ritchie Bros Auctioneers Inc', value='Ritchie Bros. Auctioneers Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Rowan Companies Inc', value='Rowan Companies PLC', inplace=True)
combined_df.assetName_y.replace(to_replace='SAP AG', value='SAP SE', inplace=True)
combined_df.assetName_y.replace(to_replace='Salesforce.Com Inc', value='Salesforce.com Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Sensata Technologies Holding NV', value='Sensata Technologies Holding PLC', inplace=True)
combined_df.assetName_y.replace(to_replace='Sohu.Com Inc', value='Sohu.com Ltd', inplace=True)
combined_df.assetName_y.replace(to_replace='Southwest Gas Corp', value='Southwest Gas Holdings Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Store Capital Corp', value='STORE Capital Corp', inplace=True)
combined_df.assetName_y.replace(to_replace='Sun Healthcare Group Inc (Pre-Merger)', value='Sabra Health Care REIT Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Sunoco Inc', value='Sunoco LP', inplace=True)
combined_df.assetName_y.replace(to_replace='Swift Transportation Co', value='Knight-Swift Transportation Holdings Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='TAL International Group Inc', value='TAL Education Group', inplace=True)
combined_df.assetName_y.replace(to_replace='TRI Pointe Homes Inc', value='TRI Pointe Group Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Take Two Interactive Software Inc', value='Take-Two Interactive Software Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Telekomunikasi Indonesia (Persero) Tbk PT', value='Telekomunikasi Indonesia (Persero) Tbk Perusahaan Perseroan PT', inplace=True)
combined_df.assetName_y.replace(to_replace='TiVo Solutions Inc', value='TiVo Corp', inplace=True)
combined_df.assetName_y.replace(to_replace='Transocean Inc', value='Transocean Ltd', inplace=True)
combined_df.assetName_y.replace(to_replace='Trimble Navigation Ltd', value='Trimble Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Ulta Salon Cosmetics and Fragrance Inc', value='Ulta Beauty Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='W. P. Carey Inc', value='WP Carey Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='WMI Holdings Corp', value='Waste Management Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Wal-Mart Stores Inc', value='Walmart Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Walgreen Co', value='Walgreens Boots Alliance Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Waste Connections US Inc', value='Waste Connections Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Western Alliance Bancorporation', value='Western Alliance Bancorp', inplace=True)
combined_df.assetName_y.replace(to_replace='WisdomTree Investments Inc', value='Wisdom Tree Investments Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Yahoo! Inc', value='Altaba Inc', inplace=True)
combined_df.assetName_y.replace(to_replace='Zillow Inc', value='Zillow Group Inc', inplace=True)

#Cleaning more complicated dual tickers / mergers
combined_df.loc[combined_df.assetCode == 'HRI.N', 'assetName_x'] = 'Herc Holdings Inc'
combined_df.loc[combined_df.assetCode == 'HRI.N', 'assetName_y'] = 'Herc Holdings Inc'
combined_df.loc[combined_df.assetCode == 'HTZ.N', 'assetName_x'] = 'Hertz Global Holdings Inc'
combined_df.loc[combined_df.assetCode == 'HTZ.N', 'assetName_y'] = 'Hertz Global Holdings Inc'

combined_df.loc[combined_df.assetCode == 'BSAC.N', 'assetName_x'] = 'Banco Santander-Chile'
combined_df.loc[combined_df.assetCode == 'BSAC.N', 'assetName_y'] = 'Banco Santander-Chile'
combined_df.loc[combined_df.assetCode == 'SAN.N', 'assetName_x'] = 'Banco Santander SA'
combined_df.loc[combined_df.assetCode == 'SAN.N', 'assetName_y'] = 'Banco <NAME>'

combined_df.loc[combined_df.assetCode == 'TLRD.N', 'assetName_x'] = 'Tailored Brands Inc'
combined_df.loc[combined_df.assetCode == 'TLRD.N', 'assetName_y'] = 'Tailored Brands Inc'
combined_df.loc[combined_df.assetCode == 'MW.N', 'assetName_x'] = 'Men\'s Wearhouse Inc'
combined_df.loc[combined_df.assetCode == 'MW.N', 'assetName_y'] = 'Men\'s Wearhouse Inc'
# Remaining problematic data to be deleted:
combined_df[combined_df.assetName_x != combined_df.assetName_y].groupby(['assetName_y', 'assetName_x', 'assetCode']).count().head(100)
# Deleting the news related columns in these rows

news_cols_del = ['time_y',
                 'subjects',
                 'wordCount',
                 'assetName_y',
                 'relevance',
                 'sentimentClass',
                 'sentimentNegative',
                 'sentimentNeutral',
                 'sentimentPositive',
                 'sentimentWordCount']

combined_df.loc[(combined_df.assetCode == 'AG.N') & (combined_df.assetName_y == 'AGCO Corp'), news_cols_del] = np.NaN
combined_df.loc[(combined_df.assetCode == 'CC.N') & (combined_df.assetName_y == 'Circuit City Stores Inc'), news_cols_del] = np.NaN
combined_df.loc[(combined_df.assetCode == 'NSM.N') & (combined_df.assetName_y == 'National Semiconductor Corp'), news_cols_del] = np.NaN
combined_df.loc[(combined_df.assetCode == 'PVG.N') & (combined_df.assetName_y == 'Penn Virginia GP Holdings LP'), news_cols_del] = np.NaN
combined_df.loc[(combined_df.assetCode == 'Q.N') & (combined_df.assetName_y == 'Qwest Communications International Inc'), news_cols_del] = np.NaN
combined_df.loc[(combined_df.assetCode == 'RARE.O') & (combined_df.assetName_y == 'Rare Hospitality International Inc'), news_cols_del] = np.NaN
combined_df.loc[(combined_df.assetCode == 'H.N') & (combined_df.assetName_y == 'Realogy Group LLC'), news_cols_del] = np.NaN
combined_df.loc[(combined_df.assetCode == 'SE.N') & (combined_df.assetName_y == 'Spectra Energy Corp'), news_cols_del] = np.NaN
combined_df.loc[(combined_df.assetCode == 'CAL.N') & (combined_df.assetName_y == 'United Airlines Inc'), news_cols_del] = np.NaN
combined_df.loc[(combined_df.assetCode == 'WMS.N') & (combined_df.assetName_y == 'WMS Industries Inc'), news_cols_del] = np.NaN
combined_df.loc[(combined_df.assetCode == 'CCU.N') & (combined_df.assetName_y == 'iHeartCommunications Inc'), news_cols_del] = np.NaN

combined_df[combined_df.assetName_x != combined_df.assetName_y].groupby(['assetName_y', 'assetName_x', 'assetCode']).count().head(100)
# Dropping the assetName_y, renaming assetName_x to assetName

combined_drop_cols = ['returnsClosePrevMktres1','returnsOpenPrevMktres1', 'returnsClosePrevMktres10',
 'returnsOpenPrevMktres10',
 'returnsOpenNextMktres10','assetName_y']

combined_df.drop(combined_drop_cols, axis=1, inplace=True)
combined_df.rename(columns = {'assetName_x':'assetName'}, inplace = True)
combined_df.rename(columns = {'time_x':'timeMkt'}, inplace = True)
combined_df.rename(columns = {'time_y':'timeNews'}, inplace = True)
# Calculating the new combined sentimentRating column

combined_df.insert(loc=20, column='sentimentRating', value=np.NaN)
combined_df.loc[(combined_df.sentimentClass == -1), 'sentimentRating'] = -1* combined_df[combined_df.sentimentClass == -1].sentimentNegative
combined_df.loc[(combined_df.sentimentClass == 0), 'sentimentRating'] = 0
combined_df.loc[(combined_df.sentimentClass == 1), 'sentimentRating'] = combined_df[combined_df.sentimentClass == 1].sentimentPositive
combined_df.head(30)
# Saving to csv

combined_df.to_csv('data/combined_data_v2.csv')
# Saving 2016 only to csv

combined2016_df = combined_df[(combined_df['date'] >= '2016-01-01') & (combined_df['date'] < '2017-01-01')]
combined2016_df.to_csv('data/combined2016_data_v2.csv')<jupyter_output><empty_output><jupyter_text>Further Data AggregationUpon using the data, we further realised that we needed to aggregate the news data by date for each day.  Since there are instances of multiple news surrounding a single company in a day, this made complicated calculations in Tableau.  Hence we aggregated the data and added a total number of news articles column, and averaged the values if there are more than one news entry per company per day.<jupyter_code>combined_df = pd.read_csv('data/combined_data_v2.csv', index_col=0)
combined2016_df = pd.read_csv('data/combined2016_data_v2.csv', index_col=0)
combined_df.head(30)
dfc2016mean = combined2016_df.groupby(['date','assetCode']).mean().reset_index()
dfc2016mean.shape
dfc2016sum = combined2016_df.groupby(['date','assetCode']).sum().reset_index()
dfc2016sum.shape
dfc2016count = combined2016_df.groupby(['date','assetCode']).timeNews.count().reset_index()
dfc2016count.shape
a0 = combined2016_df[combined2016_df.date == '2016-10-17']
a0[a0.assetCode == 'NFLX.O']
a1 = dfc2016mean[dfc2016mean.date == '2016-10-17']
a1[a1.assetCode == 'NFLX.O']
a2 = dfc2016sum[dfc2016sum.date == '2016-10-17']
a2[a2.assetCode == 'NFLX.O']
a3 = dfc2016count[dfc2016count.date == '2016-10-17']
a3[a3.assetCode == 'NFLX.O']
b = combined2016_df[combined2016_df.date == '2016-10-17']
b[b.assetCode == 'NFLX.O'].wordCount.sum()
# Getting groupby objects and getting the 
dfc2016mean = combined2016_df.groupby(['date','assetCode', 'assetName']).mean().reset_index()
dfc2016sum = combined2016_df.groupby(['date','assetCode', 'assetName']).sum().reset_index()
dfc2016count = combined2016_df.groupby(['date','assetCode', 'assetName']).timeNews.count().reset_index()

# Using mean as the basis to start since most fields mean is the most relevant
dfc2016 = dfc2016mean.copy(deep=True)

# inserting new field number of news articles
dfc2016.insert(loc=dfc2016.columns.get_loc('universe')+1, 
                   column='newsArticles', 
                   value=dfc2016count['timeNews'])

# Changing the following mean values to sum
dfc2016['wordCount'] = dfc2016sum['wordCount']
dfc2016['sentimentWordCount'] = dfc2016sum['sentimentWordCount']

# Re-evaluting the sentimentRating
dfc2016.loc[(dfc2016.sentimentClass < 0), 'sentimentRating'] = -1* dfc2016[dfc2016.sentimentClass < 0].sentimentNegative
dfc2016.loc[(dfc2016.sentimentClass == 0), 'sentimentRating'] = 0
dfc2016.loc[(dfc2016.sentimentClass > 0), 'sentimentRating'] = dfc2016[dfc2016.sentimentClass > 0].sentimentPositive
# Checking...
aX = dfc2016[dfc2016.date == '2016-10-17']
aX[aX.assetCode == 'NFLX.O']
# Getting groupby objects and getting the 
dfcmean = combined_df.groupby(['date','assetCode', 'assetName']).mean().reset_index()
dfcsum = combined_df.groupby(['date','assetCode', 'assetName']).sum().reset_index()
dfccount = combined_df.groupby(['date','assetCode', 'assetName']).timeNews.count().reset_index()

# Using mean as the basis to start since most fields mean is the most relevant
dfc = dfcmean.copy(deep=True)

# inserting new field number of news articles
dfc.insert(loc=dfc.columns.get_loc('universe')+1, 
                   column='newsArticles', 
                   value=dfccount['timeNews'])

# Changing the following mean values to sum
dfc['wordCount'] = dfcsum['wordCount']
dfc['sentimentWordCount'] = dfcsum['sentimentWordCount']

# Re-evaluting the sentimentRating
dfc.loc[(dfc.sentimentClass < 0), 'sentimentRating'] = -1* dfc[dfc.sentimentClass < 0].sentimentNegative
dfc.loc[(dfc.sentimentClass == 0), 'sentimentRating'] = 0
dfc.loc[(dfc.sentimentClass > 0), 'sentimentRating'] = dfc[dfc.sentimentClass > 0].sentimentPositive
# Checking...
bX = dfc[dfc.date == '2016-10-17']
bX[bX.assetCode == 'NFLX.O']
# Saving to csv

dfc.to_csv('data/combined_data_v3.csv')
dfc2016.to_csv('data/combined2016_data_v3.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13098180184228264, 'fraction_numerical': 0.020310042687036622, 'mean_word_length': 7.096034921789742, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '662960', 'license': 'MIT', 'n_tokens_mistral': 7906, 'n_tokens_neox': 7397, 'n_words': 1598, 'path': 'w209_final_project_data_processing.ipynb', 'repo_name': 'kuangweihuang/kuangweihuang.github.io', 'size': '22270'}