<jupyter_start><jupyter_text>Data MergingWhat if we find more data that we would like to add to our analysis? This notebook quickly works through some of the considerations you'll have to make when merging datasets. Primarily, in order to merge different datasets with different data fields, there are two general methods. With one method, we might keep every column we encounter. If a dataset lacks that particular column, we could keep the column but fill the rows with NA values. Another method, the one we'll use, is to keep only the column we have values for in the entire dataset.<jupyter_code>import re
import pandas as pd

abs_dir = "/Users/williamquinn/Desktop/DH/Python/Teaching/Python-Notebooks/"<jupyter_output><empty_output><jupyter_text>Loading Old Dataframe<jupyter_code>data = pd.read_csv(abs_dir + "data/fake-and-real-news-dataset/dataframe.csv",
                   sep = ",")

data.head()<jupyter_output><empty_output><jupyter_text>Loading New Dataframe<jupyter_code>new_data = pd.read_csv(abs_dir + "data/fake-news-dataset-2/fake.csv",
                       sep = ",")

print (new_data.shape)
new_data.head(2)<jupyter_output>(12999, 20)<jupyter_text>Making Column AdjustmentsWe can immediately tell that this new dataset had different columns. While this data may be interesting to include in another analysis, we'll want to keep only the columns that are also in the old dataframe.<jupyter_code># We'll first select only the columns to keep.
# And rename the published column name to date.
new_data = new_data[['title', 'text', 'published']] \
    .rename(columns = {'published':'date'})

# The date field is in a datetime but a different format. We'll use a regex to keep only year-month-day.
# The \\1 is our first group. 
# The group (\\1) returns only the regex that appears within the parenthesis.
new_data['date'] = new_data['date'].replace(r'(\d{4}-\d{2}-\d{2}).*', '\\1', regex = True)

# We'll want the date field to be in the same format as the old dataframe, too.
new_data['date'] = pd.to_datetime(new_data['date'], format = '%Y-%m-%d')

# We also know that these articles are fake.
new_data['veracity'] = 'fake'

# While we don't know the "subject" of each entry, we can create our own value.
new_data['subject'] = 'fake_news'

# It could also be useful to keep track of which reports come from which dataset.
data['dataset'] = 0
new_data['dataset'] = 1

new_data.head()<jupyter_output><empty_output><jupyter_text>Merging Old and New Dataframes<jupyter_code>data = pd.concat([data, new_data], sort=False)

# Sometimes, text has integers and floats. 
# In order to run functions later, we'll want to ensure these values are strings.
data['text'] = data['text'].astype(str)

print (data.shape)
data.head()<jupyter_output>(57887, 6)<jupyter_text>Save Complete DatasetWe'll now save the entire dataset. Because we're joining the data from two different locations on our computers (subdirectories), I'm going to save this in a new location (one directory above each of these).<jupyter_code>data.to_csv(abs_dir + "data/dataframe.csv",  
            index=False)<jupyter_output><empty_output><jupyter_text>Reading Text FieldsBefore we finish, though, let's read through some of the articles to look for any other issues we might want to clean. We'll look at two articles from the same day to see how they compare. Real News Report<jupyter_code>data.query('(veracity == "real") & (date == "2017-10-05")')['text'].values[0]<jupyter_output><empty_output><jupyter_text>Fake News Report<jupyter_code>data.query('(veracity == "fake") & (date == "2017-09-09")')['text'].values[0]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11080485115766263, 'fraction_numerical': 0.011300992282249173, 'mean_word_length': 5.018242122719735, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '18461', 'license': 'MIT', 'n_tokens_mistral': 1153, 'n_tokens_neox': 1097, 'n_words': 405, 'path': '05_Data-Merging.ipynb', 'repo_name': 'billquinn/Teaching-Python', 'size': '3628'}