<jupyter_start><jupyter_text>Data Pre-Processing-FY20 **Collecting the data -** data consists of budget text documents in the form of PDF files obtained from the following organizations:    * [Guilford County](https://www.guilfordcountync.gov/home/showdocument?id=9497)   * [Durham County](https://www.dconc.gov/home/showdocument?id=27985)   * [City of Durham](https://durhamnc.gov/DocumentCenter/View/27412/FY20-Final-Budget)   * [City of Charlotte](https://charlottenc.gov/budget/FY2020%20Documents/FY%202020%20Adopted%20Budget%20Book%207-31%20Complete.pdf)   * [Mecklenburg County](https://www.mecknc.gov/CountyManagersOffice/OMB/Documents/FY2020%20Adopted%20Budget.pdf)    * [Wake County](http://www.wakegov.com/budget/fy20/Documents/FY20%20Adopted%20Budget%20Book.pdf)   * [City of Raleigh](https://user-2081353526.cld.bz/FY2020AdoptedBudget)   After the PDF files are collected, they are compressed to reduce the size. Then, files are converted into CSV files using an app developed by project mentor:           **[<NAME>](https://www.linkedin.com/in/jones-jason-adam/),**           **click [here](https://jason-jones.shinyapps.io/Emotionizer/) for the App**<jupyter_code>#Importing packages
import os
import glob
import nltk
import pandas as pd
import numpy as np
# change the current directory to read the data
os.chdir(r"C:\Users\Sultan\Desktop\data\FY2020\structured\original")<jupyter_output><empty_output><jupyter_text>Reading and labling data for all organizations<jupyter_code># 1- Reading Guilford-County data file 
GC_df = pd.read_csv("GuilfordCountyOriginalDataFY20.csv", engine='python')
# inserting "organization" column with static value 
# corresponding to the organization in question 
GC_df.insert(2, "organization", "Guilford County")


# 2- For Charlotte-City data
CC_df = pd.read_csv(r'CharlotteCityOriginalDataFY20.csv', engine='python')
CC_df.insert(2, "organization", "Charlotte City")

# 3- For Durham-City data
DCity_df = pd.read_csv(r'DurhamCityOriginalDataFY20.csv', engine='python')
DCity_df.insert(2, "organization", "Durham City")

# 4- For Durham-County data
DCounty_df = pd.read_csv(r'DurhamCountyOriginalDataFY20.csv', engine='python')
DCounty_df.insert(2, "organization", "Durham County")

# 5- For Mecklenburg-County data
MC_df = pd.read_csv(r'MecklenburgCountyOriginalDataFY20.csv', engine='python')
MC_df.insert(2, "organization", "Mecklenburg County")

# 6- For Raleigh-City data
RC_df = pd.read_csv(r'RaleighCityOriginalDataFY20.csv', engine='python')
RC_df.insert(2, "organization", "Raleigh City")

# 7- For Wake-County data
WC_df = pd.read_csv(r'WakeCountyOriginalDataFY20.csv', engine='python')
WC_df.insert(2, "organization", "Wake County")

# Combine all dataframes into a single dataframe using concat() function
# Row lables are adjusted automaticlly by passing ignore_index=True
df =  pd.concat([GC_df, CC_df, DCity_df, 
                   DCounty_df, MC_df, RC_df, WC_df], ignore_index=True)
df.head()
# listing columns in data frame 
list(df)<jupyter_output><empty_output><jupyter_text>Dropping and reordering columns<jupyter_code># delete columns using the columns parameter of drop
df = df.drop(columns="Unnamed: 0")

# re-order columns
df = df[['page_number','word','organization']]

df.head()<jupyter_output><empty_output><jupyter_text>Adding "Year" column with a static value corresponding to the year in question<jupyter_code>df.insert(3, "year", "FY2020")
df.head()<jupyter_output><empty_output><jupyter_text>Dataframe to one single csv file<jupyter_code># Change the dirctory for file to be stored properly
os.chdir(r"C:\Users\Sultan\Desktop\data\PreprocessedData") 

# Export dataframe to csv
df.to_csv(r'DataFY20.csv', index=False, encoding='utf-8-sig')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12670045345425446, 'fraction_numerical': 0.03094158442251267, 'mean_word_length': 6.183908045977011, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 8, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '162164', 'license': 'MIT', 'n_tokens_mistral': 1413, 'n_tokens_neox': 1262, 'n_words': 282, 'path': 'src/DataPreprocessing/original/Data-Preprocessing-FY20.ipynb', 'repo_name': 'naseebth/Budget_Text_Analysis', 'size': '3754'}