<jupyter_start><jupyter_text>Combined-Data-PreProcessing1. **Collecting the data -** data consists of budget text documents in the form of PDF files obtained from the following organizations:    * [Guilford County](https://www.guilfordcountync.gov/our-county/budget-management-evaluation)   * [Durham County](https://www.dconc.gov/government/departments-a-e/budget-management-services)   * [City of Durham](https://durhamnc.gov/199/Budget-Management-Services)   * [City of Charlotte](https://charlottenc.gov/budget/Pages/default.aspx)   * [Mecklenburg County](https://www.mecknc.gov/CountyManagersOffice/OMB/Pages/Home.aspx)   * [Wake County](http://www.wakegov.com/budget/Pages/default.aspx)   * [City of Raleigh](https://www.raleighnc.gov/home/content/Departments/Articles/BudgetManagement.html)   After the PDF files are collected, they are compressed to reduce the their sizes. Then, the files are tokenized, and converted into CSV files using an app developed by project mentor:           **[<NAME>](https://www.linkedin.com/in/jones-jason-adam/),**           **click [here](https://jason-jones.shinyapps.io/Emotionizer/) for the App**       2. **Cleaning the data -** performing some popular text pre-processing techniques3. **Organizing the data -** organizing the cleaned data into a way that is easy to input into other algorithms<jupyter_code>import os
import glob
import nltk
import pandas as pd
import numpy as np
# Change the current directory to read the data
os.chdir(r"C:\Users\Sultan\Desktop\data\PreprocessedData")<jupyter_output><empty_output><jupyter_text>1- Obtaining the data  Transforming the csv files into dataframes<jupyter_code>FY13_df = pd.read_csv(r'DataFY13.csv', engine='python')
FY14_df = pd.read_csv(r'DataFY14.csv', engine='python')
FY15_df = pd.read_csv(r'DataFY15.csv', engine='python')
FY16_df = pd.read_csv(r'DataFY16.csv', engine='python')
FY17_df = pd.read_csv(r'DataFY17.csv', engine='python')
FY18_df = pd.read_csv(r'DataFY18.csv', engine='python')
FY19_df = pd.read_csv(r'DataFY19.csv', engine='python')
FY20_df = pd.read_csv(r'DataFY20.csv', engine='python')
# Combine all dataframes into a single dataframe using concat() function
# Row lables are adjusted automaticlly by passing ignore_index=True

df =  pd.concat([FY13_df, FY14_df, FY15_df, FY16_df, FY17_df, 
                  FY18_df, FY19_df, FY20_df], ignore_index=True)
df.head()<jupyter_output><empty_output><jupyter_text>2- Cleaning the Data<jupyter_code># listing columns in data frame 
list(df)<jupyter_output><empty_output><jupyter_text>Dropping and reordering columns<jupyter_code># delete columns using the columns parameter of drop
df = df.drop(columns="ï»¿page_number")
df.head()<jupyter_output><empty_output><jupyter_text>Removing stop words<jupyter_code># Import stop words from nltk 
from nltk.corpus import stopwords

# Define variable stop
stop = stopwords.words('english')

df['word'] = df['word'].apply(lambda w: " ".join(w for w in w.split() if w not in stop))
df.head(60)<jupyter_output><empty_output><jupyter_text>Removing punctuations<jupyter_code>df['word'] = df['word'].str.replace('[^\w\s]','')
df.head(60)<jupyter_output><empty_output><jupyter_text>Removing short words<jupyter_code>df['word'] = df['word'].apply(lambda x: ' '.join([x for x in x.split() if len(x) > 3]))
df.head(60)<jupyter_output><empty_output><jupyter_text>Handling missing text data<jupyter_code># Replace any empty strings in the 'word' column with np.nan objects
df['word'].replace('', np.nan, inplace=True)

# Drop all NaN values
df.dropna(subset=['word'], inplace=True)
df.head(20)<jupyter_output><empty_output><jupyter_text>Lowercasing<jupyter_code>df['word'] = df['word'].str.lower()
df.head()<jupyter_output><empty_output><jupyter_text>2- Organizing the Data We already created a corpus in an earlier step. The definition of a corpus is a collection of texts, and they are all put together neatly in a pandas dataframe here.<jupyter_code># Let's take a look at our dataframe
df.head(60)<jupyter_output><empty_output><jupyter_text>Dataframe to one single and clean csv file<jupyter_code># Change the dirctory for file to be exported to the proper folder
os.chdir(r"C:\Users\Sultan\Desktop\data\PreprocessedData") 

# Export dataframe to csv
df.to_csv(r"CombinedData.csv", index=False, encoding='utf-8-sig')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13113620649919336, 'fraction_numerical': 0.015902281631712378, 'mean_word_length': 6.1381578947368425, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 47, 'https://': 8, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '14814', 'license': 'MIT', 'n_tokens_mistral': 1575, 'n_tokens_neox': 1473, 'n_words': 350, 'path': 'src/DataPreprocessing/original/Combined-Data-Preprocessing.ipynb', 'repo_name': 'naseebth/Budget_Text_Analysis', 'size': '4344'}