<jupyter_start><jupyter_text>4.01: Converting R Data Into Feather Files---<jupyter_code>install.packages("rio")<jupyter_output>also installing the dependency ‘openxlsx’

Updating HTML index of packages in '.Library'
Making 'packages.html' ... done<jupyter_text>1\. Importing Files with `rio`[Helpful resource](https://cran.r-project.org/web/packages/rio/vignettes/rio.html):>  Supported file formats>> rio supports a variety of different file formats for import and export. To keep the package slim, all non-essential formats are supported via “Suggests” packages, which are not installed (or loaded) by default. To ensure rio is fully functional, install these packages the first time you use rio via:>> ```Rinstall_formats()```FormatTypical ExtensionImport PackageExport PackageInstalled by DefaultComma-separated data.csvdata.tabledata.tableYesPipe-separated data.psvdata.tabledata.tableYesTab-separated data.tsvdata.tabledata.tableYesSAS.sas7bdathavenhavenYesSPSS.savhavenhavenYesStata.dtahavenhavenYesSAS XPORT.xpthavenhavenYesSPSS Portable.porhavenYesExcel.xlsreadxlYesExcel.xlsxreadxlopenxlsxYesR syntax.RbasebaseYesSaved R objects.RData, .rdabasebaseYesSerialized R objects.rdsbasebaseYesEpiinfo.recforeignYesMinitab.mtpforeignYesSystat.sydforeignYes“XBASE” database files.dbfforeignforeignYesWeka Attribute-Relation File Format.arffforeignforeignYesData Interchange Format.difutilsYesFortran datano recognized extensionutilsYesFixed-width format data.fwfutilsutilsYesgzip comma-separated data.csv.gzutilsutilsYesCSVY (CSV + YAML metadata header).csvycsvycsvyNoFeather R/Python interchange format.featherfeatherfeatherNoFast Storage.fstfstfstNoJSON.jsonjsonlitejsonliteNoMatlab.matrmatiormatioNoOpenDocument Spreadsheet.odsreadODSreadODSNoHTML Tables.htmlxml2xml2NoShallow XML documents.xmlxml2xml2NoYAML.ymlyamlyamlNoClipboarddefault is tsvcliprcliprNoGoogle Sheetsas Comma-separated data So, using the above table, we see that there's no need to install any new packages to import .csv, .rds, .stata or .xls(x) files. This is quite helpful to us. Let's keep the following info handy:* .csv: `data.table`* .rds, .rda, .R, .RData: `base`* .sav, .dta: `haven`* .xls, .xlsx: `readxl`, `openxlsx`<jupyter_code># First import
library("rio")
cces_cumul_0616_rds <- import("/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/CCES_cumulative_2016/cumulative_2006_2016.Rds")
cces_cumul_0616_dta <- import("/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/CCES_cumulative_2016/cumulative_2006_2016.dta")
all.equal(cces_cumul_0616_rds, cces_cumul_0616_dta, check.attributes = FALSE)
# Next imports: Supplementary info from Red State-Blue State study
library("rio")
cps_income <- import_list("../../../Large Datasets/capstone/QJPS_6026_supp/cps.income.Rdata")
annen2004_processed <- import("../../../Large Datasets/capstone/QJPS_6026_supp/annen2004_processed.dta")
annen061229 <- import("../../../Large Datasets/capstone/QJPS_6026_supp/annen061229.dta")
avgincome_orig <- import("../../../Large Datasets/capstone/QJPS_6026_supp/avgincome_orig.dta")
fips_icpsr_cbs_naes <- import("../../../Large Datasets/capstone/QJPS_6026_supp/fips.icpsr.cbs.naes.dta")
region_dummies_84 <- import("../../../Large Datasets/capstone/QJPS_6026_supp/region_dummies_84.dta")
region_dummies <- import("../../../Large Datasets/capstone/QJPS_6026_supp/region_dummies.dta")
region_indic_annen2000 <- import("../../../Large Datasets/capstone/QJPS_6026_supp/region_indic_annen2000.dta")
labprocrace_1984 <- import("../../../Large Datasets/capstone/QJPS_6026_supp/1984_labeled_processed_race.dta")
labprocrace_1988 <- import("../../../Large Datasets/capstone/QJPS_6026_supp/1988_labeled_processed_race.dta")
labprocrace_1992 <- import("../../../Large Datasets/capstone/QJPS_6026_supp/1992_labeled_processed_race.dta")
labprocrace_1996 <- import("../../../Large Datasets/capstone/QJPS_6026_supp/1996_labeled_processed_race.dta")
labprocrace_2000 <- import("../../../Large Datasets/capstone/QJPS_6026_supp/2000_labeled_processed_race.dta")
labprocrace_2004 <- import("../../../Large Datasets/capstone/QJPS_6026_supp/2004_labeled_processed_race.dta")
# ANES cumulative timeseries data; more historically complete version of some of above data
anes_cumul_dta <- import("/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/anes_timeseries_cdf_dta/anes_timeseries_cdf.dta")
# ANES 2016 timeseries data
anes_2016votval_dta <- import("/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/anes_timeseries_2016_dta/anes_timeseries_2016_voteval.dta")
anes_2016_dta <- import("/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/anes_timeseries_2016_dta/anes_timeseries_2016.dta")<jupyter_output><empty_output><jupyter_text>2\. Exporting Select Dataframes to CSV or Other Preferred FormatOne idea is the [following code recipe](http://stat.ethz.ch/R-manual/R-devel/library/utils/html/write.table.html) for CSV files:> ```Rwrite.table(x, file = "", append = FALSE, quote = TRUE, sep = " ",            eol = "\n", na = "NA", dec = ".", row.names = TRUE,            col.names = TRUE, qmethod = c("escape", "double"),            fileEncoding = "")write.csv(...)write.csv2(...)```Another idea is to use `feather` to export the data into a universal format.<jupyter_code>install.packages("feather")
library("feather")
library("data.table")
library("base")
library("haven")
summary(cces_cumul_0616_rds)
write_feather(cces_cumul_0616_rds, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/CCES_cumulative_2016/cces_cumul_0616_rds.feather")<jupyter_output><empty_output><jupyter_text>Let's make a list of export commands:```Rwrite_feather(foo, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/filename.bar")```1. Add `write_feather(` to the start of each line.2. Replace ` , ` with `, `.3. Replace directory after `/capstone/` with `/feather/`.4. Replace file extension with `.feather`. ```Rwrite_feather(cces_cumul_0616_dta, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/cumulative_2006_2016_dta.feather")write_feather(cps_income, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/cps.income.feather")write_feather(annen2004_processed, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/annen2004_processed.feather")write_feather(annen061229, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/annen061229.feather")write_feather(avgincome_orig, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/avgincome_orig.feather")write_feather(fips_icpsr_cbs_naes, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/fips.icpsr.cbs.naes.feather")write_feather(region_dummies_84, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/region_dummies_84.feather")write_feather(region_dummies, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/region_dummies.feather")write_feather(region_indic_annen2000, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/region_indic_annen2000.feather")write_feather(labprocrace_1984, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/1984_labeled_processed_race.feather")write_feather(labprocrace_1988, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/1988_labeled_processed_race.feather")write_feather(labprocrace_1992, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/1992_labeled_processed_race.feather")write_feather(labprocrace_1996, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/1996_labeled_processed_race.feather")write_feather(labprocrace_2000, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/2000_labeled_processed_race.feather")write_feather(labprocrace_2004, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/2004_labeled_processed_race.feather") ANES cumulative timeseries data; more historically complete version of some of above datawrite_feather(anes_cumul_dta, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/anes_timeseries_cdf.feather") ANES 2016 timeseries datawrite_feather(anes_2016votval_dta, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/anes_timeseries_2016_voteval.feather")write_feather(anes_2016_dta, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/anes_timeseries_2016.feather")```<jupyter_code>write_feather(cces_cumul_0616_dta, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/cumulative_2006_2016_dta.feather")
write_feather(annen2004_processed, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/annen2004_processed.feather")
write_feather(annen061229, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/annen061229.feather")
write_feather(avgincome_orig, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/avgincome_orig.feather")
write_feather(fips_icpsr_cbs_naes, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/fips.icpsr.cbs.naes.feather")
write_feather(region_dummies_84, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/region_dummies_84.feather")
write_feather(region_dummies, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/region_dummies.feather")
write_feather(region_indic_annen2000, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/region_indic_annen2000.feather")
write_feather(labprocrace_1984, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/1984_labeled_processed_race.feather")
write_feather(labprocrace_1988, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/1988_labeled_processed_race.feather")
write_feather(labprocrace_1992, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/1992_labeled_processed_race.feather")
write_feather(labprocrace_1996, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/1996_labeled_processed_race.feather")
write_feather(labprocrace_2000, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/2000_labeled_processed_race.feather")
write_feather(labprocrace_2004, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/2004_labeled_processed_race.feather")
# ANES cumulative timeseries data; more historically complete version of some of above data
write_feather(anes_cumul_dta, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/anes_timeseries_cdf.feather")
# ANES 2016 timeseries data
write_feather(anes_2016votval_dta, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/anes_timeseries_2016_voteval.feather")
write_feather(anes_2016_dta, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/anes_timeseries_2016.feather")
cps_income <- load("/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/QJPS_6026_supp/cps.income.Rdata")
write_feather(cps.data, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/cps_income_data.feather")
write_feather(cps.data, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/cps_income_data.feather")
write_feather(cps.stats, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/cps_income_stats.feather")
write_feather(cps.stats.all, "/Users/pk/Documents/Data Science/2018 Winter/Large Datasets/capstone/feather/cps_income_stats_all.feather")
cps_income_data <- as.data.frame(cps.data)
library("rio")
library("feather")
library("data.table")
library("base")
library("haven")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11701507453985176, 'fraction_numerical': 0.054218372615974016, 'mean_word_length': 6.968148639681487, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 22, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 5}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '599197', 'license': 'MIT', 'n_tokens_mistral': 4665, 'n_tokens_neox': 4032, 'n_words': 594, 'path': '01-R-data-to-feather.ipynb', 'repo_name': 'kolliparap/DSI-capstone-project', 'size': '12007'}