<jupyter_start><jupyter_text>Introduction Overview PlanAt the end of this session, you will have learned how to:* Take advantage of the verbs and syntax you learned from the `dplyr` module to manipulate `RxXdfData` data objects* Summarize your `RxXdfData` objects quickly and easily* Create custom functions and use them for mutations and summarizations* Understand where and when to use the `dplyrXdf` package and when to use functions from the `RevoScaleR` package The Microsoft R Family![Microsoft R Family](images/mr-family.png) Microsoft R Component Stack![Microsoft R Family](images/mrcomponents.png)ScaleR: suite of HPA functions for data manipulation and modeling, plus some custom HPC functionalityConnectR: high speed and direct connectorsDistributedR: framework for cross-platform distributed computationDeployR: web service development kit through APIs, java, js, .net Why dplyrXdf? Simplify Your Analysis Pipeline* The `RevoScaleR` package enables R users to manipulate data that is larger than memory* It introduces a new data type, called an `xdf` (short for eXternal Data Frame), which are highly efficient out-of-memory objects* However, many of the `RevoScaleR` functions have a dramatically different syntax from base R functions* The `dplyr` package is an exceptionally popular, due to its appealing syntax, and it's extensibility Simpler Analysis with dplyrXdf* The `dplyrXdf` that exposes most of the `dplyr` functionality to `xdf` objects* Many data analysis pipelines require creating many intermediate datasets, which are only needed for their role in deriving a final dataset, but have no/little use on their own* The `dplyrXdf` abstracts this task of file management, so that you can focus on the data itself, rather than the management of intermediate files* Unlike `dplyr`, or other base R packages, `dplyrXdf` allows you to work with data residing _outside_ of memory, and therefore scales to datasets of arbitrary size Requirements What You'll Need* I expect that you have already covered the `dplyr` training* Understand the *XDF* data type and how to import data to *XDF** If you're working on a different computer than your trianer: have (`devtools`)[github.com/hadley/devtools] (and if on a Windows machine, [Rtools](https://cran.r-project.org/bin/windows/Rtools/)) Installing dplyrXdf* The `dplyrXdf` package is not yet on CRAN* You have to download it from [github](https://github.com/RevolutionAnalytics/dplyrXdf/)  - if you're on a windows machine, install [Rtools](https://cran.r-project.org/bin/windows/Rtools/) as well  - the `devtools` package provides a very handy function, `install_github`, for installing R packages saved in github repositories Create XDF from taxi data Create a local directory to save XDF<jupyter_code>your_name <- "alizaidi"
your_dir <- paste0('/datadrive/', your_name)
# File Path to your Data
your_data <- file.path(your_dir, 'tripdata_2015.xdf')
dir.create(your_dir)
download.file("http://alizaidi.blob.core.windows.net/training/yellow_tripdata_2015.xdf",
              destfile = your_data)<jupyter_output><empty_output><jupyter_text>Create a Pointer to XDF<jupyter_code>library(dplyrXdf)
taxi_xdf <- RxXdfData(your_data)
taxi_xdf %>% head
taxi_xdf %>% nrow
class(taxi_xdf)<jupyter_output><empty_output><jupyter_text>Simplified Pipelines for Data Summaries Data Transforms The rxDataStep Way* All the functionality exposed by the `dplyrXdf` package can also be completedby using the `rxDataStep` function in the `RevoScaleR` package included with your MRS installation* In fact, `dplyrXdf` consists almost entirely of wrapper functions that call on other RevoScaleR functions* Let's compare the workflow for adding a new column to a dataset with `rxDataStep` vs `dplyrXdf`---<jupyter_code>taxi_xdf %>% rxGetInfo(getVarInfo = TRUE, numRows = 4)<jupyter_output><empty_output><jupyter_text>---<jupyter_code>taxi_transform <- RxXdfData(your_data)<jupyter_output><empty_output><jupyter_text>---<jupyter_code>system.time(rxDataStep(inData = taxi_xdf,
           outFile = taxi_transform,
           transforms = list(tip_pct = tip_amount/fare_amount),
           overwrite = TRUE))<jupyter_output><empty_output><jupyter_text>Data Transforms The rxDataStep Way<jupyter_code>rxGetInfo(RxXdfData(taxi_transform), numRows = 2)<jupyter_output><empty_output><jupyter_text>Data Transforms The dplyrXdf Way* We could do the same operation with `dplyrXdf`, using the exact same syntaxthat we learned in the `dplyr` module and taking advantage of the `%>%` operator<jupyter_code>system.time(taxi_transform <- taxi_xdf %>% mutate(tip_pct = tip_amount/fare_amount))
taxi_transform %>% rxGetInfo(numRows = 2)<jupyter_output><empty_output><jupyter_text>Differences* The major difference between the `rxDataStep` operation and the `dplyrXdf` method, is that we do not specify an `outFile` argument anywhere in the `dplyrXdf` pipeline* In our case, we have assigned our `mutate` value to a new variable called `taxi_transform`* This creates a temporary file to save the intermediate `xdf`, and only saves the most recent output of a pipeline, where a pipeline is defined as all operations starting from a raw xdf file.* To copy an *xdf* from the temporary directory to permanent storage, use the `persist` verb---<jupyter_code>taxi_transform@file
persist(taxi_transform, outFile = "/datadrive/alizaidi/taxiTransform.xdf") -> taxi_transform<jupyter_output><empty_output><jupyter_text>Using dplyrXdf for Aggregations dplyrXdf Way* The `dplyrXdf` package really shines when used for data aggregations and summarizations* Whereas `rxSummary`, `rxCube`, and `rxCrossTabs` can compute a few summary statistics and do aggregations very quickly, they are not sufficiently general to be used in all places---<jupyter_code>taxi_group <- taxi_transform %>%
  group_by(pickup_nhood) %>%
taxi_group %>% head<jupyter_output><empty_output><jupyter_text>Using dplyrXdf for Aggregations rxCube Way* The above could have been done with `rxCube` as well, but would require additional considerations* We would have to make sure that the `pickup_nhood` column was a factor (can't mutate in place because of different data types)* `rxCube` can only provide summations and averages, so we cannot get standard deviations for instance.* Creating your own factors is never a pleasant experience. You may feel like everything is going right until![faceplant](http://www.ohmagif.com/wp-content/uploads/2015/02/dude-front-flip-epic-face-plant.gif)---<jupyter_code>rxFactors(inData = taxi_transform,
          outFile = "/datadrive/alizaidi/taxi_factor.xdf",
          factorInfo = c("pickup_nhood"),
          overwrite = TRUE)
head(rxCube(tip_pct ~ pickup_nhood,
            means = TRUE,
            data = "/datadrive/alizaidi/taxi_factor.xdf"))
# file.remove("data/taxi_factor.xdf")<jupyter_output><empty_output><jupyter_text>Creating Functional Pipelines with dplyrXdfAs we saw above, it's pretty easy to create a summarization or aggregation script. We can encapsulate our aggregation into it's own function.Suppose we wanted to calculate average tip as a function of dropoff and pickup neighborhoods. In the `dplyr` nonmenclature, this means grouping by dropoff and pickup neighborhoods, and summarizing/averaging tip percent.<jupyter_code>rxGetInfo(taxi_transform, numRows = 5)<jupyter_output><empty_output><jupyter_text>---<jupyter_code>mht_url <- "http://alizaidi.blob.core.windows.net/training/manhattan.rds"
manhattan_hoods <- readRDS(gzcon(url(mht_url)))<jupyter_output><empty_output><jupyter_text>---<jupyter_code>taxi_transform %>%
    filter(pickup_nhood %in% mht_hoods,
           dropoff_nhood %in% mht_hoods,
           .rxArgs = list(transformObjects = list(mht_hoods = manhattan_hoods))) %>%
    group_by(dropoff_nhood, pickup_nhood) %>%
    summarize(ave_tip = mean(tip_pct),
              ave_dist = mean(trip_distance)) %>%
    filter(ave_dist > 3, ave_tip > 0.05) -> sum_df<jupyter_output><empty_output><jupyter_text>---<jupyter_code>sum_df %>% rxGetInfo(getVarInfo = TRUE, numRows = 5)
class(sum_df)<jupyter_output><empty_output><jupyter_text>---Alternatively, we can encapsulate this script into a function, so that we can easily call it in a functional pipeline.<jupyter_code>taxi_hood_sum <- function(taxi_data = taxi_df, ...) {

  taxi_data %>%
    filter(pickup_nhood %in% manhattan_hoods,
           dropoff_nhood %in% manhattan_hoods, ...) %>%
    group_by(dropoff_nhood, pickup_nhood) %>%
    summarize(ave_tip = mean(tip_pct),
              ave_dist = mean(trip_distance)) %>%
    filter(ave_dist > 3, ave_tip > 0.05) -> sum_df

  return(sum_df)

}<jupyter_output><empty_output><jupyter_text>---The resulting summary object isn't very large (about 408 rows in this case), so it shouldn't cause any memory overhead issues if we covert it now to a `data.frame`. We can plot our results using our favorite plotting library.<jupyter_code>tile_plot_hood <- function(df = taxi_hood_sum()) {

  library(ggplot2)

  ggplot(data = df, aes(x = pickup_nhood, y = dropoff_nhood)) +
    geom_tile(aes(fill = ave_tip), colour = "white") +
    theme_bw() +
    theme(axis.text.x = element_text(angle = 45, hjust = 1),
          legend.position = 'bottom') +
    scale_fill_gradient(low = "white", high = "steelblue") -> gplot

  return(gplot)
}<jupyter_output><empty_output><jupyter_text>---<jupyter_code># tile_plot_hood(as.data.frame(sum_df))
taxi_transform <- taxi_xdf %>% mutate(tip_pct = tip_amount/fare_amount)
library(plotly)
sum_df <- taxi_hood_sum(taxi_transform,
                        .rxArgs = list(transformObjects = list(manhattan_hoods = manhattan_hoods))) %>%
  persist("/datadrive/alizaidi/summarized.xdf")
ggplotly(tile_plot_hood(as.data.frame(sum_df)))<jupyter_output><empty_output><jupyter_text>Split and Combining Operations with doXdf Custom functions across groupsThe `do` verb is an exception to the rule that dplyrXdf verbs write their output as xdf files. This is because do executes arbitrary R code, and can return arbitrary R objects; while a data frame is capable of storing these objects, an xdf file is limited to character and numeric vectors only. Custom functions across groupsThe doXdf verb is similar to do, but where do splits its input into one data frame per group, doXdf splits it into one xdf file per group. This allows do-like functionality with grouped data, where each group can be arbitrarily large. The syntax for the two functions is essentially the same, although the code passed to doXdf must obviously know how to handle xdfs.---<jupyter_code>taxi_models <- taxi_xdf %>% group_by(pickup_dow) %>% doXdf(model = rxLinMod(tip_amount ~ fare_amount, data = .))
taxi_models
taxi_models$model[[1]]<jupyter_output><empty_output><jupyter_text>Memory IssuesAll the caveats that go with working with `data.frames` apply here. While each grouped partition is it's own `RxXdfData` object, the return value must be a `data.frame`, and hence, must fit in memory.Moreover, the function you apply against the splits will determine how they are operated. If you use an `rx` function, you'll get the nice fault-tolerant, parallel execution strategies the `RevoScaleR` package provides, but for any vanilla/CRAN function will work with data.frames and can easily cause your session to crash.---<jupyter_code>library(broom)
taxi_broom <- taxi_xdf %>% group_by(pickup_dow) %>% doXdf(model = lm(tip_amount ~ fare_amount, data = .))<jupyter_output><empty_output><jupyter_text>Now we can apply the `broom::tidy` function at the row level to get summary statistics:<jupyter_code>library(broom)
tbl_df(taxi_broom) %>% tidy(model)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08894623655913979, 'fraction_numerical': 0.003096774193548387, 'mean_word_length': 5.487723214285714, 'pattern_counts': {'":': 0, '<': 96, '<?xml version=': 0, '>': 114, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '281386', 'license': 'MIT', 'n_tokens_mistral': 3683, 'n_tokens_neox': 3504, 'n_words': 1204, 'path': 'Student-Resources/notebooks/2-dplyrXdf.ipynb', 'repo_name': 'karl-kuno/mr4ds', 'size': '11625'}