<jupyter_start><jupyter_text>![](https://github.com/ashishpatel26/Rapidsai_Machine_learning_on_GPU/raw/main/images/rapidsailogo.jpg?raw=true)🤩**About Rapids**🤩- The RAPIDS suite of open source software libraries aim to enable execution of end-to-end data science and analytics pipelines entirely on GPUs. It relies on NVIDIA® CUDA® primitives for low-level compute optimization, but exposing that GPU parallelism and high-bandwidth memory speed through user-friendly Python interfaces.![](https://github.com/rapidsai/cudf/raw/branch-21.08/img/rapids_arrow.png)  **What are these Libraries?*** [cuDF](https://github.com/rapidsai/cudf) is a Python GPU DataFrame library (built on the Apache Arrow columnar memory format) for loading, joining, aggregating, filtering, and otherwise manipulating tabular data using a DataFrame style API.* [Dask](https://dask.org/) is a flexible library for parallel computing in Python that makes scaling out your workflow smooth and simple. On the CPU, Dask uses Pandas to execute operations in parallel on DataFrame partitions.* [Dask-cuDF](https://github.com/rapidsai/cudf/tree/main/python/dask_cudfhttps://github.com/rapidsai/cudf/tree/main/python/dask_cudf) extends Dask where necessary to allow its DataFrame partitions to be processed by cuDF GPU DataFrames as opposed to Pandas DataFrames. For instance, when you call dask_cudf.read_csv(…), your cluster’s GPUs do the work of parsing the CSV file(s) with underlying cudf.read_csv(). **When to use cuDF and Dask-cuDF*** If your workflow is fast enough on a single GPU or your data comfortably fits in memory on a single GPU, you would want to use cuDF. If you want to distribute your workflow across multiple GPUs, have more data than you can fit in memory on a single GPU, or want to analyze data spread across many files at once, you would want to use Dask-cuDF.  10 Min to Pandas and cuDF Comparision on GPU🎯 Built based on the Apache Arrow columnar memory format, cuDF is a GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data.🎯 cuDF provides a pandas-like API that will be familiar to data engineers & data scientists, so they can use it to easily accelerate their workflows without going into the details of CUDA programming.  **1.Installation of cuDF**> 🏹**Conda installation** - 📥Recommended```console for CUDA 11.0conda install -c rapidsai -c nvidia -c numba -c conda-forge \    cudf=21.06 python=3.7 cudatoolkit=11.0 or, for CUDA 11.2conda install -c rapidsai -c nvidia -c numba -c conda-forge \    cudf=21.06 python=3.7 cudatoolkit=11.2```> 🏹**For Nightly Version**```console for CUDA 11.0conda install -c rapidsai-nightly -c nvidia -c numba -c conda-forge \    cudf python=3.7 cudatoolkit=11.0 or, for CUDA 11.2conda install -c rapidsai-nightly -c nvidia -c numba -c conda-forge \    cudf python=3.7 cudatoolkit=11.2```> 🏹**PIP Installation**```console!pip install cudf```> 🏹**Colab Installation** - [![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1rY7Ln6rEE1pOlfSHCYOVaqt8OvDO35J0forceEdit=true&offline=true&sandboxMode=true)<jupyter_code># Install RAPIDS
!git clone https://github.com/rapidsai/rapidsai-csp-utils.git
!bash rapidsai-csp-utils/colab/rapids-colab.sh stable

import sys, os

dist_package_index = sys.path.index('/usr/local/lib/python3.7/dist-packages')
sys.path = sys.path[:dist_package_index] + ['/usr/local/lib/python3.7/site-packages'] + sys.path[dist_package_index:]
sys.path
exec(open('rapidsai-csp-utils/colab/update_modules.py').read(), globals())<jupyter_output><empty_output><jupyter_text>**2.Import Libraries**<jupyter_code>import os
import cupy as cp
import pandas as pd
import cudf
import dask_cudf
import time

cp.random.seed(2021)
print(cudf.__version__)
print(dask_cudf.__version__)
print(pd.__version__)<jupyter_output>0.19.2
0.19.2
1.1.5<jupyter_text>**3.Object Creation**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 3.84ms| 9.6ms| 18.5ms |<jupyter_code>%%time
# 1. Pandas
s = pd.Series([1,2,3,None, 4])
print(s)
%%time
# 2. cudf
s = cudf.Series([1,2,3,None, 4])
print(s)
%%time
ds = dask_cudf.from_cudf(s, npartitions=2)
print(ds.compute())<jupyter_output>0       1
1       2
2       3
3    <NA>
4       4
dtype: int64
CPU times: user 8.22 ms, sys: 1.77 ms, total: 10 ms
Wall time: 18.5 ms<jupyter_text>**4.Loading Dataset**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 2.42s| 1.52s| 1.62s |<jupyter_code>%%time
# 1.Pandas
data = pd.read_csv("https://storage.googleapis.com/industryanalytics/LoanDefaultData.csv")
display(data.head(2))
%%time
# 2.cudf
data_cudf = cudf.read_csv("https://storage.googleapis.com/industryanalytics/LoanDefaultData.csv")
display(data_cudf.head(2))
%%time
# 3.dask_cudf
data_daskcudf = dask_cudf.from_cudf(cudf.read_csv("https://storage.googleapis.com/industryanalytics/LoanDefaultData.csv"), npartitions=8)
display(data_daskcudf.head(2))<jupyter_output><empty_output><jupyter_text>**5.Convert Pandas data to cudf and dask_cudf**| Framework  | Pandas to cudf | Pandas to daskcudf || ---- | ---- | ---- | | Time | 2.73s| 2.84s|<jupyter_code>%%time
# 1. cudf from pandas
data = pd.read_csv("https://storage.googleapis.com/industryanalytics/LoanDefaultData.csv")
cudf_data = cudf.DataFrame.from_pandas(data)
cudf_data
%%time
# 2. dask_cudf from pandas
data = pd.read_csv("https://storage.googleapis.com/industryanalytics/LoanDefaultData.csv")
cudf_data = cudf.DataFrame.from_pandas(data)
daskcudf_data = dask_cudf.from_cudf(cudf_data, npartitions=2)<jupyter_output>CPU times: user 2.11 s, sys: 252 ms, total: 2.36 s
Wall time: 2.79 s<jupyter_text>**6.Viewing Data**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 241 µs| 4.32ms| 12.7ms |<jupyter_code>%%time
# 1.Pandas
data.head()
%%time
# 2.cudf
cudf_data.head()
%%time
# 3.daskcudf
daskcudf_data.head()<jupyter_output>CPU times: user 9.35 ms, sys: 0 ns, total: 9.35 ms
Wall time: 12.7 ms<jupyter_text>**7.Sorting Values**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 149ms| 37.8ms| 4.67s |<jupyter_code>%%time
# 1.Pandas
data.sort_values('cust_id').head()
%%time
# 2.cudf
cudf_data.sort_values('cust_id').head()
%%time
# 3.daskcudf
daskcudf_data.sort_values('cust_id').head()<jupyter_output>CPU times: user 3.75 s, sys: 211 ms, total: 3.96 s
Wall time: 4.67 s<jupyter_text>**8.Selection**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 46.3µs| 1.18ms| 7.37ms |<jupyter_code>%%time
# 1. Pandas
data["state"]
%%time
# 2. cudf
cudf_data["state"]
%%time
# 3. cudf
daskcudf_data["state"].compute()<jupyter_output>CPU times: user 5.27 ms, sys: 873 µs, total: 6.15 ms
Wall time: 7.37 ms<jupyter_text>**9.Selection By label**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 1.4ms| 1.82ms| 7.45ms |<jupyter_code>%%time
# 1.Pandas
data.loc[1:100, ["state","income_type"]]
%%time
# 2.cudf
cudf_data.loc[1:100, ["state","income_type"]]
%%time
# 3.daskcudf
daskcudf_data.loc[1:100, ["state","income_type"]].compute()<jupyter_output>CPU times: user 7.13 ms, sys: 0 ns, total: 7.13 ms
Wall time: 7.45 ms<jupyter_text>**10.Selection by Position**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 1.4ms| 1.82ms| 30.1ms |<jupyter_code>%%time
# 1.Pandas
data.iloc[4:10, 0:2].head()
%%time
# 2.cudf
cudf_data.iloc[4:10, 0:2].head()
%%time
# 3.daskcudf
daskcudf_data.compute().iloc[4:10, 0:2].head()<jupyter_output>CPU times: user 14 ms, sys: 14.9 ms, total: 28.9 ms
Wall time: 30.1 ms<jupyter_text>**11.Boolean Indexing**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 182ms| 34.7ms| 9.8ms |<jupyter_code>%%time
# 1. Pandas
data[data.cust_id>83000]
%%time
# 2. cudf
cudf_data[cudf_data.cust_id>83000]
%%time
# 3. daskcudf
daskcudf_data[daskcudf_data.cust_id>83000]<jupyter_output>CPU times: user 3.59 ms, sys: 0 ns, total: 3.59 ms
Wall time: 9.8 ms<jupyter_text>**12.Query API**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 189ms| 178ms| 75.5ms |<jupyter_code>%%time
# 1. pandas
data.query("cust_id > 83000")
%%time
# 2. cudf
cudf_data.query("cust_id > 83000")
%%time
# 3. daskcudf
daskcudf_data.query("cust_id > 83000").compute()<jupyter_output>CPU times: user 52.2 ms, sys: 22.3 ms, total: 74.5 ms
Wall time: 75.7 ms<jupyter_text>**13.`isin` Method**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 49.9ms| 44.2ms| 67.5ms |<jupyter_code>%%time
# 1. Pandas
data[data.state.isin(["Bihar", "Odisha"])]
%%time
# 2. cudf
cudf_data[cudf_data.state.isin(["Bihar", "Odisha"])]
%%time
# 3. daskcudf
daskcudf_data[daskcudf_data.state.isin(["Bihar", "Odisha"])].compute()<jupyter_output>CPU times: user 55.6 ms, sys: 11.1 ms, total: 66.8 ms
Wall time: 67.5 ms<jupyter_text>**14.MultiIndex**| Framework  | Pandas | cudf || ---- | ---- | ---- | | Time | 5.9ms| 12.9ms|<jupyter_code>%%time
# 1. Pandas
arrays = [['a', 'a', 'b', 'b'], [1, 2, 3, 4]]
tuples = list(zip(*arrays))
idx = pd.MultiIndex.from_tuples(tuples)
display(idx)
%%time
# 2. cudf
arrays = [['a', 'a', 'b', 'b'], [1, 2, 3, 4]]
tuples = list(zip(*arrays))
idx = cudf.MultiIndex.from_tuples(tuples)
display(idx)<jupyter_output><empty_output><jupyter_text>**15.Missing Data**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 380ms| 29.8ms| 85.4ms |<jupyter_code>%%time
# 1. Pandas
data.fillna(999)
%%time
# 2.cudf
cudf_data.fillna(999)
%%time
# 3.cudf
daskcudf_data.fillna(999).compute()<jupyter_output>CPU times: user 44.2 ms, sys: 38.7 ms, total: 82.9 ms
Wall time: 85.4 ms<jupyter_text>**16.Statistics Operation**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 5.7ms| 4.51ms| 651ms |<jupyter_code>%%time
# 1.Pandas
data.total_pymnt.mean(), data.total_pymnt.var()
%%time
# 2.cudf
cudf_data.total_pymnt.mean(), cudf_data.total_pymnt.var()
%%time
# 3.daskcudf
daskcudf_data.total_pymnt.mean().compute(), daskcudf_data.total_pymnt.var().compute()<jupyter_output>CPU times: user 640 ms, sys: 13.5 ms, total: 653 ms
Wall time: 651 ms<jupyter_text>**17.Applymap**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 211ms| 1.84ms| 7.16ms |<jupyter_code>%%time
# 1.Pandas
def add(n):
  return n + 5
data["total_pymnt"].apply(add)
%%time
# 2.cudf
def add(n):
  return n + 5
cudf_data["total_pymnt"].applymap(add)
%%time
# 3.daskcudf
def add(n):
  return n + 5
daskcudf_data["total_pymnt"].compute().applymap(add)<jupyter_output>CPU times: user 4.26 ms, sys: 2.21 ms, total: 6.48 ms
Wall time: 7.16 ms<jupyter_text>**18. Histogramming**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 68.4ms| 23.5ms| 36.5ms |<jupyter_code>%%time
# 1. Pandas
data["state"].value_counts()
%%time
# 2. cudf
cudf_data["state"].value_counts()
%%time
# 3. daskcudf
daskcudf_data["state"].value_counts().compute()<jupyter_output>CPU times: user 31.8 ms, sys: 2.07 ms, total: 33.9 ms
Wall time: 36.5 ms<jupyter_text>**19.String Methods**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 3.59ms| 6.17ms| 16.4ms |<jupyter_code>states = data.state.unique().tolist()
%%time
# 1.Pandas
s = pd.Series(states)
print(s.str.lower())
%%time
# 2.cudf
s_cudf = cudf.Series(states)
print(s_cudf.str.lower())
%%time
# 3.daskcudf
s_daskcudf = dask_cudf.from_cudf(cudf.Series(states), npartitions=2)
print(s_daskcudf.str.lower().compute())<jupyter_output>0        andhra pradesh
1             rajasthan
2               manipur
3     arunachal pradesh
4          chhattisgarh
5               mizoram
6               gujarat
7               tripura
8               haryana
9             jharkhand
10                assam
11               sikkim
12               punjab
13            telangana
14          maharashtra
15            karnataka
16                  goa
17             nagaland
18     himachal pradesh
19        uttar pradesh
20          west bengal
21                bihar
22           tamil nadu
23               kerala
24       madhya pradesh
25            meghalaya
26          uttarakhand
27               odisha
dtype: object
CPU times: user 13.4 ms, sys: 998 µs, total: 14.4 ms
Wall time: 16.4 ms<jupyter_text>**20.Concat**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 1.41ms| 2.28ms| 11.1ms |<jupyter_code>%%time
# 1. Pandas
pd.concat([s,s])
%%time
# 2. cudf
cudf.concat([s_cudf,s_cudf])
%%time
# 3. daskcudf
dask_cudf.concat([s_daskcudf,s_daskcudf]).compute()<jupyter_output>CPU times: user 9.56 ms, sys: 0 ns, total: 9.56 ms
Wall time: 11.1 ms<jupyter_text>**21.Join**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 15.7ms| 9.05ms| 30.2ms |<jupyter_code>%%time
# 1.Pandas
df_a = pd.DataFrame()
df_a['key'] = ['a', 'b', '<KEY>']
df_a['vals_a'] = [float(i + 10) for i in range(5)]

df_b = pd.DataFrame()
df_b['key'] = ['a', 'c', 'e']
df_b['vals_b'] = [float(i+100) for i in range(3)]

merged = df_a.merge(df_b, on=['key'], how='left')
merged.head()
%%time
# 2.cudf
df_a = cudf.DataFrame()
df_a['key'] = ['a', 'b', 'c', 'd', 'e']
df_a['vals_a'] = [float(i + 10) for i in range(5)]

df_b = cudf.DataFrame()
df_b['key'] = ['a', 'c', 'e']
df_b['vals_b'] = [float(i+100) for i in range(3)]

merged = df_a.merge(df_b, on=['key'], how='left')
merged.head()
%%time
# 3.daskcudf
ddf_a = dask_cudf.from_cudf(df_a, npartitions=2)
ddf_b = dask_cudf.from_cudf(df_b, npartitions=2)

merged = ddf_a.merge(ddf_b, on=['key'], how='left').compute()
merged<jupyter_output>CPU times: user 29.1 ms, sys: 812 µs, total: 29.9 ms
Wall time: 30.2 ms<jupyter_text>**22.Append**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 574µs| 4.26ms| 30.2ms |<jupyter_code>%%time
# 1. Pandas
s.append(s)
%%time
# 2. cudf
s_cudf.append(s_cudf)
%%time
# 3.daskcudf
s_daskcudf.append(s_daskcudf).compute()<jupyter_output>CPU times: user 8.67 ms, sys: 0 ns, total: 8.67 ms
Wall time: 9.61 ms<jupyter_text>**23.Grouping**| Framework  | Pandas | cudf || ---- | ---- | ---- | | Time | 792ms| 270ms|<jupyter_code>%%time
# 1.Pandas
data["agg_col1"] = [1 if x % 2 == 0 else 0 for x in range(len(data["total_pymnt"]))]
data["agg_col2"] = [1 if x % 2 == 0 else 0 for x in range(len(data["total_pymnt"]))]

data.groupby('agg_col1').sum()
%%time
# 2.cudf
cudf_data["agg_col1"] = [1 if x % 2 == 0 else 0 for x in range(len(cudf_data["total_pymnt"]))]
cudf_data["agg_col2"] = [1 if x % 2 == 0 else 0 for x in range(len(cudf_data["total_pymnt"]))]
cudf_data.groupby('agg_col1').sum()<jupyter_output>CPU times: user 255 ms, sys: 17.9 ms, total: 273 ms
Wall time: 270 ms<jupyter_text>**24.Transpose**| Framework  | Pandas | cudf | | ---- | ---- | ---- | | Time | 1.1ms| 3.13ms|<jupyter_code>%%time
# 1.Pandas
sample = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})
sample.transpose()
%%time
# 2.cudf
sample_cudf = cudf.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})
sample_cudf.transpose()<jupyter_output>CPU times: user 3.15 ms, sys: 0 ns, total: 3.15 ms
Wall time: 3.13 ms<jupyter_text>**25.Time Series**| Framework  | Pandas | cudf | daskcudf || ---- | ---- | ---- | ---- || Time | 213ms| 330ms| 79ms |<jupyter_code>%%time
# 1.Pandas
import datetime as dt

data["date_issued"] = pd.to_datetime(data["date_issued"])
search_date = dt.datetime.strptime('2015-11-23', '%Y-%m-%d')
display(data.query('date_issued <= @search_date'))
%%time
# 2.cudf
import datetime as dt

cudf_data["date_issued"] = cudf.to_datetime(cudf_data["date_issued"])
search_date = dt.datetime.strptime('2015-11-23', '%Y-%m-%d')
display(cudf_data.query('date_issued <= @search_date'))
%%time
# 3.daskcudf
date_ddf = dask_cudf.from_cudf(cudf_data, npartitions=2)
date_ddf.query('date_issued <= @search_date', local_dict={'search_date':search_date}).compute()<jupyter_output>CPU times: user 42.8 ms, sys: 35.5 ms, total: 78.3 ms
Wall time: 79.2 ms
{'fraction_non_alphanumeric': 0.18013552515999498, 'fraction_numerical': 0.051449366294390765, 'mean_word_length': 3.9607843137254903, 'pattern_counts': {'":': 0, '<': 84, '<?xml version=': 0, '>': 91, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '66439', 'license': 'MIT', 'n_tokens_mistral': 7303, 'n_tokens_neox': 6544, 'n_words': 1463, 'path': '10_Min_to_Pandas_and_Cudf_Comparision_on_GPU.ipynb', 'repo_name': 'ashishpatel26/Rapidsai_Machine_learnring_on_GPU', 'size': '15944'}