<jupyter_start><jupyter_text>Setup Let's setup Spark on your Colab environment.  Run the cell below!<jupyter_code>!pip install pyspark
!apt update
!pip install -U -q PyDrive
!apt install openjdk-8-jdk-headless -qq
import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"<jupyter_output>Requirement already satisfied: pyspark in /usr/local/lib/python3.7/dist-packages (3.1.1)
Requirement already satisfied: py4j==0.10.9 in /usr/local/lib/python3.7/dist-packages (from pyspark) (0.10.9)
Get:1 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease [3,626 B]
Ign:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease
Ign:3 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease
Get:4 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease [15.9 kB]
Get:5 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release [697 B]
Hit:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release
Get:7 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:8 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release.gpg [836 B]
Hit:9 http://archive.ubuntu.com/ubuntu bionic InRelease[...]<jupyter_text>Now we import some of the libraries usually needed by our workload.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

import pyspark
from pyspark.sql import *
from pyspark.sql.types import *
from pyspark.sql.functions import *
from pyspark import SparkContext, SparkConf
from pyspark.sql.functions import lit<jupyter_output><empty_output><jupyter_text>Mounting the drice and making the project folder as the current path<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
import os
cur_path = "/content/drive/My Drive/bigdataproj/"
os.chdir(cur_path)
!ls<jupyter_output>Big_data_project.ipynb	FR_videos_data.csv  KR_videos_data.csv
BR_videos_data.csv	GB_videos_data.csv  MX_videos_data.csv
CA_videos_data.csv	IN_videos_data.csv  RU_videos_data.csv
DE_videos_data.csv	JP_videos_data.csv  US_videos_data.csv<jupyter_text>Let's initialize the Spark context.<jupyter_code># create the session
conf = SparkConf().set("spark.ui.port", "4050")

# create the context
sc = pyspark.SparkContext(conf=conf)
spark = SparkSession.builder.getOrCreate()<jupyter_output><empty_output><jupyter_text>You can easily check the current version and get the link of the web interface. In the Spark UI, you can monitor the progress of your job and debug the performance bottlenecks (if your Colab is running with a **local runtime**).<jupyter_code>spark<jupyter_output><empty_output><jupyter_text>Data Preprocessing For convenience, given that the dataset is small, we first construct a Pandas dataframe, tune the schema, and then convert it into a Spark dataframe.  Importing all data sets<jupyter_code>!ls<jupyter_output>Big_data_project.ipynb	FR_videos_data.csv  KR_videos_data.csv
BR_videos_data.csv	GB_videos_data.csv  MX_videos_data.csv
CA_videos_data.csv	IN_videos_data.csv  RU_videos_data.csv
DE_videos_data.csv	JP_videos_data.csv  US_videos_data.csv<jupyter_text>Importing all the datasets for different countries.1. In a a Pandas data frame format2. In a spark data frame format<jupyter_code># video_* shows the pandas data frame
# df_* shows the spark data frame


video_data=pd.read_csv('US_videos_data.csv')
df = spark.read.csv('US_videos_data.csv',header=True)

video_data_BR=pd.read_csv('BR_videos_data.csv')
df_BR = spark.read.csv('BR_videos_data.csv',header=True)

video_data_CA=pd.read_csv('CA_videos_data.csv')
df_CA = spark.read.csv('CA_videos_data.csv',header=True)

video_data_DE=pd.read_csv('DE_videos_data.csv')
df_DE = spark.read.csv('DE_videos_data.csv',header=True)

video_data_FR=pd.read_csv('FR_videos_data.csv')
df_FR = spark.read.csv('FR_videos_data.csv',header=True)

video_data_GB=pd.read_csv('GB_videos_data.csv')
df_GB = spark.read.csv('GB_videos_data.csv',header=True)

video_data_IN=pd.read_csv('IN_videos_data.csv')
df_IN = spark.read.csv('IN_videos_data.csv',header=True)

video_data_JP=pd.read_csv('JP_videos_data.csv')
df_JP = spark.read.csv('JP_videos_data.csv',header=True)

video_data_KR=pd.read_csv('KR_videos_data.csv')
df_KR = spark.read.csv('KR_videos_data.csv',header=True)

video_data_MX=pd.read_csv('MX_videos_data.csv')
df_MX = spark.read.csv('MX_videos_data.csv',header=True)

video_data_RU=pd.read_csv('RU_videos_data.csv')
df_RU = spark.read.csv('RU_videos_data.csv',header=True)<jupyter_output><empty_output><jupyter_text>***About the Data***The dataset contains a total of 16 columns describing interesting features such as Title, date/time of upload, description of video, and also files were categorized based on the region of Trending, this way it is easy to see which demographics have similarities.Currently, It contains data from 10 Regions/Countries. Distinct Columnsvideo_idtitle	publishedAt	channelId	channelTitle	categoryId	trending_date	tags	view_count	likes	dislikes	comment_count	thumbnail_link	comments_disabled	ratings_disabled	description  ***US ***Initial EDA on US to see the number of views against different channel title. Basically to see which channel title get the most views.**The same methodology will be repeated for different countries in ordert to get the most trending channels.<jupyter_code>video_data.head()
df.show()<jupyter_output>+--------------------+-----------+--------------------+--------------------+--------------------+--------------------+----------+-------------+--------------------+----------+------+--------+-------------+--------------------+-----------------+----------------+--------------------+
|                 _c0|   video_id|               title|         publishedAt|           channelId|        channelTitle|categoryId|trending_date|                tags|view_count| likes|dislikes|comment_count|      thumbnail_link|comments_disabled|ratings_disabled|         description|
+--------------------+-----------+--------------------+--------------------+--------------------+--------------------+----------+-------------+--------------------+----------+------+--------+-------------+--------------------+-----------------+----------------+--------------------+
|                   0|IhO1FcjDMV4|Meatpacking: Last...|2021-02-22T07:30:01Z|UC3XTzVzaHQEd30rQ...|     LastWeekTonight|        24|     21.22.02|        [...]<jupyter_text>Lets filter out the columns that we think would be the most important both in terms of understanding the problem at hand and also making predictions.In addition change the data type to the desired type view counts, like, dislike, comment count, cateforyID as INTpublished and trending as dates<jupyter_code>"""
Here we are converting the columns to desired data type
"""

df=df.withColumn("view_count", df["view_count"].cast("int"))\
.withColumn("likes", df["likes"].cast("int"))\
.withColumn("dislikes", df["dislikes"].cast("int"))\
.withColumn("comment_count", df["comment_count"].cast("int"))\
.withColumn("trending_date", to_timestamp(df["trending_date"], "yy.dd.MM"))\
.withColumn("time_published", to_timestamp(df["publishedAt"], "yyyy-MM-dd"))\
.withColumn("categoryId", df["categoryId"].cast('int'))\
.select('view_count','title','tags','channelTitle','categoryId','likes','dislikes','comment_count')

Regression_data=df
Regression_data.show()<jupyter_output>+----------+--------------------+--------------------+--------------------+----------+------+--------+-------------+
|view_count|               title|                tags|        channelTitle|categoryId| likes|dislikes|comment_count|
+----------+--------------------+--------------------+--------------------+----------+------+--------+-------------+
|   1067147|Meatpacking: Last...|              [none]|     LastWeekTonight|        24| 60111|    1221|         6866|
|   1047854|Best 3D Pen Art W...|              [none]|          ZHC Crafts|        26| 50662|     690|         4879|
|   6133266|100 Days - [Minec...|luke thenotable|l...|     Luke TheNotable|        20|372753|    7961|        39551|
|    790238|Amazing! <NAME>ry...|American Idol|sin...|       American Idol|        24| 14267|     129|         1257|
|   3248661|Game Theory: Did ...|fnaf|five nights ...|  The Game Theorists|        20|225780|    2872|        31885|
|    297852|Deion Sanders win...|deion sanders|dei...|ESPN Colle[...]<jupyter_text>let's see the count against every category IDTherefore, we group it by category ID and the sum all views across the dfThe we will do the same for every channel title i.e. grouping on channel title in order to see the number of views<jupyter_code>df_category=df.groupBy('categoryId').sum().select('categoryId','sum(view_count)')
df_1=df.groupBy('channelTitle').sum()
df_1.orderBy(df_1['sum(view_count)'].desc()).limit(10).show()<jupyter_output>+----------------+---------------+---------------+----------+-------------+------------------+
|    channelTitle|sum(view_count)|sum(categoryId)|sum(likes)|sum(dislikes)|sum(comment_count)|
+----------------+---------------+---------------+----------+-------------+------------------+
|       BLACKPINK|      980149319|            390|  91593351|      1000541|          13620062|
|         MrBeast|      508233148|            480|  30808435|       301393|           1841741|
|            NASA|      251983185|            644|   6503560|       127019|            181139|
|  MrBeast Shorts|      239524743|            638|  14929481|       237397|            694824|
|JustinBieberVEVO|      203196414|            180|  14238196|       281495|            991264|
|      <NAME>|      199578766|            300|  10524777|        82342|            629219|
|           Dream|      195730673|            200|  13577320|       236611|            998831|
|       LazarBeam|      191755192|            600|[...]<jupyter_text>Views per channel IDBLACKPINK is the most trending channel followed by MrBeast in terms of number of views in the US<jupyter_code># Plotting total views per channel ID

plt.clf()
df_1.orderBy(df_1['sum(view_count)'].desc()).limit(10).toPandas().plot(x='channelTitle',y='sum(view_count)',kind='bar')
display()<jupyter_output><empty_output><jupyter_text>***Brazil***Lets repeat the same process for Brazil The aim is to see the total views against a channel.The details written for the above view count for US will be followed exactly in order to get the view count for Brazil by channel title. Checking the data: columns<jupyter_code>video_data_BR.head()
df_BR.show()<jupyter_output>+---+-----------+-----------------------+--------------------+--------------------+--------------------+----------+-------------+--------------------+----------+------+--------+-------------+--------------------+-----------------+----------------+--------------------+
|_c0|   video_id|                  title|         publishedAt|           channelId|        channelTitle|categoryId|trending_date|                tags|view_count| likes|dislikes|comment_count|      thumbnail_link|comments_disabled|ratings_disabled|         description|
+---+-----------+-----------------------+--------------------+--------------------+--------------------+----------+-------------+--------------------+----------+------+--------+-------------+--------------------+-----------------+----------------+--------------------+
|  0|cSxfqLr27mA|    LIVE NOSSA HISTÓRIA|2021-02-22T00:55:43Z|UCmRkuNc6evsLa3JK...|           Taty Girl|        10|     21.22.02|Taty Girl|Gospel|...|    508325| 29158|     518|          654|ht[...]<jupyter_text>Checking the data types<jupyter_code>df_BR.summary()<jupyter_output><empty_output><jupyter_text>Converting the data types into the desired data types and sub setting necessary columns as mentioned earlier<jupyter_code>df_BR=df_BR.withColumn("view_count", df_BR["view_count"].cast("int"))\
.withColumn("likes", df_BR["likes"].cast("int"))\
.withColumn("dislikes", df_BR["dislikes"].cast("int"))\
.withColumn("comment_count", df_BR["comment_count"].cast("int"))\
.withColumn("trending_date", to_timestamp(df_BR["trending_date"], "yy.dd.MM"))\
.withColumn("time_published", to_timestamp(df_BR["publishedAt"], "yyyy-MM-dd"))\
.withColumn("categoryId", df_BR["categoryId"].cast('int'))\
.select('view_count','title','tags','channelTitle','categoryId','likes','dislikes','comment_count')

Regression_data_df_BR=df_BR
Regression_data_df_BR.show()<jupyter_output>+----------+-----------------------+--------------------+--------------------+----------+------+--------+-------------+
|view_count|                  title|                tags|        channelTitle|categoryId| likes|dislikes|comment_count|
+----------+-----------------------+--------------------+--------------------+----------+------+--------+-------------+
|    508325|    LIVE NOSSA HISTÓRIA|Taty Girl|Gospel|...|           Taty Girl|        10| 29158|     518|          654|
|    631324|   JÁ TESTEI O NOVO ...|Free Fire|PlayHar...|            PlayHard|        20| 92319|     961|         3009|
|    423886|   TOUR PELO MEU APA...|              [none]|       <NAME>|        23| 81366|     407|         2395|
|   2390949|   LBFF 4 - Rodada 1...|garena|free|fire|...|  Free Fire - Brasil|        22|227436|    1241|         1583|
|    273428|   Coletiva de impre...|Clube de Regatas ...|              FLA TV|        17| 27715|     210|          557|
|    255391|   Flamengo a uma vi...|    [...]<jupyter_text>checking the necessary view counts by cateogry id and channel title<jupyter_code>df_BR_category=df_BR.groupBy('categoryId').sum().select('categoryId','sum(view_count)')
df_BR_1=df_BR.groupBy('channelTitle').sum()
df_BR_1.orderBy(df_BR_1['sum(view_count)'].desc()).limit(10).show()<jupyter_output>+--------------------+---------------+---------------+----------+-------------+------------------+
|        channelTitle|sum(view_count)|sum(categoryId)|sum(likes)|sum(dislikes)|sum(comment_count)|
+--------------------+---------------+---------------+----------+-------------+------------------+
|           BLACKPINK|      929403847|            340|  87992458|       941584|          12937958|
|                NASA|      298664453|            672|   7644454|       149194|            232762|
|              SMTOWN|      293396674|            230|  25689030|       208848|           3775261|
|          <NAME>|      274809022|            190|  13307021|       110523|            747607|
|    JustinBieberVEVO|      247419944|            210|  16730636|       334051|           1178008|
|         Brawl Stars|      216728687|            200|  10030915|       102445|           1264003|
|<NAME>...|      210143128|            350|   9275718|        88904|            138461|
|         [...]<jupyter_text>Plotting a histogram to see the most trending channel in BrazilAs you can see like America BLACKPINK is the most trending channel in Brazil<jupyter_code>plt.clf()
df_BR_1.orderBy(df_BR_1['sum(view_count)'].desc()).limit(10).toPandas().plot(x='channelTitle',y='sum(view_count)',kind='bar')
display()<jupyter_output><empty_output><jupyter_text>***Japan***Let's look at the most trending channel in japan Connvert the data types and take a look at the data setMost of it is in Japenese so I wouldn't be able to make sense of what is written inside :P<jupyter_code>df_JP=df_JP.withColumn("view_count", df_JP["view_count"].cast("int"))\
.withColumn("likes", df_JP["likes"].cast("int"))\
.withColumn("dislikes", df_JP["dislikes"].cast("int"))\
.withColumn("comment_count", df_JP["comment_count"].cast("int"))\
.withColumn("trending_date", to_timestamp(df_JP["trending_date"], "yy.dd.MM"))\
.withColumn("time_published", to_timestamp(df_JP["publishedAt"], "yyyy-MM-dd"))\
.withColumn("categoryId", df_JP["categoryId"].cast('int'))\
.select('view_count','title','tags','channelTitle','categoryId','likes','dislikes','comment_count')

Regression_data_df_JP=df_JP
Regression_data_df_JP.show()<jupyter_output>+----------+-------------------------------------+------------------------------------+-------------------------------+----------+-----+--------+-------------+
|view_count|                                title|                                tags|                   channelTitle|categoryId|likes|dislikes|comment_count|
+----------+-------------------------------------+------------------------------------+-------------------------------+----------+-----+--------+-------------+
|    381483|【新企画】宮迫さんと居酒屋堤下。本...| 宮迫さん|居酒屋堤下|コラボ|堤下食堂|                       堤下食堂|        26|12342|     823|         1209|
|    148446| 【齊藤京子（日向坂４６）×日高屋】...|            きょんこ|味噌チゲ|日高家|           日高屋公式チャンネル|        24| 7797|     109|          736|
|    821488|【アニメ遠井さん】やる気なさすぎな...| すとぷり|すとろべりーぷりんす|さ...|             すとぷりちゃんねる|        24|57909|     397|         3414|
|    708094|【カメラ】はじめてのクッキー作り！...|   莉犬|莉犬くん|りけんくん|すとぷり|     莉犬くんのサブちゃんねる！|        23|68104|     313|         7466|
|    534528|       シブヤ・ディビジョン“Fling ...|[...]<jupyter_text>Aggregate total number of views on the basis of channel title and category<jupyter_code>df_JP_category=df_JP.groupBy('categoryId').sum().select('categoryId','sum(view_count)')
df_JP_1=df_JP.groupBy('channelTitle').sum()
df_JP_1.orderBy(df_JP_1['sum(view_count)'].desc()).limit(10).show()<jupyter_output>+----------------+---------------+---------------+----------+-------------+------------------+
|    channelTitle|sum(view_count)|sum(categoryId)|sum(likes)|sum(dislikes)|sum(comment_count)|
+----------------+---------------+---------------+----------+-------------+------------------+
|       BLACKPINK|      938914542|            270|  83038240|       948482|          13143830|
|      <NAME>|      184218556|             80|   9163209|        77485|            523360|
|    東海オンエア|      180043060|           2070|   3808820|        63588|            378766|
|          SMTOWN|      176168399|             90|  11077753|       117920|           1418929|
|            NASA|      170547471|            308|   4242795|        81274|            165373|
|JustinBieberVEVO|      165285061|            110|  11080648|       229075|            794920|
|       Daft Punk|      165255269|             80|  10650441|       194102|           1260783|
|     Brawl Stars|      163861531|            160|   786[...]<jupyter_text>Lets see the most trending channelAs you can see like America blackpink is the most trending channel in japan<jupyter_code>plt.clf()
df_JP_1.orderBy(df_JP_1['sum(view_count)'].desc()).limit(10).toPandas().plot(x='channelTitle',y='sum(view_count)',kind='bar')
display()<jupyter_output>/usr/local/lib/python3.7/dist-packages/matplotlib/backends/backend_agg.py:214: RuntimeWarning: Glyph 26481 missing from current font.
  font.set_text(s, 0.0, flags=flags)
/usr/local/lib/python3.7/dist-packages/matplotlib/backends/backend_agg.py:214: RuntimeWarning: Glyph 28023 missing from current font.
  font.set_text(s, 0.0, flags=flags)
/usr/local/lib/python3.7/dist-packages/matplotlib/backends/backend_agg.py:214: RuntimeWarning: Glyph 12458 missing from current font.
  font.set_text(s, 0.0, flags=flags)
/usr/local/lib/python3.7/dist-packages/matplotlib/backends/backend_agg.py:214: RuntimeWarning: Glyph 12531 missing from current font.
  font.set_text(s, 0.0, flags=flags)
/usr/local/lib/python3.7/dist-packages/matplotlib/backends/backend_agg.py:214: RuntimeWarning: Glyph 12456 missing from current font.
  font.set_text(s, 0.0, flags=flags)
/usr/local/lib/python3.7/dist-packages/matplotlib/backends/backend_agg.py:214: RuntimeWarning: Glyph 12450 missing from current font.
  font.set_[...]<jupyter_text>***India***lets repeat the same process for India and see the total number of views per channel and category and then plot it to see the most trending channel.1. Convert data type of columns2. group by channel title and category3. plor the count against channel title<jupyter_code>df_IN=df_IN.withColumn("view_count", df_IN["view_count"].cast("int"))\
.withColumn("likes", df_IN["likes"].cast("int"))\
.withColumn("dislikes", df_IN["dislikes"].cast("int"))\
.withColumn("comment_count", df_IN["comment_count"].cast("int"))\
.withColumn("trending_date", to_timestamp(df_IN["trending_date"], "yy.dd.MM"))\
.withColumn("time_published", to_timestamp(df_IN["publishedAt"], "yyyy-MM-dd"))\
.withColumn("categoryId", df_IN["categoryId"].cast('int'))\
.select('view_count','title','tags','channelTitle','categoryId','likes','dislikes','comment_count')

Regression_data_df_IN=df_IN
Regression_data_df_IN.show()
df_IN_category=df_IN.groupBy('categoryId').sum().select('categoryId','sum(view_count)')
df_IN_1=df_IN.groupBy('channelTitle').sum()
df_IN_1.orderBy(df_IN_1['sum(view_count)'].desc()).limit(10).show()<jupyter_output>+------------------+---------------+---------------+----------+-------------+------------------+
|      channelTitle|sum(view_count)|sum(categoryId)|sum(likes)|sum(dislikes)|sum(comment_count)|
+------------------+---------------+---------------+----------+-------------+------------------+
|          T-Series|     1089022325|            530|  37467218|      1310027|           2177770|
|  Sony Music India|      843730243|            360|  15173052|      1552926|           2058362|
|         BLACKPINK|      638460209|            240|  63344838|       623681|          10774929|
|Desi Music Factory|      404880064|            250|  18539942|      1842963|           2609207|
|      Aditya Music|      335768062|            300|  10271106|       298812|            489674|
|  Vijay Television|      274406639|           2160|   6481969|        85487|            144414|
|         Crazy XYZ|      239584122|           1820|  16710586|       674501|           1152135|
|     Speed Records|      2392[...]<jupyter_text>Finally we can see that there is a difference from US and T-Series instead of BLACKPINK is the most trending channel<jupyter_code>plt.clf()
df_IN_1.orderBy(df_IN_1['sum(view_count)'].desc()).limit(10).toPandas().plot(x='channelTitle',y='sum(view_count)',kind='bar')
display()<jupyter_output><empty_output><jupyter_text>***GB - United Kingdom***lets repeat the same process for UK and see the total number of views per channel and category and then plot it to see the most trending channel.1. Convert data type of columns2. group by channel title and category3. plor the count against channel title<jupyter_code>df_GB=df_GB.withColumn("view_count", df_GB["view_count"].cast("int"))\
.withColumn("likes", df_GB["likes"].cast("int"))\
.withColumn("dislikes", df_GB["dislikes"].cast("int"))\
.withColumn("comment_count", df_GB["comment_count"].cast("int"))\
.withColumn("trending_date", to_timestamp(df_GB["trending_date"], "yy.dd.MM"))\
.withColumn("time_published", to_timestamp(df_GB["publishedAt"], "yyyy-MM-dd"))\
.withColumn("categoryId", df_GB["categoryId"].cast('int'))\
.select('view_count','title','tags','channelTitle','categoryId','likes','dislikes','comment_count')

Regression_data_df_GB=df_GB
df_GB_category=df_GB.groupBy('categoryId').sum().select('categoryId','sum(view_count)')
df_GB_1=df_GB.groupBy('channelTitle').sum()
df_GB_1.orderBy(df_GB_1['sum(view_count)'].desc()).limit(10).show()<jupyter_output>+----------------+---------------+---------------+----------+-------------+------------------+
|    channelTitle|sum(view_count)|sum(categoryId)|sum(likes)|sum(dislikes)|sum(comment_count)|
+----------------+---------------+---------------+----------+-------------+------------------+
|       BLACKPINK|      755473024|            320|  76986861|       777843|          11251981|
|         MrBeast|      389373353|            384|  23876077|       228824|           1399116|
|            NASA|      196377719|            560|   5152810|        99314|            155540|
|JustinBieberVEVO|      183798786|            170|  13110603|       256890|            903637|
|       LazarBeam|      170546869|            540|   8992030|        99471|            867873|
|      <NAME>|      155286027|            150|   8525430|        67782|            496655|
|    Dude Perfect|      145709967|            255|   5979172|        90335|            333413|
|       Daft Punk|      141527024|             70|[...]<jupyter_text>BLACKPINK is famous across the globe I guess<jupyter_code>plt.clf()
df_GB_1.orderBy(df_GB_1['sum(view_count)'].desc()).limit(10).toPandas().plot(x='channelTitle',y='sum(view_count)',kind='bar')
display()



df_RU<jupyter_output><empty_output><jupyter_text>***Russia***lets repeat the same process for Russia and see the total number of views per channel and category and then plot it to see the most trending channel.1. Convert data type of columns2. group by channel title and category3. plor the count against channel title<jupyter_code>df_RU=df_RU.withColumn("view_count", df_RU["view_count"].cast("int"))\
.withColumn("likes", df_RU["likes"].cast("int"))\
.withColumn("dislikes", df_RU["dislikes"].cast("int"))\
.withColumn("comment_count", df_RU["comment_count"].cast("int"))\
.withColumn("trending_date", to_timestamp(df_RU["trending_date"], "yy.dd.MM"))\
.withColumn("time_published", to_timestamp(df_RU["publishedAt"], "yyyy-MM-dd"))\
.withColumn("categoryId", df_RU["categoryId"].cast('int'))\
.select('view_count','title','tags','channelTitle','categoryId','likes','dislikes','comment_count')

Regression_data_df_RU=df_RU
df_RU_category=df_RU.groupBy('categoryId').sum().select('categoryId','sum(view_count)')
df_RU_1=df_RU.groupBy('channelTitle').sum()
df_RU_1.orderBy(df_RU_1['sum(view_count)'].desc()).limit(10).show()<jupyter_output>+--------------------+---------------+---------------+----------+-------------+------------------+
|        channelTitle|sum(view_count)|sum(categoryId)|sum(likes)|sum(dislikes)|sum(comment_count)|
+--------------------+---------------+---------------+----------+-------------+------------------+
|                  A4|      334439867|           1056|  34171081|       808876|           3696126|
|           BLACKPINK|      299124105|            160|  34645051|       279648|           5558716|
|               вДудь|       62582402|            220|   3614658|       191628|            381184|
|          Mamix Live|       58697796|           1342|   7142450|        95723|            315332|
|           BANGTANTV|       54411246|            110|  15225319|        24840|           1210930|
|           <NAME>|       53276441|             30|   3431183|        58869|            419221|
|    JustinBieberVEVO|       51019123|             60|   4752879|        82953|            329648|
|         [...]<jupyter_text>Finally we see another country where BLACKPINK is not at topA4 takes the lead here<jupyter_code>plt.clf()
df_RU_1.orderBy(df_RU_1['sum(view_count)'].desc()).limit(10).toPandas().plot(x='channelTitle',y='sum(view_count)',kind='bar')
display()<jupyter_output><empty_output><jupyter_text>DE<jupyter_code>df_DE=df_DE.withColumn("view_count", df_DE["view_count"].cast("int"))\
.withColumn("likes", df_DE["likes"].cast("int"))\
.withColumn("dislikes", df_DE["dislikes"].cast("int"))\
.withColumn("comment_count", df_DE["comment_count"].cast("int"))\
.withColumn("trending_date", to_timestamp(df_DE["trending_date"], "yy.dd.MM"))\
.withColumn("time_published", to_timestamp(df_DE["publishedAt"], "yyyy-MM-dd"))\
.withColumn("categoryId", df_DE["categoryId"].cast('int'))\
.select('view_count','title','tags','channelTitle','categoryId','likes','dislikes','comment_count')

Regression_data_df_DE=df_DE
df_DE_category=df_DE.groupBy('categoryId').sum().select('categoryId','sum(view_count)')
df_DE_1=df_DE.groupBy('channelTitle').sum()
df_DE_1.orderBy(df_DE_1['sum(view_count)'].desc()).limit(10).show()
plt.clf()
df_DE_1.orderBy(df_DE_1['sum(view_count)'].desc()).limit(10).toPandas().plot(x='channelTitle',y='sum(view_count)',kind='bar')
display()<jupyter_output>/usr/local/lib/python3.7/dist-packages/matplotlib/backends/backend_agg.py:214: RuntimeWarning: Glyph 50883 missing from current font.
  font.set_text(s, 0.0, flags=flags)
/usr/local/lib/python3.7/dist-packages/matplotlib/backends/backend_agg.py:183: RuntimeWarning: Glyph 50883 missing from current font.
  font.set_text(s, 0, flags=flags)<jupyter_text>We have seen that for a lot of countries the trending channels are the same and for a lot they are differentNow we will plot the top trending channel for Russian, US and India in order to see a comparison of the difference in views.TOP comparison channels US India Russia<jupyter_code># taking the top 1 for all three
US_1 = df_1.orderBy(df_1['sum(view_count)'].desc()).limit(1)
RU_1 = df_RU_1.orderBy(df_RU_1['sum(view_count)'].desc()).limit(1)
IN_1 = df_IN_1.orderBy(df_IN_1['sum(view_count)'].desc()).limit(1)
# changing the column names
US_1 = US_1.withColumn("country",lit("US"))
RU_1 = RU_1.withColumn("country",lit("RU"))
IN_1 = IN_1.withColumn("country",lit("IN"))
# adding them to make a new data frame
new_yes = US_1.union(RU_1)
new_yes = new_yes.union(IN_1)
new_yes = new_yes.union(IN_1)
new_yes.show()
new_yes.summary()
a['country'] = a['country'].astype('category')
a['channelTitle'] = a['channelTitle'].astype('category')
a = new_yes.orderBy(new_yes['sum(view_count)'].desc()).toPandas()
from matplotlib.patches import Patch
# adding patches to provide custom colors for all histogram by country

colours = {"IN": "#273c75", "US": "#44bd32", "RU": "#17becf"}
plt.clf()
a.plot(x='channelTitle',y='sum(view_count)',kind='bar',color=a['country'].replace(colours)).legend(
    [
        Patch(facecolor=colours['IN']),
        Patch(facecolor=colours['US']),
        Patch(facecolor=colours['RU'])
    ], ["IN", "US","RU"]
)

display()<jupyter_output><empty_output><jupyter_text>***Plotting the average time taken for a youtube video to trend from the time it was published***<jupyter_code>video_data.dtypes<jupyter_output><empty_output><jupyter_text>converting trending date and publish date into the appropriate format published at was converted directly, however, it took some string manipulation to convert it into the appropriate format.20 was added to the start of the string in order to make is suitable for the function to later convert it into date time<jupyter_code>video_data['publishedAt'] = video_data['publishedAt'].astype('datetime64[ns]')
video_data['trending_date'] = ['20'] + video_data['trending_date']
video_data.dtypes
video_data['trending_date'] = pd.to_datetime(video_data['trending_date'], format = "%Y.%d.%m")
video_data.dtypes<jupyter_output><empty_output><jupyter_text>Now lets make a new column time difference to see how much time is required<jupyter_code>video_data['time_diff'] = video_data['trending_date'] - video_data['publishedAt']<jupyter_output><empty_output><jupyter_text>*average* time for a video to go viral --Almost four days but for simplicit we will say it as more than three days<jupyter_code>video_data.time_diff.mean()<jupyter_output><empty_output><jupyter_text>unique channels It has 6000 row checking if all were updated by unique channels746 unique channels<jupyter_code>video_data.channelTitle.nunique()<jupyter_output><empty_output><jupyter_text>changing the date type to numerical in order for us to plot time diff adding a new variable for time diff to convert into hours<jupyter_code>video_data.dtypes<jupyter_output><empty_output><jupyter_text>Dividing it by timedelta data frame to convert it into hours<jupyter_code>video_data['time_diff_hour'] = video_data['time_diff'] / pd.Timedelta('1 hour')
us = video_data<jupyter_output><empty_output><jupyter_text>On an average 93 hours<jupyter_code>us['time_diff_hour'].mean()<jupyter_output><empty_output><jupyter_text>checking how many days - around four days<jupyter_code>us['time_diff_hour'].mean() / 24
import seaborn as sns<jupyter_output><empty_output><jupyter_text>The average value of the columns is 93 and we can see some extreme values. We are subtracting extreme values in order to have an accurate depiction<jupyter_code>sns.boxplot(data=us, y='time_diff_hour')<jupyter_output><empty_output><jupyter_text>Removing outliers - as we can see that there are only seven values so removing them<jupyter_code>len(us)
len(us[us.time_diff_hour > 600])
us = us[us.time_diff_hour <= 600]<jupyter_output><empty_output><jupyter_text>This looks good, we can clearly seee from the barchart and box plot that the average time taken in hours is around 93.<jupyter_code>sns.displot(data=us, x='time_diff_hour', kind='hist',
aspect=1.5, bins=20)
sns.boxplot(data=us, y='time_diff_hour')<jupyter_output><empty_output><jupyter_text>***Finding the most common words in trending data***Although not included in the final eda we also saw the most common words that were found in trending data<jupyter_code>us['title'][:5]<jupyter_output><empty_output><jupyter_text>removing punctuations and lowering the the strings<jupyter_code>us['title'] = us.loc[:,'title'].str.lower()
us.dtypes<jupyter_output><empty_output><jupyter_text>changing the title to a string<jupyter_code>us['title'] = us['title'].astype('string')
us['title'][:5]<jupyter_output><empty_output><jupyter_text>removing punctuations<jupyter_code>us['title'] = us['title'].str.replace(r'[^\w\s]+', '')
us['title'][:5]<jupyter_output><empty_output><jupyter_text>Views per million<jupyter_code>us['views_mil'] = us['view_count'] / 1000000<jupyter_output>/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.<jupyter_text>aggregating by mean and count of views<jupyter_code>us[['channelTitle','views_mil']].groupby('channelTitle')\
.agg(['mean','count'])\
.sort_values(by=('views_mil','count'), ascending=False)[:10]<jupyter_output><empty_output><jupyter_text>Black Pink has the highest average with 25 million view per video  ***Number of videos that became trending on each date***We will plot a line plot in order to show the total number of videos that were trending on a particular day.Again not used in the final EDA presented<jupyter_code>daily = us[['trending_date']].value_counts().reset_index()\
.sort_values(by='trending_date').reset_index(drop=True)
daily
daily[0] = daily[0].astype('float')
daily.dtypes

sns.relplot(data=daily, x='trending_date', y=0,
kind='line', aspect=2.5)
daily.dtypes
us.sort_values(by='view_count', ascending=False)[['title','view_count']].iloc[0,:]<jupyter_output><empty_output><jupyter_text>***India - Time required to trend from the time it is published***Now we will repea the whole process for India but only for the part where we have shown previously the time it will be required for a channel to trend from the point it was publishedOn an average we will see around 40 hours<jupyter_code>video_data_IN.dtypes<jupyter_output><empty_output><jupyter_text>Appropriate date time conversion<jupyter_code>video_data_IN['publishedAt'] = video_data_IN['publishedAt'].astype('datetime64[ns]')
video_data_IN['trending_date'] = ['20'] + video_data_IN['trending_date']

video_data_IN['trending_date'] = pd.to_datetime(video_data_IN['trending_date'], format = "%Y.%d.%m")
video_data_IN['time_diff'] = video_data_IN['trending_date'] - video_data_IN['publishedAt']<jupyter_output><empty_output><jupyter_text>It takes around 2 days for a video to be viral<jupyter_code>video_data_IN.time_diff.mean()<jupyter_output><empty_output><jupyter_text>871 unique channels<jupyter_code>video_data_IN.channelTitle.nunique()<jupyter_output><empty_output><jupyter_text>Time difference in hours<jupyter_code>video_data_IN['time_diff_hour'] = video_data_IN['time_diff'] / pd.Timedelta('1 hour')
IN = video_data_IN<jupyter_output><empty_output><jupyter_text>Converting into days<jupyter_code>IN['time_diff_hour'].mean()

IN['time_diff_hour'].mean() / 24<jupyter_output><empty_output><jupyter_text>Removing outliers and plotting them<jupyter_code>import seaborn as sns

sns.boxplot(data=IN, y='time_diff_hour')
len(IN)


len(IN[IN.time_diff_hour > 300])
IN = IN[IN.time_diff_hour <= 300]
sns.displot(data=IN, x='time_diff_hour', kind='hist',
aspect=1.5, bins=20)
IN = IN[IN.time_diff_hour <= 150]
sns.boxplot(data=IN, y='time_diff_hour')<jupyter_output><empty_output><jupyter_text>***Russia - Time required to trend from the time it is published***Now we will repea the whole process for India but only for the part where we have shown previously the time it will be required for a channel to trend from the point it was publishedOn an average we will see around 11 - 15  hours<jupyter_code>video_data_RU.dtypes

video_data_RU['publishedAt'] = video_data_RU['publishedAt'].astype('datetime64[ns]')
video_data_RU['trending_date'] = ['20'] + video_data_RU['trending_date']

video_data_RU['trending_date'] = pd.to_datetime(video_data_RU['trending_date'], format = "%Y.%d.%m")
video_data_RU['time_diff'] = video_data_RU['trending_date'] - video_data_RU['publishedAt']
video_data_RU.time_diff.mean()
video_data_RU.channelTitle.nunique()
video_data_RU['time_diff_hour'] = video_data_RU['time_diff'] / pd.Timedelta('1 hour')
ru = video_data_RU

ru['time_diff_hour'].mean()

ru['time_diff_hour'].mean() / 24
import seaborn as sns

sns.boxplot(data=ru, y='time_diff_hour')
len(ru)


len(ru[ru.time_diff_hour > 65])
ru[ru['time_diff_hour'] < 0] = 10
run = ru[ru.time_diff_hour <= 65]
sns.displot(data=run, x='time_diff_hour', kind='hist',
aspect=1.5, bins=20)
ru = ru[ru.time_diff_hour <= 66]
sns.boxplot(data=run, y='time_diff_hour')<jupyter_output><empty_output><jupyter_text>***Let's see that which hours makes the possibility of a video trend the most*** ***United States - 17th hour***<jupyter_code>video_data["publish_hour"] = video_data["publishedAt"].apply(lambda x: x[11:13])

# plot no of videos published by hour 
videos_by_pub_hour = video_data.groupby("publish_hour").size().reset_index(name="no_of_videos")
    
fig, ax = plt.subplots(figsize=(8,8))
vph_plot = sns.barplot(x="publish_hour", y="no_of_videos", data=videos_by_pub_hour,palette="YlGnBu")
vph_plot = ax.set(xlabel="Hour in which the video was published", ylabel="Number of Videos")
plt.title("Trending Videos by Published hour\n")
plt.show()<jupyter_output><empty_output><jupyter_text>**India - 5th hour**<jupyter_code>video_data_IN["publish_hour"] = video_data_IN["publishedAt"].apply(lambda x: x[11:13])

# plot no of videos published by hour 
videos_by_pub_hour = video_data_IN.groupby("publish_hour").size().reset_index(name="no_of_videos")
    
fig, ax = plt.subplots(figsize=(8,8))
vph_plot = sns.barplot(x="publish_hour", y="no_of_videos", data=videos_by_pub_hour,palette="YlGnBu")
vph_plot = ax.set(xlabel="Hour in which the video was published", ylabel="Number of Videos")
plt.title("Trending Videos by Published hour\n")
plt.show()
video_data_RU.drop(video_data_RU[video_data_RU.publishedAt == 10].index, inplace=True)
video_data_RU.head()
video_data_RU['publishedAt'] = video_data_RU['publishedAt'].astype('object')<jupyter_output><empty_output><jupyter_text>***Russia - 9th and 13th hour***<jupyter_code>video_data_RU["publish_hour"] = video_data_RU["publishedAt"].apply(lambda x: x[11:13])

# plot no of videos published by hour 
videos_by_pub_hour = video_data_RU.groupby("publish_hour").size().reset_index(name="no_of_videos")
    
fig, ax = plt.subplots(figsize=(8,8))
vph_plot = sns.barplot(x="publish_hour", y="no_of_videos", data=videos_by_pub_hour,palette="YlGnBu")
vph_plot = ax.set(xlabel="Hour in which the video was published", ylabel="Number of Videos")
plt.title("Trending Videos by Published hour\n")
plt.show()<jupyter_output><empty_output><jupyter_text>Categories EDA Creating the category dictionary to map category ID to actual categories<jupyter_code>category_dict={0: 'Film and Animation',
               1: 'Autos & Vehicles', 
               2: 'Music',
               3: 'Pets and Animals',
               4: 'Sports',
               5: 'Short Movies',
               6: 'Travel and Events',
               7: 'Gaming',
               8: 'Video Blogging',
               9: 'People and Blogs',
               10: 'Comedy',
               11: 'Entertainment',
               12: 'News and politics',
               13: 'Howto and Style',
               14: 'Education',
               15: 'Science and Technology',
               16: 'Nonprofits and activisim',
               17: 'Movies',
               18: 'Anime/Animation',
               19: 'Action/Adventure',
               20: 'Classics',
               21: 'Comedy',
               22: 'Documentary',
               23: 'Drama',
               24: 'Family',
               25: 'Foreign',
               26: 'Horror',
               27: 'Sci-Fi/Fantasy',
               28: 'Thriller',
               29: 'Shorts',
               30: 'Shows',
               31: 'Trailers' }<jupyter_output><empty_output><jupyter_text>Reading the data for US, India, Russia<jupyter_code>US_data=pd.read_csv('US_videos_data.csv')
IN_data=pd.read_csv('IN_videos_data.csv')
RU_data=pd.read_csv('RU_videos_data.csv')<jupyter_output><empty_output><jupyter_text>Creating function to group by category_id and get the total view count on a specific category. This will tell us what the users want to watch. We keep only top 5 categories<jupyter_code>def get_category_stats(data):
  test=(data.groupby('categoryId').sum('view_count'))
  test=test[['view_count']]
  test['category']= test.index.map(category_dict)
  
  test=test.sort_values(by='view_count', ascending=False).head(5)

  return test<jupyter_output><empty_output><jupyter_text>Creating a function to group by category_id and the count of videos in each category. This will tell us the kind of content produced on Youtube for each category. Then we filtered everything to keep only top 5 categories<jupyter_code>def get_count_videos_uploaded_by_ctg (data):
  count_data=data.groupby('categoryId').count().sort_values(by='video_id',ascending=False)
  count_data['category']= count_data.index.map(category_dict)
  count_data=count_data[['category','video_id']].head(5)

  return count_data

import seaborn as sns

ctg_US=get_category_stats(US_data)
count_US=get_count_videos_uploaded_by_ctg(US_data)

ctg_IN=get_category_stats(IN_data)
count_IN=get_count_videos_uploaded_by_ctg(IN_data)

ctg_RU=get_category_stats(RU_data)
count_RU=get_count_videos_uploaded_by_ctg(RU_data)<jupyter_output><empty_output><jupyter_text>US analysis<jupyter_code>plt.bar(ctg_US['category'],ctg_US['view_count'])
plt.ylabel("Number of views(In Billion)")
plt.xlabel("Category of video")
plt.bar(count_US['category'],count_US['video_id'])
plt.ylabel("Number of videos uploaded")
plt.xlabel("Category of video")<jupyter_output><empty_output><jupyter_text>India Analysis<jupyter_code>plt.bar(ctg_IN['category'],ctg_IN['view_count'])
plt.ylabel("Number of views (In Billion)")
plt.xlabel("Category of video")
plt.bar(count_IN['category'],count_IN['video_id'])
plt.ylabel("Number of videos uploaded")
plt.xlabel("Category of video")<jupyter_output><empty_output><jupyter_text>Russia Analysis<jupyter_code>plt.bar(ctg_RU['category'],ctg_RU['view_count'])
plt.ylabel("Number of views (In Billion)")
plt.xlabel("Category of video")
plt.bar(count_RU['category'],count_RU['video_id'])
plt.ylabel("Number of videos uploaded")
plt.xlabel("Category of video")<jupyter_output><empty_output><jupyter_text>Tried and Failed Trying Topic Modelling using LDA on Video Titles Cosine SimilarityIn addition we have tried to extract topics by using LDA through another nlp approach and then find the cosine similarity between them in order to see if two different regions are interested in similar or varying topics. However, we were not able to be successful in it.Still showing it as an effort of what we tried to acheive  Data Preprocessing For convenience, given that the dataset is small, we load the spark dataframe<jupyter_code>#video_data=pd.read_csv('US_videos_data.csv/US_videos_data.csv')
df = spark.read.csv('US_videos_data.csv',header=True)<jupyter_output><empty_output><jupyter_text>Fixing Column types<jupyter_code>df=df.withColumn("view_count", df["view_count"].cast("int"))\
.withColumn("likes", df["likes"].cast("int"))\
.withColumn("dislikes", df["dislikes"].cast("int"))\
.withColumn("comment_count", df["comment_count"].cast("int"))\
.withColumn("trending_date", to_timestamp(df["trending_date"], "yy.dd.MM"))\
.withColumn("time_published", to_timestamp(df["publishedAt"], "yyyy-MM-dd"))\
.withColumn("categoryId", df["categoryId"].cast('int'))\
.select('video_id','view_count','title','tags','channelTitle','categoryId','likes','dislikes','comment_count')

## Data for modelling
Regression_data=df<jupyter_output><empty_output><jupyter_text>Tokenizing the column 'title'<jupyter_code>from pyspark.ml.feature import HashingTF, IDF, Tokenizer
tokenizer = Tokenizer(inputCol="title", outputCol="Tokens_title")
df = tokenizer.transform(df)
df.limit(3).show()<jupyter_output>+-----------+----------+--------------------+--------------------+---------------+----------+------+--------+-------------+--------------------+
|   video_id|view_count|               title|                tags|   channelTitle|categoryId| likes|dislikes|comment_count|        Tokens_title|
+-----------+----------+--------------------+--------------------+---------------+----------+------+--------+-------------+--------------------+
|IhO1FcjDMV4|   1067147|Meatpacking: Last...|              [none]|LastWeekTonight|        24| 60111|    1221|         6866|[meatpacking:, la...|
|p4Wy84AOzj0|   1047854|Best 3D Pen Art W...|              [none]|     ZHC Crafts|        26| 50662|     690|         4879|[best, 3d, pen, a...|
|4eKXwKDdXYA|   6133266|100 Days - [Minec...|luke thenotable|l...|Luke TheNotable|        20|372753|    7961|        39551|[100, days, -, [m...|
+-----------+----------+--------------------+--------------------+---------------+----------+------+--------+-------------+-------[...]<jupyter_text>Removing Stop Words from Title<jupyter_code>from pyspark.ml.feature import StopWordsRemover
remover = StopWordsRemover(inputCol="Tokens_title", outputCol="filtered_tokens_title")
df=remover.transform(df)
df.limit(3).show()<jupyter_output>+-----------+----------+--------------------+--------------------+---------------+----------+------+--------+-------------+--------------------+---------------------+
|   video_id|view_count|               title|                tags|   channelTitle|categoryId| likes|dislikes|comment_count|        Tokens_title|filtered_tokens_title|
+-----------+----------+--------------------+--------------------+---------------+----------+------+--------+-------------+--------------------+---------------------+
|IhO1FcjDMV4|   1067147|Meatpacking: Last...|              [none]|LastWeekTonight|        24| 60111|    1221|         6866|[meatpacking:, la...| [meatpacking:, la...|
|p4Wy84AOzj0|   1047854|Best 3D Pen Art W...|              [none]|     ZHC Crafts|        26| 50662|     690|         4879|[best, 3d, pen, a...| [best, 3d, pen, a...|
|4eKXwKDdXYA|   6133266|100 Days - [Minec...|luke thenotable|l...|Luke TheNotable|        20|372753|    7961|        39551|[100, days, -, [m...| [100, days, -, [m...[...]<jupyter_text>Trying Topic Modelling using LDA on Video Titles<jupyter_code>LDA_data=df
LDA_data.columns
test_data=LDA_data.limit(3)
from pyspark.ml.feature import HashingTF, IDF, Tokenizer
hashingTF = HashingTF(inputCol="filtered_tokens_title", outputCol="TF", numFeatures=20)
featurizedData = hashingTF.transform(test_data)
featurizedData.show()
from pyspark.ml.feature import HashingTF, IDF, Tokenizer
idf = IDF(inputCol="TF", outputCol="features")
idfModel = idf.fit(featurizedData)
result_tfidf = idfModel.transform(featurizedData) 
result_tfidf.show()
from pyspark.mllib.clustering import LDA
result_tfidf=result_tfidf.select('video_id','features')
num_topics = 2
max_iterations = 3
#lda_model = LDA.train(result_tfidf.rdd, k=num_topics, maxIterations=max_iterations)
lda = LDA()
#lda=LDA.train(featurizedData['TF'].rdd,k=2)
#LDA.train()
lda.setMaxIter(10)
featurizedData.show()<jupyter_output>+-----------+----------+--------------------+--------------------+---------------+----------+------+--------+-------------+--------------------+---------------------+--------------------+
|   video_id|view_count|               title|                tags|   channelTitle|categoryId| likes|dislikes|comment_count|        Tokens_title|filtered_tokens_title|                  TF|
+-----------+----------+--------------------+--------------------+---------------+----------+------+--------+-------------+--------------------+---------------------+--------------------+
|IhO1FcjDMV4|   1067147|Meatpacking: Last...|              [none]|LastWeekTonight|        24| 60111|    1221|         6866|[meatpacking:, la...| [meatpacking:, la...|(20,[1,2,3,9],[1....|
|p4Wy84AOzj0|   1047854|Best 3D Pen Art W...|              [none]|     ZHC Crafts|        26| 50662|     690|         4879|[best, 3d, pen, a...| [best, 3d, pen, a...|(20,[1,3,5,12,13,...|
|4eKXwKDdXYA|   6133266|100 Days - [Minec...|luke thenotable[...]<jupyter_text>Reading data for multiple countries<jupyter_code>df_IN = spark.read.csv('IN_videos_data.csv',header=True)
df_BR = spark.read.csv('BR_videos_data.csv',header=True)
df_FR = spark.read.csv('FR_videos_data.csv',header=True)
df_CA= spark.read.csv('CA_videos_data.csv',header=True)
df_RU = spark.read.csv('RU_videos_data.csv',header=True)
df_JP = spark.read.csv('JP_videos_data.csv',header=True)
df_MX = spark.read.csv('MX_videos_data.csv',header=True)
df_IN.columns
df_IN=df_IN.withColumn("view_count", df_IN["view_count"].cast("int"))\
.withColumn("likes", df_IN["likes"].cast("int"))\
.withColumn("dislikes", df_IN["dislikes"].cast("int"))\
.withColumn("comment_count", df_IN["comment_count"].cast("int"))\
.withColumn("trending_date", to_timestamp(df_IN["trending_date"], "yy.dd.MM"))\
.withColumn("time_published", to_timestamp(df_IN["publishedAt"], "yyyy-MM-dd"))\
.withColumn("categoryId", df_IN["categoryId"].cast('int'))\
.select('view_count','title','tags','channelTitle','categoryId')
df_IN_category=df_IN.groupBy('categoryId').sum().select('categoryId','sum(view_count)')
IN_top5_categories=df_IN_category.filter(df_IN_category['categoryId'].isNotNull()).orderBy(df_IN_category['sum(view_count)'].desc()).limit(5)
df_BR=df_BR.withColumn("view_count", df_BR["view_count"].cast("int"))\
.withColumn("likes", df_BR["likes"].cast("int"))\
.withColumn("dislikes", df_BR["dislikes"].cast("int"))\
.withColumn("comment_count", df_BR["comment_count"].cast("int"))\
.withColumn("trending_date", to_timestamp(df_BR["trending_date"], "yy.dd.MM"))\
.withColumn("time_published", to_timestamp(df_BR["publishedAt"], "yyyy-MM-dd"))\
.withColumn("categoryId", df_BR["categoryId"].cast('int'))\
.select('view_count','title','tags','channelTitle','categoryId')

df_BR_category=df_BR.groupBy('categoryId').sum().select('categoryId','sum(view_count)')
BR_top5_categories=df_BR_category.filter(df_BR_category['categoryId'].isNotNull()).orderBy(df_BR_category['sum(view_count)'].desc()).limit(5)
df_CA=df_CA.withColumn("view_count", df_CA["view_count"].cast("int"))\
.withColumn("likes", df_CA["likes"].cast("int"))\
.withColumn("dislikes", df_CA["dislikes"].cast("int"))\
.withColumn("comment_count", df_CA["comment_count"].cast("int"))\
.withColumn("trending_date", to_timestamp(df_CA["trending_date"], "yy.dd.MM"))\
.withColumn("time_published", to_timestamp(df_CA["publishedAt"], "yyyy-MM-dd"))\
.withColumn("categoryId", df_CA["categoryId"].cast('int'))\
.select('view_count','title','tags','channelTitle','categoryId')

df_CA_category=df_CA.groupBy('categoryId').sum().select('categoryId','sum(view_count)')
CA_top5_categories=df_CA_category.filter(df_CA_category['categoryId'].isNotNull()).orderBy(df_CA_category['sum(view_count)'].desc()).limit(5)
df_FR=df_FR.withColumn("view_count", df_FR["view_count"].cast("int"))\
.withColumn("likes", df_FR["likes"].cast("int"))\
.withColumn("dislikes", df_FR["dislikes"].cast("int"))\
.withColumn("comment_count", df_FR["comment_count"].cast("int"))\
.withColumn("trending_date", to_timestamp(df_FR["trending_date"], "yy.dd.MM"))\
.withColumn("time_published", to_timestamp(df_FR["publishedAt"], "yyyy-MM-dd"))\
.withColumn("categoryId", df_FR["categoryId"].cast('int'))\
.select('view_count','title','tags','channelTitle','categoryId')

df_FR_category=df_FR.groupBy('categoryId').sum().select('categoryId','sum(view_count)')
FR_top5_categories=df_FR_category.filter(df_FR_category['categoryId'].isNotNull()).orderBy(df_FR_category['sum(view_count)'].desc()).limit(5)
df_MX=df_MX.withColumn("view_count", df_MX["view_count"].cast("int"))\
.withColumn("likes", df_MX["likes"].cast("int"))\
.withColumn("dislikes", df_MX["dislikes"].cast("int"))\
.withColumn("comment_count", df_MX["comment_count"].cast("int"))\
.withColumn("trending_date", to_timestamp(df_MX["trending_date"], "yy.dd.MM"))\
.withColumn("time_published", to_timestamp(df_MX["publishedAt"], "yyyy-MM-dd"))\
.withColumn("categoryId", df_MX["categoryId"].cast('int'))\
.select('view_count','title','tags','channelTitle','categoryId')

df_MX_category=df_MX.groupBy('categoryId').sum().select('categoryId','sum(view_count)')
MX_top5_categories=df_MX_category.filter(df_MX_category['categoryId'].isNotNull()).orderBy(df_MX_category['sum(view_count)'].desc()).limit(5)<jupyter_output><empty_output><jupyter_text>Pairwise country Cosine Similarity<jupyter_code>IN_vector=IN_top5_categories.select('categoryId').collect()
IN_array = [row[0] for row in IN_vector]
IN_array
from scipy import spatial
#US_vector=US_top5_categories.select('categoryId').collect()
US_array = [row for row in IN_array]

1 - spatial.distance.cosine(IN_array, US_array)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.20895633139168182, 'fraction_numerical': 0.04702630504953066, 'mean_word_length': 3.3201634877384194, 'pattern_counts': {'":': 3, '<': 279, '<?xml version=': 0, '>': 276, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '71', 'dir': 'jupyter-structured-clean-dedup', 'id': '649282', 'license': 'MIT', 'n_tokens_mistral': 19708, 'n_tokens_neox': 17752, 'n_words': 3234, 'path': 'EDA_Top_Trending_Videos.ipynb', 'repo_name': 'Lakshay768/Big-Data-Scaling-Youtube-Data-Analysis', 'size': '53943'}