<jupyter_start><jupyter_text>Connect Google Drive to Collab by clicking on mount Drive 👈 Change Directory to where the files are located Make sure you have uploaded the links.txt file with all youtube links<jupyter_code>cd /content/drive/MyDrive/Colab\ Notebooks/Youtube\ Data\ -\ Analytics<jupyter_output>/content/drive/MyDrive/Colab Notebooks/Youtube Data - Analytics<jupyter_text>Make sure you are in the right folder and there are no neccesary files in here<jupyter_code>!pwd
!ls<jupyter_output>/content/drive/MyDrive/Colab Notebooks/Youtube Data - Analytics
 credentials.json  'Youtube Data+Analytics.ipynb'   YOUTUBE_MERGED.csv
 links.txt	    youtube_data.json<jupyter_text>Line below will install few things. Have patience... 👼 If you don't see much red swiggly lines then all is fine<jupyter_code>!pip install --upgrade google-api-python-client google-auth-oauthlib google-auth-httplib2 requests<jupyter_output>Requirement already satisfied: google-api-python-client in /usr/local/lib/python3.7/dist-packages (2.37.0)
Requirement already satisfied: google-auth-oauthlib in /usr/local/lib/python3.7/dist-packages (0.4.6)
Requirement already satisfied: google-auth-httplib2 in /usr/local/lib/python3.7/dist-packages (0.1.0)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (2.27.1)
Requirement already satisfied: google-api-core<3.0.0dev,>=1.21.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client) (1.26.3)
Requirement already satisfied: google-auth<3.0.0dev,>=1.16.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client) (1.35.0)
Requirement already satisfied: uritemplate<5,>=3.0.1 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client) (3.0.1)
Requirement already satisfied: httplib2<1dev,>=0.15.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client) (0.17.4)
Requirement already satisfied: six[...]<jupyter_text>Let the following two cells of code compile 🙏<jupyter_code>import re
import csv
import json
import math
import os
import pickle

import requests

from google.auth.transport.requests import Request
import google.oauth2.credentials
import google_auth_oauthlib.flow
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
from google_auth_oauthlib.flow import InstalledAppFlow
# Code for Youtube Data API
os.environ["OAUTHLIB_INSECURE_TRANSPORT"] = "1"
API_KEY = "<KEY>"
CHANNEL_ID = "UCbgq4Y6WypFnBBzrBDgR0zA"
dict_DATA = None

# Note: get_videos_stats takes a list of at max 50 video ids
def get_videos_stats(video_ids_list):
    videos_ids = ",".join(video_ids_list)
    videos_stats = {}
    try:
        url = f"https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id={videos_ids}&key={API_KEY}"
        response = requests.get(url=url)
        videos_stats = response.json()
    except Exception as e:
        print(e)
    finally:
        return videos_stats

def get_video_ids_from_file(filename):
    videos_ids = []
    with open(filename, encoding="utf-8") as fp:
        for i, line in enumerate(fp, start=1):
            url = line.strip()
            if url:
                if "watch?v=" in url:
                    video_id = url.split("watch?v=")[-1].split("&")[0]
                elif "youtu.be" in url:
                    video_id = url.split("/")[-1]
                else:
                    continue
                
                videos_ids.append(video_id)
    return videos_ids

# Note: get_videos_stats_auto takes a list of variable video_ids and internally calls get_video_stats
def get_videos_stats_auto(video_ids_list):
    num_video_ids = len(video_ids_list)
    start_index = 0
    last_index = 50
    DATA = []
    while True:
        if start_index < num_video_ids and last_index > num_video_ids:
            last_index = num_video_ids
        elif start_index >= num_video_ids:
            break
        videos_ids_list_50 = video_ids_list[start_index:last_index]
        videos_stats = get_videos_stats(videos_ids_list_50)
        for video in videos_stats["items"]:
            video_data = format_video_data(video_json=video)
            DATA.append(video_data)
        start_index = last_index
        last_index = last_index + 50
    with open(f"youtube_data.json", "w", encoding="utf-8") as fp:
        json.dump(DATA, fp)
    # with open(f"youtube_data.csv", "w", encoding="utf-8") as data_fp:
    #     fieldnames = DATA[0].keys()
    #     csv_writer = csv.DictWriter(data_fp, fieldnames=fieldnames)
    #     csv_writer.writeheader()
    #     csv_writer.writerows(DATA)
    return DATA

def format_video_data(video_json):
    video_data = {}
    video_data["id"] = video_json.get("id")
    video_data["channel_id"] = video_json["snippet"].get("channelId")
    video_data["channel_title"] = video_json["snippet"].get("channelTitle")
    video_data["title"] = video_json["snippet"].get("title")
    video_data["description"] = video_json["snippet"].get("description")
    video_data["published_time"] = video_json["snippet"].get("publishedAt")
    video_data["duration"] = video_json["contentDetails"].get("duration")
    video_data["duration_secs"] = format_video_duration_seconds(duration=video_data["duration"])
    video_data["views"] = video_json["statistics"].get("viewCount", 0)
    video_data["likes"] = video_json["statistics"].get("likeCount", 0)
    video_data["dislikes"] = video_json["statistics"].get("dislikeCount", 0)
    video_data["comments"] = video_json["statistics"].get("commentCount", 0)

    return video_data

def format_video_duration_seconds(duration):
    hours = re.search(r"(\d+)H", duration)
    minutes = re.search(r"(\d+)M", duration)
    seconds = re.search(r"(\d+)S", duration)

    if hours:
        hours = int(hours.group(1))
    else:
        hours = 0
    if minutes:
        minutes = int(minutes.group(1))
    else:
        minutes = 0
    if seconds:
        seconds = int(seconds.group(1))
    else:
        seconds = 0
    time_in_sec = (hours * 60 * 60) + (minutes * 60) + seconds

    return time_in_sec


class GAPIAnalytics():
    def __init__(self):
        self.creds = self.get_credentials()
        self.service = self.get_service()
    
    def get_credentials(self):
        """Returns Service for usage of youtube analytics api v3.
        """
        # If modifying these scopes, delete the file token.pickle.
        SCOPES = ["https://www.googleapis.com/auth/yt-analytics.readonly"]
        creds = None
        # The file token.pickle stores the user"s access and refresh tokens, and is
        # created automatically when the authorization flow completes for the first
        # time.
        print(os.getcwd())
        if os.path.exists("token.pickle"):
            with open("token.pickle", "rb") as token:
                creds = pickle.load(token)
        # If there are no (valid) credentials available, let the user log in.
        if not creds or not creds.valid:
            if creds and creds.expired and creds.refresh_token:
                creds.refresh(Request())
            else:
                flow = InstalledAppFlow.from_client_secrets_file(
                    "credentials.json", SCOPES)
                creds = flow.run_console()
            # Save the credentials for the next run
            with open("token.pickle", "wb") as token:
                pickle.dump(creds, token)
        return creds
    
    def get_service(self):
        service = build("youtubeAnalytics", "v2", credentials=self.creds)
        return service
    
    def execute_api_request(self, client_library_function, **kwargs):
      response = client_library_function(
        **kwargs
      ).execute()
      # print(json.dumps(response, indent=2))
      return response

def format_video_data_analytics(video_json):
    header = ["video_id", "views_analytics", "likes_analytics", "dislikes_analytics", "comments_analytics", "shares_analytics", 
                "estimatedMinutesWatched_analytics", "averageViewDuration", "averageViewPercentage_analytics", "subcribersGained_analytics",
                "subcribersLost_analytics"]
    video_data = {}
    if video_json:
        for i, value in enumerate(header, start=0):
            video_data[value] = video_json[i]
    return video_data

def get_videos_stats_auto_analytics(video_ids_list, gapi):
    num_video_ids = len(video_ids_list)
    start_index = 0
    last_index = 50
    analytics_DATA = []
    while True:
        print(start_index, last_index, num_video_ids)
        if start_index < num_video_ids and last_index > num_video_ids:
            last_index = num_video_ids
        elif start_index >= num_video_ids:
            break
        videos_ids_list_50 = video_ids_list[start_index:last_index]
        video_ids_str_50 = ",".join(videos_ids_list_50)
        videos_stats = gapi.execute_api_request(
                                gapi.service.reports().query,
                                metrics="views,likes,dislikes,comments,shares,estimatedMinutesWatched,averageViewDuration,averageViewPercentage,subscribersGained,subscribersLost",
                                ids=f"channel=={CHANNEL_ID}",
                                startDate="2005-01-01",
                                endDate="2099-01-01",
                                filters=f"video=={video_ids_str_50}",
                                dimensions="video"
                            )
        for video in videos_stats["rows"]:
            video_data = format_video_data_analytics(video_json=video)
            analytics_DATA.append(video_data)
        start_index = last_index
        last_index = last_index + 50
    with open(f"channel_analytics_{CHANNEL_ID}.json", "w", encoding="utf-8") as fp:
        json.dump(analytics_DATA, fp, indent=2)
    return analytics_DATA

def clean_videos_id_list_analytics(dict_DATA, video_ids_list, CHANNEL_ID):
    new_list = []
    for video_id in video_ids_list:
        d = dict_DATA.get(video_id)
        if d:
            if d["channel_id"] == CHANNEL_ID:
                new_list.append(video_id)
    return new_list

def get_analytics_download_status(DATA):
    unique_channels = set()
    channels_info = []
    for i, record in enumerate(DATA, start=0):
        if record["channel_id"] not in unique_channels:
            if not os.path.exists(f"channel_analytics_{record['channel_id']}.json"):
                unique_channels.add(record["channel_id"])
                channel_info = { "channel_id": record["channel_id"], "channel_name": record["channel_title"] }
                channels_info.append(channel_info)
    
    if unique_channels:
        print("List of channel IDs for which you have to get data by running analytics API Code")
        print("CHANNEL NAME".ljust(50, " "), "CHANNEL ID")
        print()
        for channel in channels_info:
            print(channel["channel_name"].ljust(50, " "), channel["channel_id"])
    else:
        print("You have collected data for all channels")
        print("Move to the step of merging all files")

def merge_and_clean_analytics_files(DATA):
    MERGED_DATA = []
    analytics_DATA = []
    current_path = os.getcwd()
    if "Colab Notebooks/Youtube Data - Analytics" in current_path:
        filenames = os.listdir()
        for filename in filenames:
            if "channel_analytics_" in filename and filename.endswith(".json"):
                with open(filename, "r", encoding="utf-8") as channel_analytics_fp:
                    file_data = json.load(channel_analytics_fp)
                    analytics_DATA = analytics_DATA + file_data
                file_path = os.path.join(current_path, filename)
                os.remove(file_path)
    
    # Converting analytics_DATA list to dictionary for faster search
    analytics_dict_DATA = data_list_to_dict_data(analytics_DATA, lookup_key="video_id")
    
    # Merging DATA and analytics_dict_DATA
    for vd in DATA:
        va = analytics_dict_DATA.get(vd["id"])
        temp_data = {}
        for k, v in vd.items():
            temp_data[k] = v
        
        # If the data for it is available in analytics
        if va is not None:
            va.pop("video_id")
            for k, v in va.items():
                temp_data[k] = v

        MERGED_DATA.append(temp_data) 
    
    with open("YOUTUBE_MERGED.csv", "w", encoding="utf-8") as merged_fp:
        fieldnames = list(MERGED_DATA[0].keys())
        for d in MERGED_DATA:
            for k in d.keys():
                if k not in fieldnames:
                    fieldnames.append(k)
        
        csv_writer = csv.DictWriter(merged_fp, fieldnames=fieldnames)
        csv_writer.writeheader()
        csv_writer.writerows(MERGED_DATA)
    
    youtube_data_file_path = os.path.join(current_path, "youtube_data.json")
    if os.path.exists(youtube_data_file_path):
        os.remove(youtube_data_file_path)
    return MERGED_DATA

def delete_pickle_file():
    filepath = os.path.join(os.getcwd(), "token.pickle")
    if os.path.exists(filepath):
        os.remove(filepath)

def data_list_to_dict_data(DATA, lookup_key):
    dict_DATA = {}
    for d in DATA:
        dict_DATA[d[lookup_key]] = d
    
    return dict_DATA<jupyter_output><empty_output><jupyter_text>Now let the script do its magic Run below code cell to first get data from youtube DATA API As an output you'll see the list of **Channel name** and **Channel ID** for which you have to run analytics API by giving permission each time<jupyter_code># Run Youtube Data API Code
video_ids = get_video_ids_from_file(filename="links.txt")
DATA = get_videos_stats_auto(video_ids_list=video_ids)
dict_DATA = data_list_to_dict_data(DATA, lookup_key="id")
get_analytics_download_status(DATA)<jupyter_output>List of channel IDs for which you have to get data by running analytics API Code
CHANNEL NAME                                       CHANNEL ID

Krutika Plays                                      UC6GIR5W1Bm5Moc3kzKV-nhw
8bit MAMBA                                         UCPzIOAKAvphrnIfa9wFj69w
SouL Zeref                                         UC0kGLKFjnuCqndpovCaREGA
Indian Super League                                UCJk-aQ7NZtqYtpcqKDt_vZg
44Two                                              UCevsWYuNhlKt_TLFMZqj8JA
NorthEast United FC                                UCW1-MBwwDxCcQVCPw1XMtRQ
Chennaiyin FC                                      UC6qDKI_BjuvzM9MoiSzOTpA
FC Goa                                             UCiIB2oS367YVFOYT0d7qUkg
Hyderabad FC                                       UCKLoOhHjizY1GO2vBN63MWA
Kerala Blasters                                    UCCRfkpYF70QoUffbC5emuyw
ATK Mohun Bagan FC                                 UCiePrR7_GbDltlbjOvDzUbg
Jamshedpur FC       [...]<jupyter_text>Now update CHANNEL_ID variable one at a time to get it's data from analytics API. Hint: You can get channel ID from above list 👆 Run the Code. Visit the url that is generated and entered the code that you get in the end. Make sure you give access for the same channel id that you entered otherwise the data will not be captured in the file otherwise you get a **FORBIDDEN Error**. Anyway please don't do this. Each time a list with Channel name and Channel ID will be printed for which you still need to give access and get data Once the list of channel names is empty or you don't have access to rest of the channels Move to next cell for merging the data Google drive can sometimes delay the deletion of token.pickle file. Please run the code again if you get FORBIDDEN Error if you did everything right<jupyter_code>CHANNEL_ID = "UCbgq4Y6WypFnBBzrBDgR0zA"
gapi = GAPIAnalytics()
delete_pickle_file()
new_video_ids_list = clean_videos_id_list_analytics(dict_DATA, video_ids, CHANNEL_ID)
void = get_videos_stats_auto_analytics(new_video_ids_list, gapi)
delete_pickle_file()
get_analytics_download_status(DATA)<jupyter_output>/content/drive/MyDrive/Colab Notebooks/Youtube Data - Analytics
Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=934520054582-0l380vmbin5r8njt4a0evg366719kb9g.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fyt-analytics.readonly&state=bJy5dj5elUyEuhbDx3b98IROFyI1n1&prompt=consent&access_type=offline
Enter the authorization code: 4/1AX4XfWgC7cY1iVNlzhglKzdDJuzedMMcdBct2sJ5GFxPHY0tEFP3blFxJwc
0 50 20
20 70 20
List of channel IDs for which you have to get data by running analytics API Code
CHANNEL NAME                                       CHANNEL ID

Krutika Plays                                      UC6GIR5W1Bm5Moc3kzKV-nhw
8bit MAMBA                                         UCPzIOAKAvphrnIfa9wFj69w
SouL Zeref                                         UC0kGLKFjnuCqndpovCaREGA
Indian Super League                                UCJk-aQ7NZtqYtp[...]<jupyter_text>Run Below code to get MERGED_DATA.csv<jupyter_code>void = merge_and_clean_analytics_files(DATA)<jupyter_output><empty_output><jupyter_text>Well Done 👍<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09135542696826232, 'fraction_numerical': 0.015664270267110876, 'mean_word_length': 2.836322869955157, 'pattern_counts': {'":': 2, '<': 35, '<?xml version=': 0, '>': 37, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 28, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '17970', 'license': 'MIT', 'n_tokens_mistral': 5628, 'n_tokens_neox': 5336, 'n_words': 1227, 'path': 'Youtube_Data+Analytics.ipynb', 'repo_name': 'NovaChron0/GDrive', 'size': '17143'}