<jupyter_start><jupyter_text>HCA DCP User Analytics - Q1 2022 vs Q4 2021This section displays metrics collected from Google Analytics HCA DCP Data Portal and Data Browser and compares Q1 2022 with the previous quarter (Q4 2021). The generation of this report is now coded in Jupiter notebooks and can be rerun easily and modified or consolidated as desired.  Geographic exclusionsThe data presented excludes access from Cambridge, UK and surrounds, Cambridge MA and surrounds, and San Francisco, CA and surrounds an attempt to exclude the HCA DCP development team member activity from the report. Key to tablesOn the tables:* Diffs e.g. (+19.04%) compare the later quarter with the previous quarter.* "+" Indicates the entry is new in the report in the current quarter.* "↑" Indicates the entry has moved up in rank since the last quarter.* "↓" Indicates the entry has moved down in rank since the last quarter.<jupyter_code># !pip install google_auth_oauthlib 
# !pip install install google-api-python-client
# !pip install pandas 
# !pip install numpy
# !pip install matplotlib
# !pip install google-cloud-bigquery
# !pip install db-dtypes

# DEV GUIDES
#https://developers.google.com/analytics/devguides/reporting
    



import warnings
warnings.filterwarnings('ignore')
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.discovery import build
from IPython.display import display
import os
import pprint
# import pandas as pd
# import matplotlib.pyplot as plt
# import numpy as np
import analytics_api as ga

%load_ext autoreload
%autoreload 2

import analytics_charts as ac
GA_PROPERTY = "185740629" # HCA Data Portal/Browser
DCP_ANALYTICS_START = "2019-03-01"
TODAY = 'today'

ac.authenticate_ga()<jupyter_output>The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=713613812354-4nlori99e34rebtiup7ch8gcpgk40kbn.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fanalytics.readonly&state=G153u2hcG4osU8SQw1UK2fm9ssXJ6S&access_type=offline<jupyter_text>User acquisition  Daily users and pageviews (all time)<jupyter_code>users_over_time_table = ac.plot_users_over_time(GA_PROPERTY, DCP_ANALYTICS_START,TODAY)<jupyter_output><empty_output><jupyter_text>Total unique users and unique pageviews by quarter (all time)<jupyter_code>display(users_over_time_table)<jupyter_output><empty_output><jupyter_text>Users and sessions<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Total Users", None, "ga:users", None, "2022Q1", "2021Q4")
ac.show_difference_table(GA_PROPERTY, "Returning Users", None, "ga:users", None, filters="ga:userType==Returning Visitor", period="2022Q1", prev_period="2021Q4")
ac.show_difference_table(GA_PROPERTY, "Total Visits", None, "ga:sessions", None, "2022Q1", "2021Q4")
ac.show_difference_table(GA_PROPERTY, "Sessions Including Visits to Data Portal", None, "ga:sessions", None, filters="ga:pagePath!~/explore", period="2022Q1", prev_period="2021Q4")
ac.show_difference_table(GA_PROPERTY, "Sessions Including Visits to Data Browser", None, "ga:sessions", None, filters="ga:pagePath=~/explore", period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Top referrers<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Users", "Referrer", "ga:users","ga:source", "2022Q1", "2021Q4")<jupyter_output><empty_output><jupyter_text>Top countries<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Users", "Country", "ga:users","ga:country", "2022Q1", "2021Q4")<jupyter_output><empty_output><jupyter_text>Social network referrers<jupyter_code>ac.show_difference_table(GA_PROPERTY, "User Sessions", "Social Network", "ga:sessions", "ga:socialNetwork", period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Top entrances<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Entrances", "Page", "ga:entrances","ga:pagePath", "2022Q1", "2021Q4")<jupyter_output><empty_output><jupyter_text>Downloads and exports Download project manifest<jupyter_code>ac.show_difference_table(GA_PROPERTY, ["Users", "Count"], "Action", ["ga:users", "ga:hits"], ["ga:eventAction", "ga:eventCategory"], filters="ga:eventCategory==Project;ga:eventAction==Download Project Manifest", ordered=False, period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Download project matrix<jupyter_code>ac.show_difference_table(GA_PROPERTY, ["Users", "Count"], "Download Type", ["ga:users", "ga:hits"], ["ga:dimension6", "ga:eventAction", "ga:eventCategory"], filters="ga:eventCategory==Project;ga:eventAction==Download Project Matrix", ordered=False, period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Request curl command for selected data<jupyter_code>ac.show_difference_table(GA_PROPERTY, ["Users", "Count"], "curl Request From Search Results", ["ga:users", "ga:hits"], ["ga:dimension1", "ga:eventAction", "ga:eventCategory"], filters="ga:eventAction==Request;ga:dimension1==Bulk Download", ordered=False, period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Metadata request/download from selected data<jupyter_code>ac.show_difference_table(GA_PROPERTY, ["Users", "Count"], "Metadata Request/Download From Search Results", ["ga:users", "ga:hits"], ["ga:dimension1", "ga:eventAction", "ga:eventCategory"], filters="ga:eventCategory==Manifest;ga:eventAction=~Request|Download;ga:dimension1=~Cohort Manifest|Cohort Manifest Link", ordered=False, period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Request export from selected data<jupyter_code>ac.show_difference_table(GA_PROPERTY, ["Users", "Count"], "Request Export From Search Results", ["ga:users", "ga:hits"], ["ga:dimension3", "ga:dimension1", "ga:eventAction", "ga:eventCategory"], filters="ga:eventCategory==Export;ga:eventAction==Request;ga:dimension1==Cohort Export;ga:dimension3==Terra", ordered=False, period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Direct file downloads (from files tab)<jupyter_code>ac.show_difference_table(GA_PROPERTY, ["Users", "Count"], "Action", ["ga:users", "ga:hits"], ["ga:eventAction", "ga:eventCategory"], filters="ga:eventCategory==File;ga:eventAction==Download", ordered=False, period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Top direct file download file types (from files tab)<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Downloads", "File Type", "ga:hits", "ga:dimension6", filters="ga:eventCategory==File;ga:eventAction==Download", period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Top "export selected data" queries<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Selected for Export Count", "Query", "ga:pageviews", "ga:pagePath", filters="ga:pagePath=~/explore/export/export-to-terra|/explore/export/get-curl-command|/explore/export/download-manifest", period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Entities Entity tabs selections (project, samples, files)<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Count", "Entity Tab", "ga:hits", "ga:eventLabel", filters="ga:eventCategory==Entity;ga:eventAction==Select Tab", period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Search facets Top facets<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Count", "Facet", "ga:hits", "ga:dimension9", filters="ga:eventCategory==Search;ga:eventAction==Select", period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Top facet terms<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Count", ["Facet", "Term"], "ga:hits", ["ga:dimension9", "ga:dimension10"], filters="ga:eventCategory==Search;ga:eventAction==Select", period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Projects Top project detail pages<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Count", "Project", "ga:pageviews", "ga:pagePath", filters="ga:pagePath=~^(\\/explore\\/projects\\/[0-9a-fA-F]{8}\\-[0-9a-fA-F]{4}\\-[0-9a-fA-F]{4}\\-[0-9a-fA-F]{4}\\-[0-9a-fA-F]{12})(\\/?\\?{0}|\\/?\\?{1}.*)$", period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Project detail supplementary links visits (from external resources)<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Visits", "Link", "ga:hits", ["ga:dimension4", "ga:dimension6", "ga:eventLabel"], filters="ga:eventCategory==Project;ga:eventAction==View External Resource", period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Withdrawn/deprecated projects visits<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Count", "Project Type", "ga:hits", "ga:eventAction", filters="ga:eventAction=~View Deprecated Project|View Withdrawn Project", period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Analysis protocol portal links<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Count", "Analysis Protocol", "ga:hits", "ga:eventLabel", filters="ga:eventCategory==Portal Link;ga:eventAction==Click", period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Data Portal Top landing pages<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Count", "Path", "ga:entrances", "ga:landingPagePath", filters="ga:landingPagePath!~^\/explore", period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Top pages<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Count", "Path", "ga:pageviews", "ga:pagePath", filters="ga:pagePath!~^\/explore", period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Top searches<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Count", "Search Text", "ga:hits", ["ga:eventLabel", "ga:eventAction"], filters="ga:eventCategory==Search;ga:eventAction==Enter Text", period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output><jupyter_text>Top selected search results<jupyter_code>ac.show_difference_table(GA_PROPERTY, "Count", "Search Result", "ga:hits", ["ga:eventLabel", "ga:eventAction"], filters="ga:eventCategory==Search;ga:eventAction==Click", period="2022Q1", prev_period="2021Q4")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14883362251783305, 'fraction_numerical': 0.038750722961249276, 'mean_word_length': 11.904228855721392, 'pattern_counts': {'":': 0, '<': 108, '<?xml version=': 0, '>': 108, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '454876', 'license': 'Apache-2.0', 'n_tokens_mistral': 3792, 'n_tokens_neox': 3552, 'n_words': 622, 'path': 'analytics/hdgar-book/.ipynb_checkpoints/user-analytics-checkpoint.ipynb', 'repo_name': 'DataBiosphere/data-browser', 'size': '10374'}