<jupyter_start><jupyter_text>Part 5: Competing Journals AnalysisIn this notebook we are going to * Load the researchers impact metrics data previously extracted (see parts 1-2-3)* Get the full publications history for these researchers * Use this new publications dataset to determine which are the most frequent journals the researchers have also published in* Build some visualizations in order to have a quick overview of the results   Prerequisites: Installing the Dimensions Library and Logging in<jupyter_code># @markdown # Get the API library and login
# @markdown Click the 'play' button on the left (or shift+enter) after entering your API credentials

username = "" #@param {type: "string"}
password = "" #@param {type: "string"}
endpoint = "https://app.dimensions.ai" #@param {type: "string"}


!pip install dimcli plotly tqdm -U --quiet
import dimcli
from dimcli.shortcuts import *
dimcli.login(username, password, endpoint)
dsl = dimcli.Dsl()

#
# load common libraries
import time
import sys
import os
import json
import pandas as pd
from pandas.io.json import json_normalize
from tqdm.notebook import tqdm as progress

#
# charts libs
# import plotly_express as px
import plotly.express as px
if not 'google.colab' in sys.modules:
  # make js dependecies local / needed by html exports
  from plotly.offline import init_notebook_mode
  init_notebook_mode(connected=True)
#
# create output data folder
if not(os.path.exists("data")):
    os.mkdir("data")<jupyter_output>DimCli v0.6.7 - Succesfully connected to <https://app.dimensions.ai> (method: dsl.ini file)<jupyter_text>Competing Journals From our researchers master list, we now want to extract the following:* full list of publications for a 5 year period* full list of journals with counts of how many publications per journal This new dataset will let us draw up some conclusions re. which are the competing journals of the one we selected at the beginning.  First let's reload the data obtained in previous steps<jupyter_code>#
researchers = pd.read_csv("data/2.researchers_impact_metrics.csv")
#
print("Total researchers:", len(researchers))
researchers.head(5)<jupyter_output>Total researchers: 18341<jupyter_text>What the query looks like The approach we're taking consists in pulling all publications data, so that we can count journals as a second step. This approach may take some time (as we're potentially retrieving a lot of publications data), but it will lead to precise results.  The query template to use looks like this (for a couple of researchers only):<jupyter_code>%%dsldf 
search publications where researchers.id in ["ur.01277776417.51", "ur.0637651205.48"] 
    and year >= 2015 and journal is not empty 
    and journal.id != "jour.1103138" 
return publications[id+journal] limit 10<jupyter_output>Returned Publications: 10 (total = 149)<jupyter_text>Extracting all publications/journals informationThis part may take some time to run (depending on how many years back one wants to go) so you may want to get a coffee while you wait..<jupyter_code>#
journal_id = "jour.1103138" # Nature genetics
start_year = 2018

# our list of researchers
llist = list(researchers['researcher_id'])
#
# the query
q2 = """search publications 
            where researchers.id in {} 
            and year >= {} and journal is not empty and journal.id != "{}" 
    return publications[id+journal+year]"""

VERBOSE = False
RESEARCHER_ITERATOR_NO = 400

pubs = pd.DataFrame
for chunk in progress(list(chunks_of(llist, RESEARCHER_ITERATOR_NO))):
    # get all pubs
    query = q2.format(json.dumps(chunk), start_year, journal_id)
    res = dsl.query_iterative(query, verbose=VERBOSE)
    if pubs.empty:
        # first time, init the dataframe         
        pubs = res.as_dataframe()
    else:
        pubs.append(res.as_dataframe())
# remove duplicate publications, if they have the same PUB_ID
pubs = pubs.drop_duplicates(subset="id")
# save
pubs.to_csv("data/5.journals-via-publications-RAW.csv", index=False)
# preview the data
pubs<jupyter_output><empty_output><jupyter_text>Now we can create a journals-only dataset that includes counts per year, and grant total.<jupyter_code>journals = pubs.copy()
# drop pub_id column
journals = journals.drop(['id'], axis=1)
# 
# add total column 
journals['total'] = journals.groupby('journal.id')['journal.id'].transform('count')
journals['total_year'] = journals.groupby(['journal.id', 'year'])['journal.id'].transform('count')
# 
# remove multiple counts for same journal
journals = journals.drop_duplicates() 
journals.reset_index(drop=True)
#
# sort by total count
journals = journals.sort_values('total', ascending=False)
# #
# # save
journals.to_csv("data/5.journals-via-publications.csv", index=False)
print("======\nDone")

# download the data
if COLAB_ENV:
  files.download("data/5.journals-via-publications.csv")
#preview the data 
journals.head(10)<jupyter_output>======
Done<jupyter_text>Visualizations<jupyter_code>threshold = 100
temp = journals.sort_values("total", ascending=False)[:threshold]

px.bar(journals[:threshold], 
       x="journal.title", y="total_year", 
       color="year",
       hover_name="journal.title", 
       hover_data=['journal.id', 'journal.title', 'total' ], 
       title=f"Top {threshold} competitors for {journal_id} (based on publications data from {start_year})")
threshold = 200
temp = journals.sort_values("year", ascending=True).groupby("year").head(threshold)

px.bar(journals[:threshold], 
       x="journal.title", y="total_year", 
       color="year",
       facet_row="year",
       height=900,
       hover_name="journal.title", 
       hover_data=['journal.id', 'journal.title', 'total' ], 
       title=f"Top {threshold} competitors for {journal_id} - segmented by year")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10617326647068995, 'fraction_numerical': 0.01538993601936711, 'mean_word_length': 4.914110429447852, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 24, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '10832', 'license': 'MIT', 'n_tokens_mistral': 1834, 'n_tokens_neox': 1705, 'n_words': 569, 'path': 'archive/2020-04-Publishers-Usecases/5-Competing-Journals.ipynb', 'repo_name': 'eschares/dimensions-api-lab', 'size': '5783'}