<jupyter_start><jupyter_text>Clinical Trials by Volume of PublicationsThis notebook shows how use the [Dimensions Analytics API](https://www.dimensions.ai/dimensions-apis/) in order to get a list of clinical trials records and then sort them by the total number of publications they cite.<jupyter_code>import datetime
print("==\nCHANGELOG\nThis notebook was last run on %s\n==" % datetime.date.today().strftime('%b %d, %Y'))<jupyter_output>==
CHANGELOG
This notebook was last run on Jan 25, 2022
==<jupyter_text>PrerequisitesThis notebook assumes you have installed the [Dimcli](https://pypi.org/project/dimcli/) library and are familiar with the ['Getting Started' tutorial](https://api-lab.dimensions.ai/cookbooks/1-getting-started/1-Using-the-Dimcli-library-to-query-the-API.html).<jupyter_code>!pip install dimcli plotly tqdm -U --quiet 

import dimcli
from dimcli.utils import *

import os, sys, time, json
from tqdm.notebook import tqdm 
import pandas as pd
import plotly.express as px
from plotly.offline import plot
if not 'google.colab' in sys.modules:
  # make js dependecies local / needed by html exports
  from plotly.offline import init_notebook_mode
  init_notebook_mode(connected=True)

print("==\nLogging in..")
# https://digital-science.github.io/dimcli/getting-started.html#authentication
ENDPOINT = "https://app.dimensions.ai"
if 'google.colab' in sys.modules:
  import getpass
  KEY = getpass.getpass(prompt='API Key: ')  
  dimcli.login(key=KEY, endpoint=ENDPOINT)
else:
  KEY = ""
  dimcli.login(key=KEY, endpoint=ENDPOINT)
dsl = dimcli.Dsl()<jupyter_output><empty_output><jupyter_text>Query for Clinical Trials<jupyter_code>q = """search clinical_trials where category_rcdc.name="Multiple Sclerosis" 
        and active_years=[2017, 2018, 2019] 
        return clinical_trials[basics+publication_ids]"""
df = dsl.query_iterative(q).as_dataframe()
df.head()<jupyter_output>Starting iteration with limit=1000 skip=0 ...[0m
0-1000 / 3353 (3.08s)[0m
1000-2000 / 3353 (3.93s)[0m
2000-3000 / 3353 (2.83s)[0m
3000-3353 / 3353 (1.22s)[0m
===
Records extracted: 3353[0m<jupyter_text>Counting publications per clinical trialBefore we can count publications, we should ensure that all the values are 'countable'. So we have to transform all None values in `publication_ids` into empty lists first.<jupyter_code># replace empty values with empty lists so that they can be counted
for row in df.loc[df.publication_ids.isnull(), 'publication_ids'].index:
    df.at[row, 'publication_ids'] = []<jupyter_output><empty_output><jupyter_text>Now it's ok to count publications<jupyter_code># create new column
df['pubs_tot'] = df['publication_ids'].apply(lambda x: len(x))
# sort 
df.sort_values("pubs_tot", ascending=False, inplace=True)
df.head(5)<jupyter_output><empty_output><jupyter_text>A simple data visualization<jupyter_code>px.bar(df[:200], x="id", y="pubs_tot", 
      hover_name="title", hover_data=["active_years"])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12331536388140162, 'fraction_numerical': 0.03234501347708895, 'mean_word_length': 5.825287356321839, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '416779', 'license': 'MIT', 'n_tokens_mistral': 1093, 'n_tokens_neox': 997, 'n_words': 257, 'path': 'docs/cookbooks/4-clinical-trials/Clinical_Trials_by_Volume_of_Pubs.ipynb', 'repo_name': 'eschares/dimensions-api-lab', 'size': '2968'}