<jupyter_start><jupyter_text>A first look at ELIXIR tools in Bio.Tools registry  Load data into a pandas DataFrame via json<jupyter_code>%matplotlib inline
import requests
import json
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import re
import pylab

# Make the graphs a bit prettier, and bigger
pd.set_option('display.mpl_style', 'default') 
pd.set_option('display.width', 5000) 
pd.set_option('display.max_columns', 60) 
pylab.rcParams['figure.figsize'] = 15,5
matplotlib.rcParams.update({'font.size': 14})<jupyter_output><empty_output><jupyter_text>Convert json  into a pandas dataframeThis will generate some complicated datastructures for nested attributes but these are dealt with when (if) analysed<jupyter_code>%time r = requests.get('https://bio.tools/api/tool')
print ("Response Status:", r.status_code)
r.headers['content-type']
bt=json.loads(r.text)
bioTools=pd.DataFrame(bt)<jupyter_output>CPU times: user 70.7 ms, sys: 46.7 ms, total: 117 ms
Wall time: 8.46 s
Response Status: 200<jupyter_text>Check content interactively<jupyter_code>bioTools<jupyter_output><empty_output><jupyter_text>Current size of the registry , Number of top-level attributes<jupyter_code>bioTools.shape<jupyter_output><empty_output><jupyter_text>Filter out the ELIXIR tools using heuristicsNext try out the selection tools for the DataFrame to select the ones that either have "ELIXIR" or "de.NBI" in collections attribute or, alternatively, have the "elixirInfo" attribute These are added to separate dataframes that are later combined. Use binary selection vector to filter out elements with the 'elixirInfo' flag set<jupyter_code>elixirInfo=bioTools[bioTools.elixirInfo.notnull()]
elixirInfo.shape<jupyter_output><empty_output><jupyter_text>For the other ELIXIR tools - which forms part and are annotated as 'collections' use regexInclude Germany for completeness<jupyter_code>pat=re.compile(r'ELIXIR|de\.NBI')<jupyter_output><empty_output><jupyter_text>Drop the tools without any collection annotation and then select if they match the regex<jupyter_code>ct=bioTools[bioTools['collection'].notnull()]
elixirCol=ct[[True if (pat.search(' '.join(x))) else False for x in ct['collection']]]
elixirCol.shape
elixirTools=pd.concat([elixirInfo,elixirCol])
elixirTools.shape
elixirTools<jupyter_output><empty_output><jupyter_text>Use count function for overview of attributes<jupyter_code>elixirTools.count()<jupyter_output><empty_output><jupyter_text>Analysis of ELIXIR tools in the registry Currently not very complete and needs checking against node proposals   Most of the tools associated with ELIXIR Nodes don't have the elixirInfo attribute<jupyter_code>elixirTools.elixirInfo
eInfo=pd.concat(
    [pd.DataFrame(x,index=[0],columns=x.keys()) if type(x)==type({}) else pd.DataFrame([x],columns=['_ignore_'])
     for x in elixirTools.elixirInfo]
    ,ignore_index=True
)

eInfo['elixirNode'].fillna('Missing', inplace=True)
eInfo.groupby('elixirNode').aggregate('size').plot('bar')
eInfo['elixirStatus'].fillna('Missing', inplace=True)
eInfo.groupby('elixirStatus').aggregate('size').plot('bar')<jupyter_output><empty_output><jupyter_text>EDAM annotations of the ELIXIR Tools<jupyter_code>topic=pd.concat(
    [pd.DataFrame(x[0],index=[0],columns=x[0].keys()) for x in elixirTools.topic ]
    ,ignore_index=True
)
print(topic.groupby('term').aggregate('size'))
topic.groupby('term').aggregate('size').plot('bar')<jupyter_output><empty_output><jupyter_text>ELIXIR tools are annotated at very different levels in the ontology - need to agree on annotation practice for e.g.  gap analysis   License information availiable for ELIXIR Tools<jupyter_code>elixirTools['license'].fillna('Missing', inplace=True)

print( elixirTools.groupby('license').aggregate('size'))
elixirTools.groupby('license').aggregate('size').plot('bar')<jupyter_output>license
Apache License 2.0                                        2
BSD 3-Clause License (Revised)                            6
Creative Commons Attribution NonCommercial ShareAlike     1
Creative Commons Attribution Share Alike                  4
GNU General Public License v2                             3
GNU General Public License v3                            10
GNU Lesser General Public License v2.1                    7
MIT License                                               2
Missing                                                  96
dtype: int64<jupyter_text>Most tools associated with ELIXIR Nodes have no information on the  license - but those that have are open sourceThis is obviously the reasone for  the open source working group  Life cycle information of ELIXIR ToolsWP3 in EXCELERATE are developing a life-cycle framework. This should be adopted consistently for the ELIXIR Tools<jupyter_code>elixirTools['maturity'].fillna('Missing', inplace=True)
print(elixirTools.groupby('maturity').aggregate('size'))
elixirTools.groupby('maturity').aggregate('size').plot('bar')<jupyter_output>maturity
Deprecated     5
Early         10
Missing       47
Stable        69
dtype: int64<jupyter_text>Most tools have some level of annotation. There are Deprecated tools associeted with ELIXIR Nodes - we need active management of the tools. But this could be an artefact of the heuristics above  Conditions for tool access<jupyter_code>elixirTools['accessibility'].fillna('Missing', inplace=True)
print(elixirTools.groupby('accessibility').aggregate('size'))
elixirTools.groupby('accessibility').aggregate('size').plot('bar')<jupyter_output>accessibility
Missing    42
Public     89
dtype: int64<jupyter_text>Supported platforms for tools<jupyter_code>elixirTools.platform<jupyter_output><empty_output><jupyter_text>Reshape the list in the attribute to a proper dataframe. Summarise and plot<jupyter_code>p=pd.concat(
    [pd.DataFrame(
            dict(zip(x,[1]*len(x))),
            index=[0]
            )
    if type(x)==type([])
    else pd.DataFrame([x],index=[0],columns=['_ignore_']) 
    for x in elixirTools.platform
    ],
    ignore_index=True
)
platforms=p.drop('_ignore_', axis=1)        
platforms
platforms.count().plot('bar')<jupyter_output><empty_output><jupyter_text>Reasonably cross platform - but most tools have no platform assignment:<jupyter_code>noplat=platforms[platforms.Mac.isnull() & platforms.Windows.isnull()&  platforms.Linux.isnull()]
pd.DataFrame({'Platform Specified':platforms.shape[0]-noplat.shape[0], 'No Platform specified':noplat.shape[0]},
             index=[0]).sum().plot('bar')
platforms.sum(axis=1).value_counts().plot('bar')<jupyter_output><empty_output><jupyter_text>Majority of tools have no platform assignment - but of those that have most are cross-platform  Resource type? Database or Tool?<jupyter_code>p=pd.concat(
    [pd.DataFrame(
            dict(zip(x,[1]*len(x))),
            index=[0]
            )
    if type(x)==type([])
    else pd.DataFrame([x],index=[0],columns=['_ignore_']) 
    for x in elixirTools.resourceType
    ],
    ignore_index=True
)
if '_ignore_' in p.keys():
    resourceType=p.drop('_ignore_', axis=1)
else:
    resourceType=p
resourceType.count().plot('bar')<jupyter_output><empty_output><jupyter_text>This sums to more than 100% - multiple annotations (which is fine of course)<jupyter_code>elixirTools.groupby('affiliation').aggregate('size')<jupyter_output><empty_output><jupyter_text>Documentation, publication and interfaces<jupyter_code>pd.DataFrame({
        "Docs provided":elixirTools.docs.notnull().sum(),
        "Docs missing":elixirTools.docs.isnull().sum()
             }, index=[0]
            ).sum().plot('bar')<jupyter_output><empty_output><jupyter_text>User interfaces - webtools dominate<jupyter_code>interface=pd.concat(
    [pd.DataFrame(x[0],index=[0],columns=x[0].keys()) for x in elixirTools.interface ]
    ,ignore_index=True
)
interface.groupby('interfaceType').aggregate('size').plot('bar')<jupyter_output><empty_output><jupyter_text>References to the tool<jupyter_code>pubs=pd.DataFrame([x for x in elixirTools.publications], index=elixirTools.name)
pd.DataFrame(
    {'Reference given':pubs[pubs.publicationsPrimaryID!='None'].shape[0],
     'No reference given':pubs[pubs.publicationsPrimaryID=='None'].shape[0]
     }, index=[0]).sum().plot('bar')<jupyter_output><empty_output><jupyter_text>However where there is a reference given the format of the reference is very variable and requires heuristics to resolve. Perhaps add another key to the dictionary for ePMC / PM identifier and doi?<jupyter_code>pubs<jupyter_output><empty_output><jupyter_text>And given the reference we can retrieve the citations Citation counter using ePMC restful API_this is a bit of a quick hack but does the job. Note the regex branching to deal with PMID or DOI as publication identifier. It would be nice with 2 fields..._<jupyter_code>def citationCount(ref):
    pmid=re.compile(r'\d{8,8}')
    doi=re.compile(r'10\.\d{4,4}\S+')
    pmcsearch='http://www.ebi.ac.uk/europepmc/webservices/rest/search?'
    m1=pmid.search(ref)
    m2=doi.search(ref)
    if m1: #matching a PMID
        qstring='query=%28EXT_ID:"'+m1.group()+'"%29&format=json'
        qparam=pmcsearch+qstring
        r=requests.get(qparam)
        if r.status_code ==200 :
            art=json.loads(r.text)
            try:
                return(int(art['resultList']['result'][0]['citedByCount']))
            except:
                print (art)
                return None
    elif m2: 
        #print (ref, 'doi', m2.group())
        qstring='query=%28DOI:"'+m2.group()+'"%29&format=json'
        qparam=pmcsearch+qstring
        r=requests.get(qparam)
        if r.status_code ==200 :
            art=json.loads(r.text)
            try:
                return(int(art['resultList']['result'][0]['citedByCount']))
            except:
                print (art)
                return None
        else:
            return 0<jupyter_output><empty_output><jupyter_text>Retrieve citations from ePMC webservices, for records that fail we simply echo the JSON header...<jupyter_code>cit=pd.DataFrame([citationCount(x) for x in pubs.publicationsPrimaryID], columns=['Citations'], index=pubs.index)
pylab.rcParams['figure.figsize'] = 15,40
cit.sum('columns').plot('barh')
pylab.rcParams['figure.figsize'] = 15,5<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1183558449130308, 'fraction_numerical': 0.012340880380915364, 'mean_word_length': 4.636363636363637, 'pattern_counts': {'":': 2, '<': 93, '<?xml version=': 0, '>': 93, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '273798', 'license': 'MIT', 'n_tokens_mistral': 3384, 'n_tokens_neox': 3254, 'n_words': 751, 'path': 'ELIXIR annotations in Bio.Tools registry.ipynb', 'repo_name': 'elixirhub/ELIXIR-service-analytics', 'size': '10291'}