<jupyter_start><jupyter_text>__PyScopus__: An example for author disambiguity Sometimes Scopus would mix up people with similar names. I recently come up with a not that difficult method to clean author publication profiles, which needs some manual work. If you can think of a better way, please do let me know! ---<jupyter_code>import pyscopus
pyscopus.__version__
from pyscopus import Scopus
key = 'YOUR_OWN_APIKEY'
scopus = Scopus(key)<jupyter_output><empty_output><jupyter_text>--- Wrapper functions for disambiguity<jupyter_code>import requests, time
import pandas as pd
from bs4 import BeautifulSoup as Soup
BASE_URL = "https://api.elsevier.com/content/abstract/scopus_id/"
def _check_pub_validity(sid, author_id, author_affil_id_list, apikey):
    global BASE_URL
    r = requests.get(BASE_URL+sid, params={'apikey': apikey})
    soup = Soup(r.content, 'lxml')
    author_list = soup.find('authors').find_all('author')

    ## go through the author list to find the author first, by matching author id
    for au in author_list:
        if au['auid'] == author_id:
            ## find it and break
            break

    ## check the affiliation id: note that an author may have a list of affiliations
    this_affil_id_list = [affil_tag['id'] for affil_tag in au.find_all('affiliation')]
    ## get the affiliation id and check if there are any overlap
    if len(set(author_affil_id_list).intersection(set(this_affil_id_list))) > 0:
        return True
    return False
def check_pub_validity(scopus_obj, author_id, author_affil_id_list, apikey):
    ## first find out all pub
    pub_df = scopus_obj.search_author_publication(author_id)
    ## do this for all non-null scopus ids
    pub_df = pub_df[~pub_df.scopus_id.isnull()]
    ## list to save all eligble scopus ids
    eligible_scopus_id_list = list()
    for i, sid in enumerate(pub_df.scopus_id.values):
        if (i+1)%5==0:
            time.sleep(pd.np.random.random()+.3)
        if _check_pub_validity(sid, author_id, author_affil_id_list, apikey):
            ## if true, save it
            eligible_scopus_id_list.append(sid)
    ## finally, get a subset of the original pub_df
    filtered_pub_df = pub_df.query("scopus_id in @eligible_scopus_id_list")
    return filtered_pub_df<jupyter_output><empty_output><jupyter_text>--- When I was collecting data for my own research, I found that [Dr. <NAME>](http://web.media.mit.edu/~singhv/) has a very [noisy profile in Scopus](https://www.scopus.com/authid/detail.uri?authorId=7404651152). Let's use this as an example. The basic idea is to match author-affiliation pair:- For all the paper found in the _mixed profile_    - Find the focal author (in this case, Dr. Singh)    - Look at his/her affiliation        - Keep this paper if the affiliation is indeed where he/she is        - If not, discard the paper For Dr. Singh, I manually obtained his affiliation ids by searching through [Scopus affiliation search](https://www.scopus.com/search/form.uri?display=affiliationLookup). Upon obtaining that, create a dictionary containing _name (first/last)_, _affiliation name_, and _a list of affiliation ids_. Author and affiliation names would be used to search for this author. The list of affiliation ids would be used for cleaning papers:  - UC Irvine `60007278`- MIT `60022195`- Rutgers `60030623`<jupyter_code>d = {'authfirst': 'Vivek', 'authlastname': 'Singh', 'affiliation': 'Rutgers',
     'affil_id_list': ['60030623', '60022195', '60007278']
    }
d
query = "AUTHLASTNAME({}) and AUTHFIRST({}) and AFFIL({})".format(d['authlastname'], d['authfirst'], d['affiliation'])
author_search_df = scopus.search_author(query)
author_search_df<jupyter_output>The history saving thread hit an unexpected error (OperationalError('database is locked',)).History will not be written to the database.<jupyter_text>Sometimes we would obtain a list of author profiles for each author. In this case, we only have one and it is clear that the author profile is highly noisy.In the following step, I would use the helper functions in `utils` to screen each paper by this `author_id`<jupyter_code>author_id = '7404651152'
author_id, d['affil_id_list']<jupyter_output><empty_output><jupyter_text>The filtering process may take a while, depending on how many documents are mixd up.<jupyter_code>filterd_pub_df = check_pub_validity(scopus, author_id, d['affil_id_list'], key)
filterd_pub_df.shape[0], filterd_pub_df.scopus_id.unique().size, filterd_pub_df.scopus_id.isnull().sum()<jupyter_output><empty_output><jupyter_text>Obviously, the number of papers is highly reduced. We can now check a random subset to see if the filtered papers make sense for this author.<jupyter_code>filterd_pub_df.iloc[pd.np.random.randint(0, high=134, size=20)][['title', 'publication_name']]<jupyter_output><empty_output><jupyter_text>However, there may still be noise in it (e.g., papers published in optics/photonics venues). We can manually exclude those as well:<jupyter_code>filterd_pub_df = filterd_pub_df.query("not publication_name.str.lower().str.contains('optic')")
filterd_pub_df = filterd_pub_df.query("not publication_name.str.lower().str.contains('photonic')")
filterd_pub_df = filterd_pub_df.query("not publication_name.str.lower().str.contains('nano')")
filterd_pub_df = filterd_pub_df.query("not publication_name.str.lower().str.contains('quantum')")
filterd_pub_df = filterd_pub_df.query("not publication_name.str.lower().str.contains('sensor')")
filterd_pub_df = filterd_pub_df.query("not publication_name.str.lower().str.contains('cleo')")
filterd_pub_df = filterd_pub_df.query("not publication_name.str.lower().str.contains('materials')")
filterd_pub_df = filterd_pub_df.query("not publication_name.str.lower().str.contains('physics')")
filterd_pub_df = filterd_pub_df.query("not publication_name.str.lower().str.contains('chip')")
filterd_pub_df.shape<jupyter_output><empty_output><jupyter_text>And let's check again<jupyter_code>filterd_pub_df.iloc[pd.np.random.randint(0, high=59, size=20)][['title', 'publication_name']]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10817941952506596, 'fraction_numerical': 0.014182058047493404, 'mean_word_length': 5.390937829293994, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 34, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '352394', 'license': 'MIT', 'n_tokens_mistral': 2056, 'n_tokens_neox': 1943, 'n_words': 545, 'path': 'Disambiguition.ipynb', 'repo_name': 'skilkis/python-scopus', 'size': '6072'}