<jupyter_start><jupyter_text>With this notebook you can search all available databases seperately. You can edit the search string for each database and download the results for each. If desired you can join all the results and download a combined excel at the bottom of the page  How-to:    - Start Institutional VPN    - Change variables: Location (where do you want to download to?), query and run the cell    - Search the header for your desired database and run the cells below     - For each database the query is set to the one declared at the top (default). If needed it can be changed for every database to whatever you want. Count-limit is also adjustable to your needs    - Api-callable: Scopus, Science Direct and Arxiv    - The rest is scraped via chromedriver and can take a few minutes, depending how many search results are found<jupyter_code>import scraper_tobiashilt as scrape
from datetime import datetime

#Location where the results should be downloaded to
Location = '/Users/Tobias/Desktop/Python/'
Date = datetime.today().strftime('%Y-%m-%d')


# Search string
query = '("DLT" OR "Distributed Ledger") AND ("Circular Economy" OR "Sustainable supply chain")'<jupyter_output><empty_output><jupyter_text>Scopus<jupyter_code># Api-Key --> https://dev.elsevier.com
# max value for count: 6000 (Api-limit)

key = '...'
query_scopus = query
count = 50

df_Scopus = scrape.scrape_scopus(key, query_scopus, count)
df_Scopus<jupyter_output><empty_output><jupyter_text>Download<jupyter_code>Excel_Scopus = Location + Date + "_Scopus.xlsx"
df_Scopus.to_excel(Excel_Scopus)<jupyter_output><empty_output><jupyter_text>----------------------------------------------------------------------------------------------   Science Direct<jupyter_code># Get your Api-Key --> https://dev.elsevier.com
# Count kann auf max. 6000 gesetzt werden (API-Limit)


key = '...'
insttoken = '...'
query_sd = query
count = 50

df_sciencedirect = scrape.scrape_sd(key,insttoken, query_sd, count)
df_sciencedirect<jupyter_output><empty_output><jupyter_text>Download<jupyter_code>Excel_ScienceDirect = Location + Date + "_ScienceDirect.xlsx"
df_sciencedirect.to_excel(Excel_ScienceDirect)<jupyter_output><empty_output><jupyter_text>----------------------------------------------------------------------------------------------   Arxiv<jupyter_code># Unser Searchstring muss angepasst werden, da keine Ergebnisse gefunden werden. Count anpassen

#query_arxix = query
query_arxiv = "test"
count = 10

df_arxiv = scrape.scrape_arxiv(query_arxiv, count)
df_arxiv<jupyter_output><empty_output><jupyter_text>Download<jupyter_code>Excel_Arxiv = Location + Date + "_Arxiv.xlsx"
df_arxiv.to_excel(Excel_Arxiv)<jupyter_output><empty_output><jupyter_text>----------------------------------------------------------------------------------------------   Web of Science<jupyter_code># Mit Webdriver, da keine API vorhanden und wir keinen IP-Block bekommen möchten (automatischer request laut robots.txt nicht erlaubt)
#--> Kaffee trinken gehen und machen lassen
# Count kann beliebig groß sein
count = 5
#query_wos = "test"
query_wos = query

df_wos = scrape.scrape_wos(query_wos, count)
df_wos<jupyter_output><empty_output><jupyter_text>Download<jupyter_code>Excel_WoS = Location + Date  + "_WoS.xlsx"
df_wos.to_excel(Excel_WoS)<jupyter_output><empty_output><jupyter_text>----------------------------------------------------------------------------------------------   ACM digital<jupyter_code># Mit Webdriver, da keine API vorhanden und wir keinen IP-Block bekommen möchten (automatischer request laut robots.txt nicht erlaubt)
#--> Kaffee trinken gehen und machen lassen
# Count kann beliebig groß sein
count = 51
#query_wos = "test"
query_acm = query

df_acm = scrape.scrape_acm(query_acm, count)
df_acm<jupyter_output><empty_output><jupyter_text>Download<jupyter_code>Excel_acm = Location + Date + "_ACM.xlsx"
df_acm.to_excel(Excel_acm)<jupyter_output><empty_output><jupyter_text>----------------------------------------------------------------------------------------------   IEEE<jupyter_code># Mit Webdriver, da keine API vorhanden und wir keinen IP-Block bekommen möchten (automatischer request laut robots.txt nicht erlaubt)
#--> Kaffee trinken gehen und machen lassen
# Count kann beliebig groß sein
count = 51
#query_wos = "test"
query_ieee = query

df_ieee = scrape.scrape_ieee(query_ieee, count)
df_ieee
Excel_ieee = Location + Date + "_IEEE.xlsx"
df_ieee.to_excel(Excel_ieee)<jupyter_output><empty_output><jupyter_text>----------------------------------------------------------------------------------------------   Emerald Insight<jupyter_code>count = 
query_emerald = query

df_emerald = scrape.scrape_emerald(query_emerald, count)
df_emerald
Excel_emerald = Location + Date + "_Emerald.xlsx"
df_emerald.to_excel(Excel_emerald)<jupyter_output><empty_output><jupyter_text>Combine and download all  Combine your desired databases by including them in "frames"<jupyter_code>import pandas as pd
# default
#frames = [df_Scopus, df_sciencedirect, df_arxiv, df_wos, df_acm, df_ieee, df_emerald]
frames = [df_Scopus, df_sciencedirect, df_arxiv, df_wos]

result = pd.concat(frames, ignore_index=True)
pre = len(result)
result[result['DOI'].isnull() | ~result[result['DOI'].notnull()].duplicated(subset='DOI',keep='first')]
result['Title'] = result['Title'].str.lower()
result.drop_duplicates(subset ='Title', keep = 'first', inplace = True)
after = len(result)
print('Es wurden', pre-after, 'Duplikate entfernt! (Basierend auf DOI oder Titel)')
result.reset_index(inplace = True, drop = True)
result['Cited_by'] = pd.to_numeric(result['Cited_by'])
result
detailed_combined = Location + Date + "_detailed_combined.xlsx"
result.to_excel(detailed_combined)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.18764103454261413, 'fraction_numerical': 0.0032980385349765666, 'mean_word_length': 3.5766481334392375, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 62, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '7683', 'license': 'MIT', 'n_tokens_mistral': 1948, 'n_tokens_neox': 1803, 'n_words': 463, 'path': 'notebooks/Detailed.ipynb', 'repo_name': 'TobiasHilt/scientific_research_scraper', 'size': '5761'}