<jupyter_start><jupyter_text>Parsing GEPRIS pages of NFDI consortia for project applicants and participantsThe list of funded NFDI projects with their GEPRIS IDs was saved in the notebook 02. Let's read that file into a dataframe.<jupyter_code>import pandas as pd
df = pd.read_csv("../../../data/GEPRIS_NFDI_all.csv").fillna('')
df<jupyter_output><empty_output><jupyter_text>For testing let's start with BERD@BW consortium only.<jupyter_code>import requests
GEPRIS = "https://gepris.dfg.de/gepris/projekt/460037581"
params = {'language': 'en'}
r = requests.get(GEPRIS, params=params)
text = r.text.encode(r.encoding).decode('utf8')
print(text[0:100].strip())<jupyter_output><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www<jupyter_text>Then the subject area is parsed:<jupyter_code>from bs4 import BeautifulSoup
soup = BeautifulSoup(text, 'html.parser')
results = soup.find_all("div", class_="firstUnderAntragsbeteiligte")
subject_area = results[0].find_all('span')[-1].text.strip().replace('  ', '').replace('\n', ', ')
print(subject_area)<jupyter_output>Social and Behavioural Sciences<jupyter_text>The homepage is parsed:<jupyter_code>try:
    homepage = soup.find('a', class_="extern").get('href')
except:
    homepage = ''
print(homepage)<jupyter_output><jupyter_text>The project description is parsed:<jupyter_code>description = soup.find('div', id="projekttext").text.strip()
print(description)<jupyter_output>The research domains of business, economics, and other social sciences are concerned with the relationships among individuals and organizations within a society. To understand these complex systems, social science disciplines have been using empirical methods for a long time. However, unstructured and non-standard data, i.e., information that either does not have a previously defined data model or is not organized in a predefined manner (e.g., images or videos from social media), are gaining in relevance. The generation of continuous data streams in society and economy (datafication) strengthens this trend: It is estimated that by 2025 80% of the data processed in economic applications will be available in unstructured form. Because of the sheer size and, more importantly, the lack of structure and the heterogeneity of raw digital data, the BERD@NFDI community calls for innovative and reusable methods, mostly from artificial intelligence and machine learning, as well as a suitable stor[...]<jupyter_text>Let's parse the rest of useful data, put it into `berd` dictionary and pretty print it:<jupyter_code>s = soup.find_all('div', class_="content_frame")
berd = {}
for k in s[1].find_all('div'):
    try:
        t = k.find('span', class_="name")
        if t.get_text().strip()!="DFG Programme" and k.find_all('a', class_="intern"):
            berd[t.get_text().strip()] = [ (g.get_text(), "https://gepris.dfg.de" + g.get('href')) for g in k.find_all('a', class_="intern")]
    except:
        pass
from pprint import pprint
pprint(berd)<jupyter_output>{'Applying institution': [('Universität Mannheim',
                           'https://gepris.dfg.de/gepris/institution/10045')],
 'Co-Spokespersons': [('Professor Dr. Bernd  Bischl',
                       'https://gepris.dfg.de/gepris/person/289708818'),
                      ('Professor Dr. Stefan  Dietze',
                       'https://gepris.dfg.de/gepris/person/218646654'),
                      ('Professor Dr. Marc  Fischer',
                       'https://gepris.dfg.de/gepris/person/1804971'),
                      ('Dr. Sabine  Gehrlein',
                       'https://gepris.dfg.de/gepris/person/398007310'),
                      ('Professor Dr. Mark  Heitmann',
                       'https://gepris.dfg.de/gepris/person/216564778'),
                      ('Professor Dr. Hartmut  Höhle',
                       'https://gepris.dfg.de/gepris/person/442455860'),
                      ('Professor Dr. Göran  Kauermann',
                       'https://gepris.dfg.de/gepris/pers[...]<jupyter_text>Now we can loop over all consortia and get that data about them automatically:<jupyter_code>nfdi = {}
for GEPRIS, title in zip(df['GEPRIS'],df['Title']):
    nfdi[GEPRIS] = {}
    nfdi[GEPRIS]['title'] = title
    params = {'language': 'en'}
    r = requests.get(GEPRIS, params=params)
    text = r.text.encode(r.encoding).decode('utf8')
    soup = BeautifulSoup(text, 'html.parser')
    results = soup.find_all("div", class_="firstUnderAntragsbeteiligte")
    subject_area = results[0].find_all('span')[-1].text.strip().replace('  ', '').replace('\n', ', ')
    nfdi[GEPRIS]['subject_area'] = subject_area
    try:
        homepage = soup.find('a', class_="extern").get('href')
    except:
        homepage = ''
    nfdi[GEPRIS]['homepage'] = homepage
    description = soup.find('div', id="projekttext").text.strip()
    nfdi[GEPRIS]['description'] = description
    s = soup.find_all('div', class_="content_frame")
    for k in s[1].find_all('div'):
        try:
            t = k.find('span', class_="name")
            if t.get_text().strip()!="DFG Programme" and k.find_all('a', class_="intern"):
                nfdi[GEPRIS][t.get_text().strip()] = [ (g.get_text(), "https://gepris.dfg.de" + g.get('href')) for g in k.find_all('a', class_="intern")]
        except:
            pass
import pandas as pd
df_nfdi = pd.DataFrame(nfdi).fillna('')
df_nfdi.T<jupyter_output><empty_output><jupyter_text>Let's save the parsed data into a CSV-file:<jupyter_code>df_nfdi.T.to_csv("../../../data/GEPRIS_NFDI_project_pages.csv", index=True, index_label='gepris', encoding='utf-8')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14326241134751774, 'fraction_numerical': 0.016134751773049644, 'mean_word_length': 3.635168447000822, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 28, 'https://': 11, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '480137', 'license': 'MIT', 'n_tokens_mistral': 1921, 'n_tokens_neox': 1798, 'n_words': 436, 'path': 'docs/docs/parsing/03_parsing_GEPRIS_projects.ipynb', 'repo_name': 'UB-Mannheim/NFDI', 'size': '5640'}