<jupyter_start><jupyter_text>Compare EC orphan drugs with all EC registered active drugsSource of data: 1. Active: https://ec.europa.eu/health/documents/community-register/html/reg_od_act.htm?sort=n2. Withdrawn, suspended, expired, not renewed: https://ec.europa.eu/health/documents/community-register/html/reg_hum_nact.htm?sort=a3. Rejected: https://ec.europa.eu/health/documents/community-register/html/reg_hum_refus.htm  Import data  EC all drugs<jupyter_code>import os

import numpy as np
import pandas as pd

path = os.path.dirname(os.getcwd())
data_all_drugs = pd.read_csv(path + '/data/ec_all_drugs.csv')
data_all_drugs['EU #'] = data_all_drugs['EU #'].astype(str)
data_all_drugs['Brand name'] = data_all_drugs['Brand name'].str.upper()
data_all_drugs<jupyter_output><empty_output><jupyter_text>EC orphan drug designations<jupyter_code>path = os.path.dirname(os.getcwd())
data_od_raw = pd.read_csv(path + '/data/ec_orphan_designations.csv', header=2)
data_od_raw['EU #'] = data_od_raw['EU #'].astype(str)
data_od_raw<jupyter_output><empty_output><jupyter_text>Only keep orphan drug designations between 2001 and 2019<jupyter_code>years = pd.to_datetime(
    data_od_raw['Designation date']).apply(lambda x: x.year)
data_od = data_od_raw.where((years > 2000) & (years < 2020))
data_od = data_od.dropna()
data_od['EU #'] = data_od['EU #'].astype(str)
data_od<jupyter_output><empty_output><jupyter_text>1. How many ODs can be identified by EU  in the all drugs registry<jupyter_code>import plotly.colors
import plotly.graph_objects as go

# Get all EU # in all drug registry
eu_numbers_all_drugs = data_all_drugs['EU #'].unique()

# Get all EU # in OD drug registry
eu_numbers_od_drugs = data_od['EU #'].unique()

# Check whether OD drug number can be found
is_found = np.zeros(shape=len(eu_numbers_od_drugs), dtype=bool)
for idx, number in enumerate(eu_numbers_od_drugs):
    if number in eu_numbers_all_drugs:
        is_found[idx] = True

# Create dataframe
matched_data = pd.DataFrame(columns=['EU #', 'Match status'])
matched_data = matched_data.append(pd.DataFrame({
    'EU #': eu_numbers_od_drugs[is_found],
    'Match status': 'In all drugs registry'}))
matched_data = matched_data.append(pd.DataFrame({
    'EU #': eu_numbers_od_drugs[~is_found],
    'Match status': 'Not in all drugs registry'}))

# Create figure
fig = go.Figure()
colours = plotly.colors.qualitative.Plotly

n_in_all_drugs = len(
    matched_data[matched_data['Match status'] == 'In all drugs registry'])
n_not_in_all_drugs = len(
    matched_data[matched_data['Match status'] == 'Not in all drugs registry'])

fig.add_trace(go.Pie(
    labels=['In all drugs registry', 'Not in all drugs registry'],
    values=[n_in_all_drugs, n_not_in_all_drugs],
    sort=False,
    pull=[0, 0.2],
    marker=dict(
        colors=colours[:2], 
        line=dict(color='#000000', width=1))))
fig.show()<jupyter_output><empty_output><jupyter_text>The EC all drugs registry does not include any OD applications  2. How many OD drugs are repurposed drugs?We check how many of the OD drug products (i.e. the international non-proprietary name which identifies the active ingredient of the medicine (INN)) are identical to those of drugs in the EC all drugs registry.TODO:1. Get INNs of EC drugs by cross-referencing EC drugs with EMA dataset.2. Then check how many EC ODs products are in EC all drugs.Procedure:- Filter NaNs- Capitalise all product names / INNs- Check whether OD product is identical to an INN in all drugs dataset  2.1 Get INN / product names from EMA datasetThe EC dataset does not keep track of the INN for all drugs, so we would like to match drugs by EU  to get the INN from the EMA dataset. But **EU ** is not available for EMA all drugs registry. Alternative we attempt to match drugs based on Brand name.<jupyter_code>path = os.path.dirname(os.getcwd())
data_ema = pd.read_excel(path + '/data/ema_all_drugs.xlsx', header=8)
data_ema['Medicine name'] = data_ema['Medicine name'].str.upper()
data_ema
data_ema.columns<jupyter_output><empty_output><jupyter_text>Check whether some brand names occur multiple times<jupyter_code>non_unique_brands = []
for brand in data_all_drugs['Brand name'].unique():
    mask = data_all_drugs['Brand name'] == brand
    if len(data_all_drugs[mask]) > 1:
        non_unique_brands.append(brand)

non_unique_brands<jupyter_output><empty_output><jupyter_text>We manually checked: each non-unique brand appears only twice, one of which is 'active' the other is either 'not active' or 'rejected'. So it's likely that the same brand name labels the exact same drug and the multiple matches correspond to repeated applications or renewals.  Match unique EC brands with EMA dataset and thereby find product numberFor now: If EC brand name is contained in EMA brand name, we match. In the worst case scenario, this includes more active subtances in the resulting dataset than there truly are. But on the other hand, those active may also belong to non-OD drugs according to the EMA database (possibly explicitly exclude OD drugs in EMA database).<jupyter_code>is_identified = []
inn = []
ec_brand = []
matched_ema_brand = []
brands = data_all_drugs['Brand name'].unique()
ema_brands = data_ema['Medicine name'].unique()
for brand in brands:
    is_matched = False
    for ema_brand in ema_brands:
        # Check whether ec brand is subset of ema brand
        if brand in ema_brand:
            is_identified.append('yes')
            matched_ema_brand.append(ema_brand)
            ec_brand.append(brand)

            # Mask for product
            mask = data_ema['Medicine name'] == ema_brand
            product = data_ema[mask]['Active substance'].iloc[0]
            inn.append(product)

            is_matched = True
    
    if is_matched is True:
        continue

    # If not identified
    is_identified.append('no')
    matched_ema_brand.append('none')
    inn.append('none')
    ec_brand.append(brand)


matched_drugs = pd.DataFrame({
    'Brand name': ec_brand,
    'EMA brand name': matched_ema_brand,
    'is identified': is_identified,
    'INN': inn})
matched_drugs<jupyter_output><empty_output><jupyter_text>Check unmatched drugsSome EC brand names could not be found. For EC brands of the form "name 1/name 2" or "name 1 (name 2)" the above matching method might miss the drugs, because the complete EC brand name has to be contained in the EMA brand name. We check those candidates manually below.<jupyter_code>mask = matched_drugs['is identified'] == 'no'
matched_drugs[mask]<jupyter_output><empty_output><jupyter_text>Filter all names which are just one word<jupyter_code>critical_brand_names = []
uncritical_brand_names = []
for brand in matched_drugs[mask]['Brand name']:
    if (' ' in brand) or ('/' in brand) or ('(' in brand) or ('-' in brand):
        critical_brand_names.append(brand)
    else:
        uncritical_brand_names.append(brand)

print('Critical drugs: ', len(critical_brand_names), '\n')
print(critical_brand_names)
print('\n')
print('Uncritical drugs: ', len(uncritical_brand_names), '\n')
print(uncritical_brand_names)<jupyter_output>Critical drugs:  36 

['AMLODIPINE/VALSARTAN MYLAN', 'ARIKAYCE LIPOSOMAL', 'BROPAIR SPIROMAX', 'CLOPIDOGREL TAW PHARMA', 'CLOPIDOGREL/ACETYLSALICYLIC ACID MYLAN', 'COVID-19 VACCINE JANSSEN', 'COVID-19 VACCINE MODERNA', 'EMTRICITABINE /TENOFOVIR DISOPROXIL KRKA D.D.', 'EXPAREL LIPOSOMAL', 'HEPLISAV B', 'IRBESARTAN / HYDROCHLOROTHIAZIDE TEVA', 'IRBESARTAN HCT ZENTIVA', 'LAMIVUDINE / ZIDOVUDINE TEVA', 'LENALIDOMIDE KRKA', 'LENALIDOMIDE KRKA D.D.', 'LENALIDOMIDE KRKA D.D. NOVO MESTO', 'LENALIDOMIDE MYLAN', 'OBILTOXAXIMAB SFL', 'RIBAVIRIN TEVA PHARMA BV', 'RIVAROXABAN ACCORD', 'SEFFALAIR SPIROMAX', 'SUNITINIB ACCORD', 'THIOTEPA RIEMSER', 'TRIXEO AEROSPHERE', 'CLOPIDOGREL / ACETYLSALICYLIC ACID TEVA', 'HUMALOG-HUMAJECT', 'HUMALOG-PEN', 'IRBESARTAN HCT BMS', 'RIVASTIGMINE 3M HEALTH CARE LTD.', 'TECNEMAB-K-1', 'TENECTEPLASE BOEHRINGER INGELHEIM PHARMA GMBH & CO. KG', 'VALDYN (EX KUDEQ)', 'MYLOTARG REFUSAL', 'RAMELTEON TAKEDA GLOBAL RESEARCH AND DEVELOPMENT CENTRE (EUROPE) LTD', 'RAXONE REFUSAL[...]<jupyter_text>Try to match critical brands that are obvious to parse<jupyter_code># Parse critical name manually
parse_dict = {
    'AMLODIPINE/VALSARTAN MYLAN': ['AMLODIPINE', 'VALSARTAN MYLAN'],
    'CLOPIDOGREL/ACETYLSALICYLIC ACID MYLAN': ['CLOPIDOGREL', 'ACETYLSALICYLIC ACID MYLAN'],
    'EMTRICITABINE /TENOFOVIR DISOPROXIL KRKA D.D.': ['EMTRICITABINE', 'TENOFOVIR DISOPROXIL KRKA D.D.'],
    'IRBESARTAN / HYDROCHLOROTHIAZIDE TEVA': ['IRBESARTAN', 'HYDROCHLOROTHIAZIDE TEVA'],
    'LAMIVUDINE / ZIDOVUDINE TEVA': ['LAMIVUDINE', 'ZIDOVUDINE TEVA'],
    'CLOPIDOGREL / ACETYLSALICYLIC ACID TEVA': ['CLOPIDOGREL', 'ACETYLSALICYLIC ACID TEVA']}

# Find parsed names in EMA dataset
ema_brands = data_ema['Medicine name'].unique()
for brand, parsed_brands in parse_dict.items():
    matched_brands = []
    inn = []
    for parsed_brand in parsed_brands:
        for ema_brand in ema_brands:
            # Check whether ec brand is subset of ema brand
            if parsed_brand in ema_brand:
                matched_brands.append(ema_brand)

                # Mask for product
                mask = data_ema['Medicine name'] == ema_brand
                product = data_ema[mask]['Active substance'].iloc[0]
                inn.append(product)

                # is_matched = True
    
    # Make sure that parsed drugs are not matched to different drugs
    if len(matched_brands) == 2:
        assert matched_brands[0] == matched_brands[1]

    # Update if one of the parsed drugs was matched
    if len(matched_brands) > 0:
        mask = matched_drugs['Brand name'] == brand
        matched_drugs.loc[mask, 'EMA brand name'] = matched_brands[0]
        matched_drugs.loc[mask, 'is identified'] = 'yes'
        matched_drugs.loc[mask, 'INN'] = inn[0]
mask = \
    (matched_drugs['Brand name'] == 'AMLODIPINE/VALSARTAN MYLAN') | \
    (matched_drugs['Brand name'] == 'CLOPIDOGREL/ACETYLSALICYLIC ACID MYLAN') | \
    (matched_drugs['Brand name'] == 'EMTRICITABINE /TENOFOVIR DISOPROXIL KRKA D.D.') | \
    (matched_drugs['Brand name'] == 'IRBESARTAN / HYDROCHLOROTHIAZIDE TEVA') | \
    (matched_drugs['Brand name'] == 'LAMIVUDINE / ZIDOVUDINE TEVA') | \
    (matched_drugs['Brand name'] == 'CLOPIDOGREL / ACETYLSALICYLIC ACID TEVA')
matched_drugs[mask]<jupyter_output><empty_output><jupyter_text>Check double matches<jupyter_code>double_matches = []
brands = matched_drugs['Brand name'].unique()
for brand in brands:
    mask = matched_drugs['Brand name'] == brand
    temp = matched_drugs[mask]
    if len(temp) > 1:
        double_matches.append(temp['Brand name'].iloc[0])

len(double_matches)<jupyter_output><empty_output><jupyter_text>Match EC OD drugs based on product  1. Identical match  Check INN of EC all drugs (mostly single words, 2 words, or even a list of compounds?)<jupyter_code>critical_inns = []
uncritical_inns = []
inns = matched_drugs['INN'].unique()
for inn in inns:
    # Count number of spaces
    space_count = 0
    for letter in inn:
        if letter.isspace():
            space_count += 1
    if (space_count > 1) or (',' in inn) or ('/' in inn):
        critical_inns.append(inn)
    else:
        uncritical_inns.append(inn)

print('Critical INNs: ', len(critical_inns), '\n')
print(critical_inns)
print('\n')
print('Uncritical INNs: ', len(uncritical_inns), '\n')
print(uncritical_inns)<jupyter_output>isartan, amlodipine', 'cobicistat on silicon dioxide', 'indacaterol, Glycopyrronium bromide', 'Glycopyrronium bromide, indacaterol maleate', 'meropenem trihydrate, vaborbactam', 'vibrio cholerae, strain cvd 103-hgr, live', 'Diphtheria toxoid, tetanus toxoid, Bordetella pertussis antigens: pertussis toxoid, filamentous haemagglutinin, pertactin, fimbriae Types 2 and 3, hepatitis B surface antigen produced in yeast cells, poliovirus (inactivated): type 1 (Mahoney), type 2 (MEF-1), type 3 (Saukett) produced in Vero cells/ Haemophilus influenzae type b polysaccharide (polyribosylribitol phosphate) conjugated to meningococcal protein.', 'mixture of polynuclear iron(III)-oxyhydroxide, sucrose and starches', 'patiromer sorbitex calcium', 'tenofovir alafenamide fumarate', 'Ombitasvir, paritaprevir, ritonavir', 'recombinant human n-acetylgalactosamine-6-sulfatase (rhgalns)', 'alogliptin benzoate, metformin hydrochloride', 'tenofovir disoproxil fumarate', 'canagliflozin, metformin hydrochloride'[...]<jupyter_text>Check INN of EC OD drugs (mostly single words, 2 words, or even a list of compounds?)<jupyter_code>critical_inns = []
uncritical_inns = []
inns = data_od['Product'].unique()
for inn in inns:
    # Count number of spaces
    space_count = 0
    for letter in inn:
        if letter.isspace():
            space_count += 1
    if (space_count > 1) or (',' in inn) or ('/' in inn):
        critical_inns.append(inn)
    else:
        uncritical_inns.append(inn)

print('Critical INNs: ', len(critical_inns), '\n')
print(critical_inns)
print('\n')
print('Uncritical INNs: ', len(uncritical_inns), '\n')
print(uncritical_inns)<jupyter_output>inant human hepatitis C monoclonal antibody against C4 region of E1', 'Mercaptopurine (oral liquid)', 'Methotrexate (oral liquid)', '4-ethoxy-2-(piperazin-1-yl)-7-(pyridin-4-yl)-5H-pyrimido[5,4-b]indol', 'Adenovirus associated viral vector serotype 4 containing the human RPE65 gene', '4-Amino-1-[5-O-[(2R,4S)-2-oxido-4-(4-pyridinyl)-1,3,2-dioxaphosphorinan-2-yl]-ß-D-arabinofuranosyl]-2(1H)-pyrimidinone', 'Alginate oligosaccharide (G-block) fragment', 'Human coagulation factor X', 'L-threo-3,4-dihydroxyphenylserine', 'Pyridoxalated haemoglobin polyoxyethylene', '1-{3-[3-(4-chlorophenyl)propoxy]propyl}piperidine, hydrochloride', 'Recombinant fusion protein consisting of human coagulation factor IX attached to the Fc domain of human IgG1', 'R-1-[2,3-dihydro-2-oxo-1-pivaloylmethyl-5-(2-pyridyl)-1 H -1,4-benzodiazepin-3-yl]-3-(3-methylaminophenyl)urea', 'Autologous CD34+ cells transfected with lentiviral vector containing the human arylsulfatase A cDNA', 'Antisense Oligonucleotide (TATCCGGAG[...]<jupyter_text>It's not obvious how to parse most of the INNs, except for deleting additional information such as "(oral use)", "(rectal use)", etc.  1. Identical match based on "raw" INNs<jupyter_code>is_identified = []
matched_all_drug_inns = []
matched_od_inns = []
all_drug_brand = []
od_inns = data_od['Product'].str.upper()
all_drugs_inns = matched_drugs['INN'].str.upper().unique()
for od_inn in od_inns:
    is_matched = False
    for all_drug_inn in all_drugs_inns:
        # Check whether ec brand is subset of ema brand
        if od_inn in all_drug_inn:
            is_identified.append('yes')
            matched_all_drug_inns.append(all_drug_inn)
            matched_od_inns.append(od_inn)

            # Mask for brand
            mask = matched_drugs['INN'].str.upper() == all_drug_inn
            brand = matched_drugs[mask]['Brand name'].iloc[0]
            all_drug_brand.append(brand)

            is_matched = True
    
    if is_matched is True:
        continue

    # If not identified
    is_identified.append('no')
    matched_all_drug_inns.append('none')
    matched_od_inns.append(od_inn)
    all_drug_brand.append('none')


matched_ods = pd.DataFrame({
    'OD INN': matched_od_inns,
    'All drug INN': matched_all_drug_inns,
    'is identified': is_identified,
    'All drug brand name': all_drug_brand})
matched_ods
mask = matched_ods['is identified'] == 'yes'
matched_ods[mask]<jupyter_output><empty_output><jupyter_text>There are some ODs that are falsely matched to multiple drugs, e.g. CHOLIC ACID is matched OBETICHOLIC ACID. Those will have to be filtered out manually. 1.1 Filter multiple matches<jupyter_code>double_matches = []
mask = matched_ods['is identified'] == 'yes'
inns = matched_ods[mask]['OD INN'].unique()
for inn in inns:
    mask = matched_ods['OD INN'] == inn
    temp = matched_ods[mask]
    if len(temp) > 1:
        double_matches.append(temp['OD INN'].iloc[0])

len(double_matches)<jupyter_output><empty_output><jupyter_text>How many of those 55 double matches are falsely matched multiple times? Keep only those double matches that are matched to different All drug INNs.<jupyter_code>double_matches = pd.DataFrame(columns=['OD INN', 'All drug INN', 'is identified', 'All drug brand name'])
mask = matched_ods['is identified'] == 'yes'
inns = matched_ods[mask]['OD INN'].unique()
for inn in inns:
    mask = matched_ods['OD INN'] == inn
    temp = matched_ods[mask]
    if len(temp) > 1:
        unique_inns = temp['All drug INN'].unique()
        if len(unique_inns) > 1:
            double_matches = double_matches.append(temp)

double_matches
double_matches.iloc[:50]
double_matches.iloc[50:]
fig = go.Figure()
fig.add_trace(go.Pie(
    labels=['Recycled INNs', 'Not recycled'],
    values=[219, 1629 - 219],
    sort=False,
    pull=[0, 0.2],
    marker=dict(
        colors=colours[:2], 
        line=dict(color='#000000', width=1))))
fig.show()<jupyter_output><empty_output><jupyter_text>Make the matching more strict<jupyter_code>is_identified = []
inn = []
ec_brand = []
matched_ema_brand = []
brands = data_all_drugs['Brand name'].unique()
ema_brands = data_ema['Medicine name'].unique()
for brand in brands:
    is_matched = False
    for ema_brand in ema_brands:
        # Check whether ec brand is subset of ema brand
        if brand == ema_brand:
            is_identified.append('yes')
            matched_ema_brand.append(ema_brand)
            ec_brand.append(brand)

            # Mask for product
            mask = data_ema['Medicine name'] == ema_brand
            product = data_ema[mask]['Active substance'].iloc[0]
            inn.append(product)

            is_matched = True
    
    if is_matched is True:
        continue

    # If not identified
    is_identified.append('no')
    matched_ema_brand.append('none')
    inn.append('none')
    ec_brand.append(brand)


matched_drugs = pd.DataFrame({
    'Brand name': ec_brand,
    'EMA brand name': matched_ema_brand,
    'is identified': is_identified,
    'INN': inn})
matched_drugs
len(matched_drugs[matched_drugs['is identified'] == 'yes'])
is_identified = []
matched_all_drug_inns = []
matched_od_inns = []
all_drug_brand = []
od_inns = data_od['Product'].str.upper()
all_drugs_inns = matched_drugs['INN'].str.upper().unique()
for od_inn in od_inns:
    is_matched = False
    for all_drug_inn in all_drugs_inns:
        # Check whether ec brand is subset of ema brand
        if od_inn == all_drug_inn:
            is_identified.append('yes')
            matched_all_drug_inns.append(all_drug_inn)
            matched_od_inns.append(od_inn)

            # Mask for brand
            mask = matched_drugs['INN'].str.upper() == all_drug_inn
            brand = matched_drugs[mask]['Brand name'].iloc[0]
            all_drug_brand.append(brand)

            is_matched = True
    
    if is_matched is True:
        continue

    # If not identified
    is_identified.append('no')
    matched_all_drug_inns.append('none')
    matched_od_inns.append(od_inn)
    all_drug_brand.append('none')


matched_ods = pd.DataFrame({
    'OD INN': matched_od_inns,
    'All drug INN': matched_all_drug_inns,
    'is identified': is_identified,
    'All drug brand name': all_drug_brand})
matched_ods
len(matched_ods[matched_ods['is identified'] == 'yes'])
fig = go.Figure()
fig.add_trace(go.Pie(
    labels=['Recycled INNs', 'Not recycled'],
    values=[175, 1629 - 175],
    sort=False,
    pull=[0, 0.2],
    marker=dict(
        colors=colours[:2], 
        line=dict(color='#000000', width=1))))
fig.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11575415586396218, 'fraction_numerical': 0.009506379950180468, 'mean_word_length': 4.281073825503356, 'pattern_counts': {'":': 0, '<': 67, '<?xml version=': 0, '>': 75, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '20577', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 7147, 'n_tokens_neox': 6659, 'n_words': 1728, 'path': 'analysis/ec_all_drugs.ipynb', 'repo_name': 'DavAug/euod', 'size': '19671'}