<jupyter_start><jupyter_text>How are organisations identified in 360Giving's GrantNav? **Author**: <NAME>**Last Updated**: 2017-07-11**Description**:  Analyses the organisation id's used in [GrantNav](http://grantnav.threesixtygiving.org) using the prefixes in the **Recipient Org: Identifier** field.**Contents**: This Jupyter Notebook, data folder AboutThis is an experiment to understand how funding organisations talk about the recipients of their grants by analysing the identifiers they provide for them. **Why identifiers matter**Identifiers are an important part of any dataset. They let a computer uniquely identify and refer to specific grants, organisations, transactions and so-on. Whilst a human being may be good at recognising that:>INDIGO TRUST, The Indigo Trust, and indigo-trust... all refer to the same organisation, computers find this a lot trickier.Identifiers can be valuable for joining up pools of information and improving the quality of the information we hold.**What will this analysis tell us?**We can better understand what identifiers publishers prefer, if we should provide some interventions and where to target our efforts.For example, do publishers prefer their own internal reference? Can we understand why? Do publishers prefer charity numbers over company numbers? How can we support them while making the information more widely useful? Instructions Things to note Things to do<jupyter_code>import os.path
import pandas as pd
from urllib.request import urlretrieve
fileDetails = {'grantnav.csv': 'http://grantnav.threesixtygiving.org/search.csv', 
               'org-id.csv': 'http://org-id.guide/download.csv',
               'country-codes.csv': 'https://raw.githubusercontent.com/datasets/country-codes/master/data/country-codes.csv'}

excludeFunders = ['360G-blf'] # Excludes any Funding Org:Identifier listed
excludeText = ''
if len(excludeFunders) > 0:
    excludeText = '_excludes_'+'-'.join(excludeFunders)
    
outputFileName = 'grantnav_recipient_prefix' + excludeText + '.csv'
# Get GrantNav and Org-Id files
for fname in fileDetails:
    print('Checking for:',fname)
    if not os.path.isfile(fname):
        print('Downloading:',fname)
        urlretrieve(fileDetails[fname], fname)
        
# =======================
# Process GrantNav data
# =======================

# Load files
dfGN = pd.read_csv('grantnav.csv',usecols=['Recipient Org:Identifier','Funding Org:Identifier'])

# Remove excluded funders
if len(excludeFunders) > 0:
    dfGN = dfGN[~dfGN['Funding Org:Identifier'].isin(excludeFunders)]
    
# Split column 
# (Replace anything containing 360G with 360G- to group all publisher-own prefixes)
dfGN.loc[dfGN['Recipient Org:Identifier'].str.contains('360G'), 'Recipient Org:Identifier'] = '360G'
dfGN = pd.DataFrame(dfGN['Recipient Org:Identifier'].str.split('-',2).tolist(),columns = ['split1','split2','discard']).fillna('')
dfGN['code'] = dfGN['split1']+'-'+dfGN['split2']
dfGN = dfGN[['code']]
dfGN['code'] = dfGN.code.str.strip('-')
dfGN = pd.DataFrame({'count' : dfGN.groupby( ['code'] ).size()}).reset_index()
# =======================
# Process GrantNav data
# =======================

# Load files
dfOID = pd.read_csv('org-id.csv',usecols=['code', 'quality', "name/en",'coverage','structure'])
dfOID.rename(columns={"name/en": 'name','coverage': 'country_code'}, inplace=True)
# =====================
# Process country codes
# =====================

dfCC = pd.read_csv('country-codes.csv',usecols={'name','ISO3166-1-Alpha-2'})
dfCC.rename(columns={'name': 'country','ISO3166-1-Alpha-2': 'country_code'}, inplace=True)
# ========================
# Merge & export datasets
# ========================

df = pd.merge(dfGN, dfOID, how='left', on=['code'])
df = pd.merge(df, dfCC.dropna(), how='left', on=['country_code'])
df.loc[df['code'].str.contains('360G'), 'name'] = "Publisher's internal reference"
df[['code','name','country_code','country','structure','count','quality']].fillna('').to_csv(outputFileName,index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.18132141082960754, 'fraction_numerical': 0.011922503725782414, 'mean_word_length': 5.967128027681661, 'pattern_counts': {'":': 1, '<': 6, '<?xml version=': 0, '>': 9, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '101937', 'license': 'MIT', 'n_tokens_mistral': 1260, 'n_tokens_neox': 1162, 'n_words': 332, 'path': 'grantnav_publisher_prefixes/Recipient Organisation Prefixes in GrantNav (360Giving).ipynb', 'repo_name': 'ekoner/labs', 'size': '4034'}