<jupyter_start><jupyter_text>Co-Occurring Tag AnalysisAnalysing how tags co-occur across various Parliamentary publications. The idea behind this is to see whether there are naturally occurring groupings of topic tags by virtue of their co-occurence when used to tag different classes of Parlimanetary publication.The data is provided as a set of Linked Data triples exported as *Turtle* (`.ttl`) data files. The data represents, among other things, Parlimentary resources (such as early day motions or other proceedings records) and subject/topic labels they are tagged with.The data allows us to generate a graph that associates tags with resources, and from that a graph that directly associates tags with other tags by virtue of their commonly tagging the same resource or set of resources.<jupyter_code>#Data files
!ls ../data/dataexport<jupyter_output>[34medms[m[m        [34mproceedings[m[m [34mterms[m[m<jupyter_text>Utils Import a library that lets us work with the data files:<jupyter_code>#Data is provided as Turtle/ttl files - rdflib handles those

#!pip3 install rdflib
from rdflib import Graph<jupyter_output><empty_output><jupyter_text>Simple utility to load all the `.ttl` files in a particular directory into a graph:<jupyter_code>import os
def ttl_graphbuilder(path,g=None,debug=False):
    #We can add the triples to an existing graph or create a new one for them
    if g is None:
        g=Graph()
        
    #Loop through all the files in the directory and then load the ones that have a .ttl suffix
    for ttl in [f for f in os.listdir(path) if f.endswith('.ttl')]:
        if debug: print(ttl)
        g.parse('{}/{}'.format(path,ttl), format='turtle')
    return g<jupyter_output><empty_output><jupyter_text>Tools for running queries over a graph and either printing the result or putting it into a `pandas` dataframe:<jupyter_code>def rdfQuery(graph,q):
    ans=graph.query(q)
    for row in ans:
        for el in row:
            print(el,end=" ")
        print()

#ish via https://github.com/schemaorg/schemaorg/blob/sdo-callisto/scripts/dashboard.ipynb
import pandas as pd
def sparql2df(graph,q, cast_to_numeric=True):
    a=graph.query(q)
    c = []
    for b in a.bindings:
        rowvals=[]
        for k in a.vars:
            rowvals.append(b[k])
        c.append(rowvals)

    df = pd.DataFrame(c)
    df.columns = [str(v) for v in a.vars]
    if cast_to_numeric:
        df = df.apply(lambda x: pd.to_numeric(x, errors='ignore'))

    return df<jupyter_output><empty_output><jupyter_text>Tools to support the export and display of graphs - `networkx` package is handy in this respect, eg exporting to GEXF format for use with Gephi. We can also run projections on the graph quite easily.<jupyter_code>import networkx as nx<jupyter_output><empty_output><jupyter_text>Exploring the Data - Terms<jupyter_code>path='../data/dataexport/terms'
termgraph=ttl_graphbuilder(path)
#What's in the graph generally?
q='''
SELECT DISTINCT ?x ?y ?z {
    ?x ?y ?z.
} LIMIT 10
'''
rdfQuery(termgraph,q)
#What does a term have associated with it more specifically?
q='''
SELECT DISTINCT ?y ?z {
    <http://data.parliament.uk/terms/95551> ?y ?z.
} LIMIT 10
'''
rdfQuery(termgraph,q)<jupyter_output>http://www.w3.org/2004/02/skos/core#narrower http://data.parliament.uk/terms/95494 
http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://www.w3.org/2004/02/skos/core#Concept 
http://www.w3.org/2004/02/skos/core#prefLabel Defence policy 
http://www.w3.org/2004/02/skos/core#narrower http://data.parliament.uk/terms/95550 
http://www.w3.org/2004/02/skos/core#narrower http://data.parliament.uk/terms/95502 
http://www.w3.org/2004/02/skos/core#notation TPG 
http://www.w3.org/2004/02/skos/core#narrower http://data.parliament.uk/terms/95586 
http://www.w3.org/2004/02/skos/core#broader http://data.parliament.uk/terms/95548<jupyter_text>Looks like the `prefLabel` is what we want:<jupyter_code>q='''
SELECT DISTINCT ?z ?topic {
    ?z <http://www.w3.org/2004/02/skos/core#prefLabel> ?topic.
} LIMIT 10
'''
sparql2df(termgraph,q)<jupyter_output><empty_output><jupyter_text>Exploring the Data - EDMS<jupyter_code>path='../data/dataexport/edms'
g=ttl_graphbuilder(path)
#See what's there generally...
q='''
SELECT DISTINCT ?x ?y ?z {
    ?x ?y ?z.
} LIMIT 10
'''
rdfQuery(g,q)
#Explore a specific EDM
q='''
SELECT DISTINCT ?y ?z {
    <http://data.parliament.uk/edms/50457> ?y ?z.
}
'''
rdfQuery(g,q)<jupyter_output>http://data.parliament.uk/schema/parl#motionText That this House notes the announcement of 300 redundancies at the Nestlé manufacturing factories in York, Fawdon, Halifax and Girvan and that production of the Blue Riband bar will be transferred to Poland; acknowledges in the first three months of 2017 Nestlé achieved £21 billion in sales, a 0.4 per cent increase over the same period in 2016; further notes 156 of these job losses will be in York, a city that in the last six months has seen 2,000 job losses announced and has become the most inequitable city outside of the South East, and a further 110 jobs from Fawdon, Newcastle; recognises the losses come within a month of triggering Article 50, and as negotiations with the EU on the UK leaving the EU and the UK's future with the EU are commencing; further recognises the cost of importing products, including sugar, cocoa and production machinery, has risen due to the weakness of the pound and the uncertainty over the UK's future relatio[...]<jupyter_text>Let's merge the EDM graph data with the terms data.<jupyter_code>path='../data/dataexport/edms'
g=ttl_graphbuilder(path,termgraph)<jupyter_output><empty_output><jupyter_text>Now we can look at the term labels associated with a particular EDM.<jupyter_code>q='''
SELECT DISTINCT ?t ?z {
    <http://data.parliament.uk/edms/50114> <http://data.parliament.uk/schema/parl#topic> ?z.
    ?z <http://www.w3.org/2004/02/skos/core#prefLabel> ?t.
} LIMIT 10
'''
rdfQuery(g,q)<jupyter_output>Arms control http://data.parliament.uk/terms/95494 
International politics and government http://data.parliament.uk/terms/95650 
North America http://data.parliament.uk/terms/95690 
Defence policy http://data.parliament.uk/terms/95551<jupyter_text>We can also create a table that links topic labels with EDMs.<jupyter_code>q='''
SELECT DISTINCT ?edms ?topic {
    ?edms <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://data.parliament.uk/schema/parl#EarlyDayMotion>.
    ?edms <http://data.parliament.uk/schema/parl#topic> ?z.
    ?z <http://www.w3.org/2004/02/skos/core#prefLabel> ?topic.
}
'''
g_df=sparql2df(g,q)
g_df.head()<jupyter_output><empty_output><jupyter_text>From this table, we can a generate a bipartite `networkx` graph that links topic labels with EDMs.<jupyter_code>nxg=nx.from_pandas_dataframe(g_df, 'edms', 'topic')
#nx.write_gexf(nxg,'edms.gexf')<jupyter_output><empty_output><jupyter_text>We can then project this bipartite graph onto just the topic label nodes - edges will now connect nodes that are linked through one or more common EDMs.<jupyter_code>from networkx.algorithms import bipartite
#We can find the sets of names/tags associated with the disjoint sets in the graph
#I think the directedness of the graph means we can be reasonably sure the variable names are correctly ordered?
edms,topic=bipartite.sets(nxg)

#Collapse the bipartite graph to a graph of topic labels connected via a common EDM
topicgraph= bipartite.projected_graph(nxg, topic)
nx.write_gexf(topicgraph,'edms_topics.gexf')<jupyter_output><empty_output><jupyter_text>We can also generate a weighted graph, where edges are weighted relative to how many times topics are linked through different EDMs.<jupyter_code>topicgraph_weighted= bipartite.weighted_projected_graph(nxg, topic)
nx.write_gexf(topicgraph_weighted,'edms_topics_weighted.gexf')<jupyter_output><empty_output><jupyter_text>Predicting Topics<jupyter_code>#!pip3 install sklearn
#via https://stackoverflow.com/a/19172087/454773
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.multiclass import OneVsRestClassifier
from sklearn.preprocessing import MultiLabelBinarizer
#via https://stackoverflow.com/questions/22219004/grouping-rows-in-list-in-pandas-groupby
g_df['topic']=g_df['topic'].astype(str)
topicsbyedm_df=g_df.groupby('edms')['topic'].apply(list).to_frame().reset_index()
topicsbyedm_df.head()
q='''
SELECT DISTINCT ?edms ?motiontext {
    ?edms <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://data.parliament.uk/schema/parl#EarlyDayMotion>.
    ?edms <http://data.parliament.uk/schema/parl#motionText> ?motiontext.
}
'''
m_df=sparql2df(g,q)
m_df=m_df.merge(topicsbyedm_df,on='edms')
m_df.head()
X_train= np.array(m_df['motiontext'][:-100].tolist())
X_test = np.array(m_df['motiontext'][-100:].tolist())   
target_names=g_df['topic'].astype(str).tolist()
target_names[:3]
#ytrain= [[target_names.index(i) for i in t] for t in m_df['topic'][:-100] ]
#ytrain[:3]
y_train_text = [ t for t in m_df['topic'][:-100] ]
y_train_text[:3]
mlb = MultiLabelBinarizer()
Y = mlb.fit_transform(y_train_text)

classifier = Pipeline([
    ('vectorizer', CountVectorizer(analyzer='word',stop_words='english')),
    ('tfidf', TfidfTransformer()),
    ('clf', OneVsRestClassifier(LinearSVC()))])

classifier.fit(X_train, Y)
predicted = classifier.predict(X_test)
all_labels = mlb.inverse_transform(predicted)

hits=[]
misses=[]
for item, labels in zip(X_test, all_labels):
    if labels!=(): hits.append('{0} => {1}'.format(item, ', '.join(labels)))
    else: misses.append('{0} => {1}'.format(item, ', '.join(labels)))
        
print("some hits:\n{}\n\nsome misses:\n{}".format('\n'.join(hits[:3]),'\n'.join(misses[:3])))
labels<jupyter_output><empty_output><jupyter_text>Exploring the Data - proceedings<jupyter_code>path='../data/dataexport/proceedings'
p=ttl_graphbuilder(path,debug=True)
!ls {path}
!cat {path}/0006D323-D0B5-4E22-A26E-75ABB621F58E.ttl<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12363564199245883, 'fraction_numerical': 0.027485612224647748, 'mean_word_length': 5.119611414693382, 'pattern_counts': {'":': 0, '<': 74, '<?xml version=': 0, '>': 76, 'https://': 3, 'lorem ipsum': 0, 'www.': 14, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '288914', 'license': 'MIT', 'n_tokens_mistral': 3740, 'n_tokens_neox': 3456, 'n_words': 924, 'path': 'notebooks/Co-Occurring Tag Analysis.ipynb', 'repo_name': 'psychemedia/parlihacks', 'size': '10078'}