<jupyter_start><jupyter_text>About - Coref Resolution of articles: Section8 2080-7567; Row 838-977- Output File: [ReferenceChainReplace_ID_2080_7567.csv](https://drive.google.com/file/d/1fM-1zF6hF-TN7j882iNtwC8yZ0XdmCD1/view?usp=sharing)- File Columns:   - ids	  - month	  - class	  - title	  - text	  - text_coref  Installing All dependencies- spacy 2.1.0 (later versions have some issues with neuralcoref)- neuralcoref - news-please<jupyter_code># uninstall default version & install 2.1.0
!pip uninstall spacy 
!pip install spacy==2.1.0

# download pretrained english model
!python -m spacy download en

# install neuralcoref
!pip install neuralcoref

# pickled articles need this
!pip install news-please<jupyter_output><empty_output><jupyter_text>Loading Dependencies<jupyter_code>import pandas as pd
import pickle
import spacy
import neuralcoref
from newsplease import NewsPlease
from google.colab import drive
# mount google drive
drive.mount('/content/drive')<jupyter_output>Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).<jupyter_text>Clone WRI Github Repo to Google Drive [Skip if already did]Skip this if you already have WRI repo and the new manually_annotated.csv file<jupyter_code># dir where WRI repo will be cloned
!mkdir '/content/drive/My Drive/WRI'
!git clone 'https://github.com/omdena/WRI.git' '/content/drive/My Drive/WRI'<jupyter_output><empty_output><jupyter_text>Unzip Article Files [Skip if already did]I'm extracting files to WRI_DATA folder in my Google Drive. It might take 10-15 minutes<jupyter_code>!mkdir '/content/drive/My Drive/WRI_DATA'
!unzip -qq '/content/drive/My Drive/WRI/data/texts/*.zip' -d '/content/drive/My Drive/WRI_DATA'<jupyter_output><empty_output><jupyter_text>Loading Text Data from Google Drive<jupyter_code># path where manual text data is located
file_path = '/content/drive/My Drive/WRI/data/gold-standard/manually_annotated.csv'
gold_df  = pd.read_csv(file_path)
gold_df.head()
# starting row #838
gold_df[gold_df.ids == 2080 ]
# end row #977
gold_df[gold_df.ids == 7567 ]
# change it to rows assigned to you
begin_row, end_row = 838, 977<jupyter_output><empty_output><jupyter_text>Util Functions<jupyter_code># util function to load articles for given month & ids
# taken from the WRI repo 
def load_obj(month, idx):
    month = str(month).zfill(2)
    idx = str(idx).zfill(5)
    with open("/content/drive/My Drive/WRI_DATA/{}/{}.pkl".format(month, idx), "rb") as f:
        return pickle.load(f)<jupyter_output><empty_output><jupyter_text>Coref Resolution<jupyter_code>output_df = pd.DataFrame() # DataFrame to store resulted text
nlp = spacy.load('en')
neuralcoref.add_to_pipe(nlp)
print(f'starting at {begin_row}th row....')
for row in range(begin_row, end_row+1):
  article = load_obj(gold_df['month'][row], gold_df['ids'][row])
  doc = nlp(article.text)
  temp_df = pd.DataFrame([
      {'ids': gold_df['ids'][row], 
       'month': gold_df['month'][row], 
       'class':gold_df['class'][row], 
       'title': gold_df['title'][row], 
       'text':article.text, 
       'text_coref':doc._.coref_resolved}])
  output_df = pd.concat([output_df, temp_df])
  
  if row%5==0:
    print(f'Completed Upto {row}th row...')
print(f'Done')
output_df.head()<jupyter_output><empty_output><jupyter_text>Example: For Positive class<jupyter_code># a random article for comparison
p_ex = output_df.loc[output_df['class'] =='positive'].sample(n=1, random_state=1)
p_ex.title
print(str(p_ex['text'][0]))
print(str(p_ex['text_coref'][0]))<jupyter_output>File photo
GURUGRAM: PWD minister <NAME> on Sunday announced that work on the stalled Greater Southern Peripheral Road (G-SPR) project, to connect Delhi-Gurugram border to IMT Manesar, should gain pace, now that Punjab and Haryana high court has vacated a stay order on land acquisition that Punjab and Haryana high court had imposed in 2016.Huda had started acquisition for the 40km-long GSPR in 2016, but faced a challenge from a land owner. Some villagers had filed a case before Punjab and Haryana high court, challenging the acquisition in Naurangpur village and nearby areas, and got a stay from court.With a stay from court now vacated, the state will have to issue a fresh notification for acquisition, but Huda is raring to start Punjab and Haryana high court has vacated the stay over acquisition by dismissing the case filed by land owners, paving the way for construction of G-SPR as per the Master Plan,” said the PWD minister, adding that once complete, this road would offer [...]<jupyter_text>Original Text - 🍎Incorrect Replacement- 🍏Correct ReplacementGURUGRAM: PWD minister <NAME> on Sunday announced that work on the stalled Greater Southern Peripheral Road (G-SPR) project, to connect Delhi-Gurugram border to IMT Manesar, should gain pace, now that Punjab and Haryana high court has vacated a stay order on land acquisition that   🍏it  🍏 had imposed in 2016.Huda had started acquisition for the 40km-long GSPR in 2016, but faced a challenge from a land owner. Some villagers had filed a case before the high court, challenging the acquisition in Naurangpur village and nearby areas, and got a stay from court.With the stay now vacated, the state will have to issue a fresh notification for acquisition, but Huda is raring to start the process.“The high court has vacated the stay over acquisition by dismissing the case filed by land owners, paving the way for construction of G-SPR as per the Master Plan,” said the PWD minister, adding that once complete, this road would offer an alternative route between Delhi-Gurugram border and IMT Manesar, which are connected only through Delhi-Gurugram expressway (NH-8) at present, and thus reduce congestion on the NH-8.The minister added the road will branch out from NH-8 at IMT Manesar, pass through Badshapur as  🍎 it  🍎 proceeds towards Gurugram-Faridabad road, cross MG Road and again merge with NH-8 near Sirhaul on the Delhi-Gurugram border from behind Ambience mall. The road is supposed to be constructed as per Gurugram-Manesar Plan 2031.“This road will be more like an ‘outer ring road’ for the city,”   🍏 he 🍏 said, adding  they are trying to convince NHAI to build  🍎it 🍎, for which,   🍏he  🍏 has already met Union minister <NAME> . The road is expected to benefit villages along the road, while also reducing congestion on NH-8, and thereby pollution in the city.As per sources, NHAI is not keen on building the road, due to high cost of land acquisition. “The initial survey had indicated the project was not viable due to high cost of land acquisition. That’s why a fresh feasibility study is now being carried out, in view of Haryana’s new policy, announced in late 2017, of transit oriented development (TOD), under which land along the road will get additional FAR,” said the source.Around 395 acres has to be acquired in eight villages for G-SPR.  They are — Aklimpur (5.95 acre), Tekliki (62.95 acre), Sakatpur (68.90 acre), Sikohour (15.92 acre), Naurangpur (49.40 acre), <NAME> (99.13 acre), Nainwal (59.92 acre) and Manesar (33.15 acre). The road will be 90m-wide with a 30m-green belt on either side. The land will be acquired by Huda and handed over to the developer — which could be NHAI — for construction.  Text After Coref ResolutionGURUGRAM: PWD minister <NAME> on Sunday announced that work on the stalled Greater Southern Peripheral Road (G-SPR) project, to connect Delhi-Gurugram border to IMT Manesar, should gain pace, now that Punjab and Haryana high court has vacated a stay order on land acquisition that   🍏Punjab and Haryana high court  🍏 had imposed in 2016.Huda had started acquisition for the 40km-long GSPR in 2016, but faced a challenge from a land owner. Some villagers had filed a case before Punjab and Haryana high court, challenging the acquisition in Naurangpur village and nearby areas, and got a stay from court.With a stay from court now vacated, the state will have to issue a fresh notification for acquisition, but Huda is raring to start Punjab and Haryana high court has vacated the stay over acquisition by dismissing the case filed by land owners, paving the way for construction of G-SPR as per the Master Plan,” said the PWD minister, adding that once complete, this road would offer an alternative route between Delhi-Gurugram border and IMT Manesar, which are connected only through Delhi-Gurugram expressway (NH-8) at present, and thus reduce congestion on the NH-8.The minister added this road will branch out from NH-8 at IMT Manesar, pass through Badshapur as  🍎Badshapur 🍎 proceeds towards Gurugram-Faridabad road, cross MG Road and again merge with NH-8 near Sirhaul on the Delhi-Gurugram border from behind Ambience mall. this road is supposed to be constructed as per Gurugram-Manesar Plan 2031.“This road will be more like an ‘outer ring road’ for the city,”   🍏the PWD minister  🍏 said, adding they are trying to convince NHAI to build  🍎NHAI 🍎, for which,   🍏the PWD minister  🍏 has already met Union minister <NAME> . this road is expected to benefit villages along this road, while also reducing congestion on NH-8, and thereby pollution in the city.As per sources, NHAI is not keen on building this road, due to high cost of land acquisition. “The initial survey had indicated the project was not viable due to high cost of land acquisition. That’s why a fresh feasibility study is now being carried out, in view of Haryana’s new policy, announced in late 2017, of transit oriented development (TOD), under which land along this road will get additional FAR,” said the source.Around 395 acres has to be acquired in eight villages for G-SPR. They are — Aklimpur (5.95 acre), Tekliki (62.95 acre), Sakatpur (68.90 acre), Sikohour (15.92 acre), Naurangpur (49.40 acre), <NAME> (99.13 acre), Nainwal (59.92 acre) and Manesar (33.15 acre). this road will be 90m-wide with a 30m-green belt on either side. The land will be acquired by Huda and handed over to the developer — which could be NHAI — for construction.  Saving Coref Resolution DataFrame<jupyter_code>output_df.to_csv('/content/drive/My Drive/WRI_DATA/ReferenceChainReplace_ID_2080_7567.csv')
# if saved file takes time to show up in drive then flush data
drive.flush_and_unmount()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07498785818358426, 'fraction_numerical': 0.020883924235065566, 'mean_word_length': 4.830124575311438, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '367416', 'license': 'MIT', 'n_tokens_mistral': 3308, 'n_tokens_neox': 3018, 'n_words': 1375, 'path': 'notebooks/Task_3_Coreference_Resolution_Unsupervised.ipynb', 'repo_name': 'kathleenbuckingham/WRI', 'size': '10347'}