<jupyter_start><jupyter_text>data['semantic_similarity'] = data.apply(lambda row: get_sim(row['abstract'], row['review']), axis=1)print(len(data[data['semantic_similarity'] == 999999999]))data.tail()<jupyter_code>from spacy.lang.en import English
import re
import spacy

def data_preprocessing(text):
    
    nlp = English()
    
    tokenizer = nlp.Defaults.create_tokenizer(nlp)
    text = tokenizer(text)
    
    spacy_stopwords = spacy.lang.en.stop_words.STOP_WORDS
    prep = []
    
    for token in text:
        token = token.lemma_
        if token.lower() not in spacy_stopwords:
            token = re.sub('<[^>]*>', '', token)
            token = re.sub('[\W]+', '', token.lower())
            prep.append(token)

                
    return ' '.join(prep)
    
        
data['abstract_clean'] = data.apply(lambda row: data_preprocessing(row['abstract']), axis=1)
data['review_clean'] = data.apply(lambda row: data_preprocessing(row['review']), axis=1)
data.tail()
# conda install -c conda-forge spacy
data['abstract_vectors'], data['review_vectors'], data['semantic_similarity_aftercleaning'] = data.apply(lambda row: get_sim(row['abstract_clean'], row['review_clean']), axis=1)
data.tail()
len(data[data['semantic_similarity_aftercleaning'] == 999999999])
data.to_csv('scores_vectors.csv', index = False)
a = """
 authors propose transform code solution extend work balle 2016  define hyperprior entropy coder model
 spatial relation transform coefficient   paper good write  trouble follow  result proposal state
 art  extremely exhaustive comparison method   opinion work good quality present iclr   think excellent improve
 detail think improve      main issues  main concern motivation relate  1 refer hyperprior motivation  clear
 gdn propose eliminate statistical dependency pixel image  main motivation gdn coefficient independent
 confusion resolve broaden explanation figure 2  2 concern clear modify probability distribution entropy
 encoder improve gdn model  think interest issue  outside scope work  far know  theoretical
 solution find right balance complexity transformation entropy encoder   interest discuss main novelty work
 compare method image compression base deep learn      issues  introduction    model optimize end   end
 minimize total expect code length learn balance information expect improvement entropy model    think point
 interest  good number happen result present  train procedure  example  simple comparison numb bit signal
 information depend compression rate numb iteration model train   compression variational models   miss 
 sentence    arithmetic code   transmit     fig1  clear read leave hand scheme  possible include distribution
 specifically  strange tiledy  scheme different conditional dependency  thing symbol  
 appear figure use section 2    easy follow change symbol function parameter like theta theta     distortion
 expect difference    expect  word use      substitute additive uniform noise   phrase correct  author balle
 2016 substitute additive uniform noise    equation  1   1 term zero constant  talk equation  7  author
 1 term constant       sentence  previous work assume   sound strange    example fig  2 extremely important
 understand motivation hyperprior think need little explanation  example important need explain begin work
 real example  model train normalization  specify  gdn able eliminate spatial dependency  dependency eliminate
 normalization apply spatial coefficient  remove dependency layer different parameter gdn   introduction
 scale hyperprior   typo   center pane       propose follow extension model  figure 3    colon  maybe miss
 maybe dot instead colon  lack explanation model   results      probability mass function  need construct
 fly     computationally costly      batch normalization learn rate decay find beneficial effect  local
 normalization property gdn  contain global normalization special case     extremely interest  connection batch 
 normalization  decay learn rate   clarify  mean use gdn instead regular nonlinearity long need use batch 
 normalization  word  think batch normalization useful special case gsn  useful community assess benefit 
 local normalization versus global normalization      combination 8 different value  order cover range rate  
 distortion tradeoff    possible method include lambda input model parameter information    guess include 
 information compute total entropy  numb bit   different way compress image information    metric train evaluate 
 little bite mislead  evaluation plot use different perceptual metric helpful   since ms  ssim yield value
 0  wrong  1    compare method achieve value good 09  convert quantity decibel order improve legibility    
 difference ms  ssim conversion significant  transformation necessary  lose intuition   probably fault able  
 unconvert  db ms  ssim unit  instance  20 curve surpass value       
 result differ substantially depend distortion metric use loss function train    informative understand 
 parameter change depend metric employ  little intuition set parameter adapt     figs 5  8 9  curve aggregate 
 different image  mean rate value  note depend totally mislead    nice include result method  like bpg rippel 
 2017  compare visually   related work  balle et al  publish work include perceptual metric end   
 end train procedure  think main contribution work  include relate work    end   end optimization 
 nonlinear transform code perceptual quality    laparra  ep simoncelli  pcs  picture coding symposium   
 2016   discussion  paragraph discussion section look like 2 section  relate work    think interest author 
 discuss relevance putt effort model hyperprior distribution image  transformation   thing equivalent  
 reason include hyperprior model ga transformation  clear model distribution output  principle   transformation
 enforce  use train procedure  transform datum follow impose distribution  gdn powerful output independent  
 beneficial compression divide problem   references   balle 2016 theis 2017 publish conference year  different 
 year reference confuse    strange reference    j  v laparra  e p simoncelli  2016    density modeling 
 images use generalized  normalization transformation    intl  conf  learning representations  iclr2016   
 url   httpsarxivorgabs151106281  valero laparra  eero p simoncelli  2015    density modeling images gen  
 eralized normalization transformation    arxiv e  print  published conference paper  4th international 
 conference learning representations  san juan  2016  arxiv  1511   06281     2016    end   end optimized 
 image compression    arxiv e  print  5th int  conf  learn  ing representations  
"""
b = """
describe end   end trainable model image compression base variational autoencoders  model incorporate 
hyperprior effectively capture spatial dependency latent representation  hyperprior relate information  concept 
universal virtually modern image codecs  largely unexplore image compression use artificial neural network  
anns   unlike exist autoencoder compression method  model train complex prior jointly underlie autoencoder  
demonstrate model lead state    art image compression measure visual quality use popular ms  ssim index  yield 
rate  distortion performance surpass publish ann  base method evaluate use traditional metric base square error  
psnr   furthermore  provide qualitative comparison model train different distortion metric
"""
get_similarity(a, b)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.03064215294431122, 'fraction_numerical': 0.014388489208633094, 'mean_word_length': 4.173673328738801, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '203936', 'license': 'MIT', 'n_tokens_mistral': 1772, 'n_tokens_neox': 1613, 'n_words': 830, 'path': 'Sci-BERT semantic similarity.ipynb', 'repo_name': 'harishsiravuri/SciSci', 'size': '7506'}