<jupyter_start><jupyter_text>Extract relevant sentencesIn this notebook a method is constructed to extract relevant sentences from a given text and the corresponding abstract. For this method, a score of similarity between sentences would be useful. The Jaccard index is used on top of a BOW (Bag-of-Words) model of a sentence. The Jaccard index is defined as follows on two sets $A$ and $B$:$J(A, B) = \frac{|A \cap B|}{|A \cup B|}$It is a score between $0$ and $1$ and is $1$ if all elements of $A$ are equal to all elements of $B$. The score equals $0$ if every element of $A$ differs from any element of $B$. Therefore, it is a measure for overlap.It is important to keep the sentence comparison as fast as possible, since it is used on every training document.<jupyter_code>import sys
sys.path.append('..')

import nltk
import unidecode
import re
from utils import compute_jaccard_index
import numpy as np<jupyter_output><empty_output><jupyter_text>In the next cell, a few examples are defined. One of the examples is a sentence in a different order and one of the examples is completely different from the original.<jupyter_code>examples = [
    ('This is a test sentence.', 'This sentence is a test.'),
    ('This is a Jupyter Notebook.', 'All dogs are animals.')
]<jupyter_output><empty_output><jupyter_text>The overlap scores for each of the examples are computed in the next cell. The `clean_word` method removes accents and non-alphabetic characters. By doing so, punctiation symbols are removed from the Bag-of-Words which is desired. Otherwise, most of the scores are non-zero for completely different sentences.<jupyter_code>clean_word = lambda word: re.sub(r'[^a-z]+', '', unidecode.unidecode(word).lower())

for sentence1, sentence2 in examples:
    print('Comparing "%s" to "%s":' % (sentence1, sentence2))
    words1 = {clean_word(word) for word in nltk.word_tokenize(sentence1) if len(clean_word(word)) > 0}
    words2 = {clean_word(word) for word in nltk.word_tokenize(sentence2) if len(clean_word(word)) > 0}
    jaccard_index = compute_jaccard_index(words1, words2)
    print('- Unique words in first sentence: %s' % words1)
    print('- Unique words in second sentence: %s' % words2)
    print('- Jaccard index: %.2f' % jaccard_index)
    print()<jupyter_output>Comparing "This is a test sentence." to "This sentence is a test.":
- Unique words in first sentence: {'this', 'test', 'is', 'sentence', 'a'}
- Unique words in second sentence: {'this', 'test', 'is', 'sentence', 'a'}
- Jaccard index: 1.00

Comparing "This is a Jupyter Notebook." to "All dogs are animals.":
- Unique words in first sentence: {'this', 'jupyter', 'is', 'notebook', 'a'}
- Unique words in second sentence: {'dogs', 'animals', 'all', 'are'}
- Jaccard index: 0.00<jupyter_text>Now this approach is tested on a real-world example consisting of both an abstract and the full text.<jupyter_code>text = """The domestic dog (Canis lupus familiaris or Canis familiaris) is a member of the genus Canis (canines), which forms part of the wolf-like canids, and is the most widely abundant terrestrial carnivore. The dog and the extant gray wolf are sister taxa as modern wolves are not closely related to the wolves that were first domesticated, which implies that the direct ancestor of the dog is extinct. The dog was the first species to be domesticated and has been selectively bred over millennia for various behaviors, sensory capabilities, and physical attributes. Their long association with humans has led dogs to be uniquely attuned to human behavior and they are able to thrive on a starch-rich diet that would be inadequate for other canid species. New research seems to show that dogs have mutations to equivalent genetic regions in humans where changes are known to trigger high sociability and somewhat reduced intelligence. Dogs vary widely in shape, size and colors. Dogs perform many roles for people, such as hunting, herding, pulling loads, protection, assisting police and military, companionship and, more recently, aiding handicapped individuals and therapeutic roles. This influence on human society has given them the sobriquet "man's best friend"."""
abstract = """The dog was the first species to be domesticated. Dogs vary widely in shape, size and colors."""

print('Text:')
print('-' * 80)
print(text)
print('-' * 80)
print()
print('Abstract:')
print('-' * 80)
print(abstract)
print('-' * 80)
for sentence1 in nltk.sent_tokenize(abstract):
    scores = []
    text_sentences = nltk.sent_tokenize(text)
    for sentence2 in text_sentences:
        words1 = {clean_word(word) for word in nltk.word_tokenize(sentence1) if len(clean_word(word)) > 0}
        words2 = {clean_word(word) for word in nltk.word_tokenize(sentence2) if len(clean_word(word)) > 0}
        scores.append(compute_jaccard_index(words1, words2))
    related_indices = set()
    if np.max(scores) > 0.:
        related_indices.add(np.argmax(scores))
    for index, score in enumerate(scores):
        if score > 0.6:
            related_indices.add(index)
    print('Sentence in abstract: "%s"' % sentence1)
    print('-' * 80)
    for index in related_indices:
        print('Related sentence in text: "%s"' % text_sentences[index])
    print('-' * 80)
    print()<jupyter_output>Sentence in abstract: "The dog was the first species to be domesticated."
--------------------------------------------------------------------------------
Related sentence in text: "The dog was the first species to be domesticated and has been selectively bred over millennia for various behaviors, sensory capabilities, and physical attributes."
--------------------------------------------------------------------------------

Sentence in abstract: "Dogs vary widely in shape, size and colors."
--------------------------------------------------------------------------------
Related sentence in text: "Dogs vary widely in shape, size and colors."
--------------------------------------------------------------------------------<jupyter_text>There is not a straightforward algorithm to compute relatedness between two sentences. But this is a greedy approach which is required for fast preprocessing. It might introduce some amount of noise. Now lets test it on a different example.<jupyter_code>text = """While architects have publicly proclaimed the World Trade Center site proposals displayed at the Winter Garden in Lower Manhattan as the greatest architecture show ever, many have privately expressed reservations about the designs' details, the handling of the competition and even the spotlight in which the contestants now stand.\n\n        ''Architecture is finally having a visible presence, perhaps too visible,'' said <NAME> of Diller & Scofidio in Manhattan.\n\n      \n\n      \n\n        Many architects fear that the attention and intensity given to the World Trade Center designs will greatly affect the project.\n\n      \n\n      \n\n        While architects have publicly proclaimed the World Trade Center site proposals displayed at the Winter Garden in Lower Manhattan as the greatest architecture show ever, many have privately expressed reservations about the designs' details, the handling of the competition and even the spotlight in which the contestants now stand.\n\n        ''Architecture is finally having a visible presence, perhaps too visible,'' said <NAME> of Diller & Scofidio in Manhattan.\n\n        The popular image of the architect as a creative genius whipping up great designs on a cocktail napkin is at odds with the reality. More often, architects say, great design is the result of constant, sometimes painful give-and-take between the architect and the client. Letting the public in on the process from the start, even as spectators, has pulled back the veil on a ritual that is most often conducted in the hush of boardrooms and private offices.\n\n        By contrast, the Lower Manhattan Development Corporation announced that its design priorities for the site would be determined ''by conducting the most comprehensive public outreach campaign ever undertaken.'' The power of public opinion to sway the process was amply demonstrated in July when six initial site plans were universally rejected. In this, the second round, the public has been treated to front-row seats: the presentations by the seven competing architectural teams were televised live for more than three hours, and an exhibition of their models, renderings and video walk-throughs was open to the public almost immediately. Several architectural institutions have stepped in quickly to arrange their own forums, discussion groups and exhibitions on the process, and television networks have devoted unusual amounts of air time to explaining site plans and computer-animated design.\n\n        Architects ''presenting on TV has never happened before,'' Mr. Scofidio added. ''But at this phase, letting the public say what it likes and doesn't like will only make the water muddier,'' he said, explaining that what may be a great spectacle was no way to select a design.\n\n        <NAME>, a design consultant and adviser to the jury on architecture's highest honor, the Pritzker Prize, said that the Lower Manhattan redevelopment was ''far too important to be judged by public opinion poll.''\n\n        ''I feel sorry for these architects designing in a fish bowl,'' he continued. ''The first team did a credible job but was crucified by being exposed to the public prematurely. People are so eager for something positive to happen, but land use and massing studies are never exciting. You can't design for seven million clients.''\n\n        Mindful of the effort involved in preparing such complex and historically significant designs in just eight weeks (and with fees of only $40,000), the 16 architects interviewed for this article were loath to single out any team's design. But they did not hesitate to criticize the process as too exposed and the requirements as too vague.\n\n        The attention and its intensity are mixed blessings, said some architects, who worried that some of the more implausible designs might be taken literally, leaving the entire profession open to ridicule and condemnation. ''There is something a little grotesque in the interpretation of ground zero as a lucky break for art,'' <NAME>, literary editor of The New Republic, said last September in a debate with <NAME>, one of the competing architects, at Columbia University.\n\n        The development corporation has frequently said that the object of the competition, a master land-use plan, is not to ''include the detailed architecture of individual structures.'' But many architects worry that the teams' detailed models and impressively realistic video presentations will encourage the public to perceive them as concrete plans.\n\n        <NAME>, a semifinalist in the competition and the dean of the Columbia Graduate School of Architecture, Planning and Preservation, described the process as backward. ''They are starting with a design and hope to arrive at a program,'' he said. ''It strikes me as unusual. And since each design is based on its own premises, you really can't compare them to each other at all. The ambiguity is not right.''\n\n        While some architects championed the competition as a way to educate the public about the importance of architecture, many faulted the proposals for the way the buildings met the ground and integrated with the city.\n\n        ''There should be more talk about activities, not buildings,'' said the architect Denise <NAME> of Venturi, <NAME> & Associates in Philadelphia. ''A great deal of money will be spent quickly on the transit system, and that will affect what else happens. All those people coming up out of the subway will surely affect the design.'' She said she wasn't sure that factor was reflected in the proposals, ''while, in fact, it should be the generator of these designs.''\n\n        Other architects said too much creative vision was expended on towers and not enough on street-level elements.\n\n        ''The ground plan and infrastructure are surprisingly conservative in contrast to the boldness of the architecture,'' said <NAME>, a Princeton, N.J., architect and former dean of the Princeton University School of Architecture, who is now working on the design of several areas adjacent to the World Trade Center site. ''There weren't many new thoughts on how to treat ground transportation.''\n\n        Many architects, however, commended the building proposals for incorporating the latest innovations in energy efficiency. ''This will be the first time that European daring in ecological issues has been introduced at such a scale in the U.S.,'' said Raymond <NAME>, executive director of the Van Alen Institute, a nonprofit organization devoted to increasing awareness of public architecture, ''but it will create new standards for all skyscrapers.''\n\n        The Van Alen Institute recently published a report, ''Information Exchange: How Cities Renew, Rebuild and Remember,'' exploring how seven cities, including Beirut, Sarajevo and Berlin, rebuilt themselves in the wake of both natural and political disasters.\n\n        As for building height, architects' opinions varied about what was appropriate for structures that would stand not in, but next to, the footsteps of the lanky twin towers.\n\n        ''I'm offended by everyone reaching to the sky again,'' said <NAME>, an architect in Phoenix who focuses on environmental and recycling issues. Of the tall designs, he found <NAME>'s 1,776-foot tapering tower the most convincing. ''At least he reached up to the sky with sculpture instead of a bulky mass,'' Mr. Bruder said.\n\n        Did any of the competitors succeed at reinventing the skyscraper for a new era? Only if you've never seen Hong Kong, Mr. Lerner said. United Architects' tall, angled structures, which combined into a single public floor high in the sky, were the only proposals suggesting a new way of thinking about large buildings in groups, he added.\n\n        <NAME> of <NAME> in Manhattan, who did not participate in the competition, said he was not convinced that a new kind of skyscraper was possible at this time. The circumstances that created landmarks like the Chrysler and Empire State buildings were different, he said. ''Not in our lifetime has anyone been able to figure out what New York should be,'' Mr. Hardy explained. ''We're all out of practice, and there's no powerful leadership. Without someone in charge, it's all going to have to be worked out each step of the way.''\n\n        All the architects wondered how the development corporation would proceed. The interested public, already well informed on the issues, has still more opportunities to learn. On Monday the Architectural League will open an exhibition that is like a continuing public tutorial. It will display a range of documents connected to the design proposals, from the architects' video presentations to the reactions of the European news media.\n\n        The exhibition is intended to be ''an archive of the process,'' said <NAME>, the league's executive director, and it will be updated as more materials become available. ''The first round was so bland, there was nothing to talk about,'' she said. ''Now there's so much more to look at and to sort out. And there's more emotion.'' The exhibition will run through the end of February, when, the development corporation announced, it will adopt a final master land-use plan and undertake a competition for a ground zero memorial.\n\n        On Tuesday Architectural Record magazine is sponsoring a forum of architects and architectural critics, including Mr. Tschumi and <NAME>, the former chief executive of the Battery Park City Authority, who oversaw the creation of the master plan for Battery Park City in the 1980's. All the architects in the competition have been invited, along with representatives of the development corporation and Port Authority.\n\n        ''It's an intellectual exercise,'' said <NAME>, the editor in chief of Architectural Record. ''Have there ever been so many wonderful ideas to discuss, such depth of feeling to explore? My great fear is that they are trying to make a camel with three humps.''\n\n        But fears and criticism pale beside the excitement that most architects said they felt at the opportunity to see so much world-class architecture on display. ''This is a fantastic show of talent,'' said <NAME>, the architect of the World Financial Center and the Winter Garden, who estimated that the architects involved must have spent as much as $4 million on their combined presentations.\n\n        ''The community is getting a huge gift from these architects,'' Mr. Pelli said, adding, ''Of course, the architects are also getting phenomenal P.R.''""".replace('\n', ' ')
abstract = """Architects privately note difficulties resulting from power of public opinion in choosing design for World Trade Center site; note unheard-of live TV broadcast presenting six initial site plans, which resulted in rejection of all designs; interviews reveal variety of opinions among architects on unusual selection process; photo (M)"""

text = re.sub(r'[ ]+', ' ', text)
print('Text:')
print('-' * 80)
print(text)
print('-' * 80)
print()
print('Abstract:')
print('-' * 80)
print(abstract)
print('-' * 80)
for sentence1 in nltk.sent_tokenize(abstract):
    scores = []
    text_sentences = nltk.sent_tokenize(text)
    for sentence2 in text_sentences:
        words1 = {clean_word(word) for word in nltk.word_tokenize(sentence1) if len(clean_word(word)) > 0}
        words2 = {clean_word(word) for word in nltk.word_tokenize(sentence2) if len(clean_word(word)) > 0}
        scores.append(compute_jaccard_index(words1, words2))
    related_indices = set()
    if np.max(scores) > 0.:
        related_indices.add(np.argmax(scores))
    for index, score in enumerate(scores):
        if score > 0.6:
            related_indices.add(index)
    print('Sentence in abstract: "%s"' % sentence1)
    print('-' * 80)
    for index in related_indices:
        print('Related sentence in text: "%s"' % text_sentences[index])
    print('-' * 80)
    print()<jupyter_output>Sentence in abstract: "Architects privately note difficulties resulting from power of public opinion in choosing design for World Trade Center site; note unheard-of live TV broadcast presenting six initial site plans, which resulted in rejection of all designs; interviews reveal variety of opinions among architects on unusual selection process; photo (M)"
--------------------------------------------------------------------------------
Related sentence in text: "The power of public opinion to sway the process was amply demonstrated in July when six initial site plans were universally rejected."
--------------------------------------------------------------------------------<jupyter_text>It is interesting to see to what amount the text is reduced. A large reduction results in a great speed-up of the processing.<jupyter_code>related_sentences = []

for sentence1 in nltk.sent_tokenize(abstract):
    scores = []
    text_sentences = nltk.sent_tokenize(text)
    for sentence2 in text_sentences:
        words1 = {clean_word(word) for word in nltk.word_tokenize(sentence1) if len(clean_word(word)) > 0}
        words2 = {clean_word(word) for word in nltk.word_tokenize(sentence2) if len(clean_word(word)) > 0}
        scores.append(compute_jaccard_index(words1, words2))
    related_indices = set()
    if np.max(scores) > 0.:
        related_indices.add(np.argmax(scores))
    for index, score in enumerate(scores):
        if score > 0.6:
            related_indices.add(index)
    for index in related_indices:
        related_sentences.append(text_sentences[index])
        
constructed_abstract = ' '.join(related_sentences)
print('Original size:     %d characters' % len(text))
print('New size:          %d characters' % len(constructed_abstract))
print('Compression ratio: %.2f%%' % (-100. * (len(constructed_abstract) - len(text)) / float(len(text))))<jupyter_output>Original size:     10340 characters
New size:          133 characters
Compression ratio: 98.71%
{'fraction_non_alphanumeric': 0.09045052173483013, 'fraction_numerical': 0.006033331684882053, 'mean_word_length': 3.8459142104001915, 'pattern_counts': {'":': 3, '<': 39, '<?xml version=': 0, '>': 53, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '25528', 'license': 'MIT', 'n_tokens_mistral': 5059, 'n_tokens_neox': 4748, 'n_words': 2708, 'path': 'master/salience-detection-master/salience-detection-master/notebook/1_extract_relevant_sentences.ipynb', 'repo_name': 'AlexRogalskiy/DevArtifacts', 'size': '20349'}