<jupyter_start><jupyter_text>Chapter 2: Large-scale data analysis with spaCyIn this chapter, you'll use your new skills to extract specific information from large volumes of text. You''ll learn how to make the most of spaCy's data structures, and how to effectively combine statistical and rule-based approaches for text analysis.<jupyter_code>import spacy
# Import the Doc class
from spacy.tokens import Doc, Span
from spacy.lang.en import English

from spacy.matcher import Matcher, PhraseMatcher
nlp = spacy.load("en_core_web_sm")<jupyter_output><empty_output><jupyter_text>**String to hashesh**- Look up the string “cat” in nlp.vocab.strings to get the hash.- Look up the hash to get back the string.<jupyter_code>doc = nlp("I have a cat")

# Look up the hash for the word "cat"
cat_hash = nlp.vocab.strings["cat"]
print(cat_hash)

# Look up the cat_hash to get the string
cat_string = nlp.vocab.strings[cat_hash]
print(cat_string)<jupyter_output>5439657043933447811
cat<jupyter_text>- Look up the string label “PERSON” in nlp.vocab.strings to get the hash.- Look up the hash to get back the string.<jupyter_code>doc = nlp("<NAME> is a PERSON")

# Look up the hash for the string label "PERSON"
person_hash = nlp.vocab.strings["PERSON"]
print(person_hash)

# Look up the person_hash to get the string
person_string = nlp.vocab.strings[person_hash]
print(person_string)<jupyter_output>380
PERSON<jupyter_text>**Creating a Doc from scratch**<jupyter_code># Desired text: "spaCy is cool!"
words = ["spaCy", "is", "cool", "!"]
spaces = [True, True, False, False]

# Create a Doc from the words and spaces
doc = Doc(nlp.vocab, words=words, spaces=spaces)
print(doc.text)<jupyter_output>spaCy is cool!<jupyter_text>Create a sentence: "Go, get started!"<jupyter_code># Desired text: "Go, get started!"
words = ["Go", ",", "get", "started", "!"]
spaces = [False, True, True, False, False]
doc = Doc(nlp.vocab, words=words, spaces=spaces)
print(doc.text)<jupyter_output>Go, get started!<jupyter_text>Create a sentence: "Oh, really?!"<jupyter_code># Desired text: "Oh, really?!"
words = ["Oh", ",", "really", "?", "!"]
spaces = [False, True, False, False, False]

# Create a Doc from the words and spaces
doc = Doc(nlp.vocab, words = words, spaces = spaces)
print(doc.text)<jupyter_output>Oh, really?!<jupyter_text>**Doc, Span, Entities from Scratch** - In this exercise, you’ll create the Doc and Span objects manually, and update the named entities – just like spaCy does behind the scenes. A shared nlp object has already been created.<jupyter_code>nlp_en = English()

# Import the Doc and Span classes
from spacy.tokens import Doc, Span

words = ["I", "like", "David", "Bowie"]
spaces = [True, True, True, False]

# Create a doc from the words and spaces
doc = Doc(nlp_en.vocab, words=words, spaces=spaces)
print(doc.text)

# Create a span for "<NAME>" from the doc and assign it the label "PERSON"
span = Span(doc, 2, 4, label="PERSON")
print(span.text, span.label_)

# Add the span to the doc's entities
doc.ents = [span]

# Print entities' text and labels
print([(ent.text, ent.label_) for ent in doc.ents])<jupyter_output>I like <NAME>
<NAME> PERSON
[('<NAME>', 'PERSON')]<jupyter_text>**Data Structure Best Practices**<jupyter_code>nlp = spacy.load("en_core_web_sm")
doc = nlp("Berlin is a nice capital. Munich is historical city. We should stay there for a while. ")

# Get all tokens and part-of-speech tags
token_texts = [token.text for token in doc]
pos_tags = [token.pos_ for token in doc]

for index, pos in enumerate(pos_tags):
    # Check if the current token is a proper noun
    if pos == "PROPN":
        # Check if the next token is a verb
        if pos_tags[index + 1] == "VERB":
            result = token_texts[index]
            print("Found proper noun before a verb:", result)
print(pos_tags)
nlp = spacy.load("en_core_web_sm")
doc = nlp("Berlin is a nice capital. Munich is historical city. We should stay there for a while. ")

for token in doc:
  if(token.pos_ == "PROPN"):
    if( (token.i + 1) < len(doc) and (doc[token.i + 1].pos_ == "VERB") ):
      print("Found proper noun before a verb:", token.text)
# print(token.tag_, token.pos_, token.ent_type_)<jupyter_output>Found proper noun before a verb: Berlin
Found proper noun before a verb: Munich<jupyter_text>**Inspect Word Vector**<jupyter_code># Process a text
doc = nlp("Two bananas in pyjamas")

# Get the vector for the token "bananas"
bananas_vector = doc[1].vector
print(bananas_vector)
print(len(bananas_vector))<jupyter_output>96<jupyter_text>**Compare Similarity** Use the doc.similarity method to compare doc1 to doc2 and print the result.<jupyter_code>!python -m spacy download en_core_web_md
nlp_md = spacy.load("en_core_web_md")
doc1 = nlp_md("It's a warm summer day")
doc2 = nlp_md("It's sunny outside")

# Get the similarity of doc1 and doc2
similarity = doc1.similarity(doc2)
print(similarity)<jupyter_output>0.8789265574516525<jupyter_text>Use the token.similarity method to compare token1 to token2 and print the result.<jupyter_code>doc = nlp_md("TV and books")
token1, token2 = doc[0], doc[2]

# Get the similarity of the tokens "TV" and "books"
similarity = token1.similarity(token2)
print(similarity)<jupyter_output>0.22325331<jupyter_text><jupyter_code>doc = nlp_md("This was a great restaurant. Afterwards, we went to a really nice bar.")

# Create spans for "great restaurant" and "really nice bar"
span1 = Span(doc, 3, 5)
span2 = Span(doc, 12, 15)

print(span1)
print(span2)
# Get the similarity of the spans
similarity = span1.similarity(span2)
print(similarity)
for token in doc:
  print(token.text, token.i)
doc = nlp_md("This was a great restaurant. Afterwards, we went to a really nice bar.")

pattern1 = [{"TEXT": "great"}, {"TEXT": "restaurant"}]

matcher = Matcher(nlp_md.vocab)
matcher.add("PATTERN1", None, pattern1)

# Iterate over the matches
for match_id, start, end in matcher(doc):
    # Print pattern string name and text of matched span
    print(doc.vocab.strings[match_id], doc[start:end].text)<jupyter_output>PATTERN1 great restaurant<jupyter_text>**Combining models and rules** - Both patterns in this exercise contain mistakes and won’t match as expected. Can you fix them? If you get stuck, try printing the tokens in the doc to see how the text will be split and adjust the pattern so that each dictionary represents one token.- Edit pattern1 so that it correctly matches all case-insensitive mentions of "Amazon" plus a title-cased proper noun.- Edit pattern2 so that it correctly matches all case-insensitive mentions of "ad-free", plus the following noun.<jupyter_code>doc = nlp(
    "Twitch Prime, the perks program for Amazon Prime members offering free "
    "loot, games and other benefits, is ditching one of its best features: "
    "ad-free viewing. According to an email sent out to Amazon Prime members "
    "today, ad-free viewing will no longer be included as a part of Twitch "
    "Prime for new members, beginning on September 14. However, members with "
    "existing annual subscriptions will be able to continue to enjoy ad-free "
    "viewing until their subscription comes up for renewal. Those with "
    "monthly subscriptions will have access to ad-free viewing until October 15."
)

# Create the match patterns
pattern1 = [{"LOWER": "amazon"}, {"POS": "PROPN"}]
#pattern1 = [{"LOWER": "amazon"}, {"IS_TITLE": True, "POS": "PROPN"}]
pattern2 = [{"LOWER": "ad"}, {"POS": "PUNCT"}, {"LOWER": "free"}, {"POS": "NOUN"}]

# Initialize the Matcher and add the patterns
matcher = Matcher(nlp.vocab)
matcher.add("PATTERN1", None, pattern1)
matcher.add("PATTERN2", None, pattern2)

# Iterate over the matches
for match_id, start, end in matcher(doc):
    # Print pattern string name and text of matched span
    print(doc.vocab.strings[match_id], start, end, doc[start:end].text)<jupyter_output>PATTERN1 7 9 Amazon Prime
PATTERN2 27 31 ad-free viewing
PATTERN1 39 41 Amazon Prime
PATTERN2 44 48 ad-free viewing
PATTERN2 82 86 ad-free viewing
PATTERN2 102 106 ad-free viewing<jupyter_text>**Efficient PharseMatcher** - Sometimes it’s more efficient to match exact strings instead of writing patterns describing the individual tokens. This is especially true for finite categories of things – like all countries of the world. We already have a list of countries, so let’s use this as the basis of our information extraction script. A list of string names is available as the variable COUNTRIES.- Import the PhraseMatcher and initialize it with the shared vocab as the variable matcher.- Add the phrase patterns and call the matcher on the doc.<jupyter_code>COUNTRIES = ['Afghanistan', 'Åland Islands', 'Albania', 'Algeria', 'American Samoa', 'Andorra', 'Angola', 'Anguilla', 'Antarctica', 'Antigua and Barbuda', 'Argentina', 'Armenia', 'Aruba', 'Australia', 'Austria', 'Azerbaijan', 'Bahamas', 'Bahrain', 'Bangladesh', 'Barbados', 'Belarus', 'Belgium', 'Belize', 'Benin', 'Bermuda', 'Bhutan', 'Bolivia (Plurinational State of)', 'Bonaire, Sint Eustatius and Saba', 'Bosnia and Herzegovina', 'Botswana', 'Bouvet Island', 'Brazil', 'British Indian Ocean Territory', 'United States Minor Outlying Islands', 'Virgin Islands (British)', 'Virgin Islands (U.S.)', 'Brunei Darussalam', 'Bulgaria', 'Burkina Faso', 'Burundi', 'Cambodia', 'Cameroon', 'Canada', 'Cabo Verde', 'Cayman Islands', 'Central African Republic', 'Chad', 'Chile', 'China', 'Christmas Island', 'Cocos (Keeling) Islands', 'Colombia', 'Comoros', 'Congo', 'Congo (Democratic Republic of the)', 'Cook Islands', 'Costa Rica', 'Croatia', 'Cuba', 'Curaçao', 'Cyprus', 'Czech Republic', 'Denmark', 'Djibouti', 'Dominica', 'Dominican Republic', 'Ecuador', 'Egypt', 'El Salvador', 'Equatorial Guinea', 'Eritrea', 'Estonia', 'Ethiopia', 'Falkland Islands (Malvinas)', 'Faroe Islands', 'Fiji', 'Finland', 'France', 'French Guiana', 'French Polynesia', 'French Southern Territories', 'Gabon', 'Gambia', 'Georgia', 'Germany', 'Ghana', 'Gibraltar', 'Greece', 'Greenland', 'Grenada', 'Guadeloupe', 'Guam', 'Guatemala', 'Guernsey', 'Guinea', 'Guinea-Bissau', 'Guyana', 'Haiti', 'Heard Island and McDonald Islands', 'Holy See', 'Honduras', 'Hong Kong', 'Hungary', 'Iceland', 'India', 'Indonesia', "Côte d'Ivoire", 'Iran (Islamic Republic of)', 'Iraq', 'Ireland', 'Isle of Man', 'Israel', 'Italy', 'Jamaica', 'Japan', 'Jersey', 'Jordan', 'Kazakhstan', 'Kenya', 'Kiribati', 'Kuwait', 'Kyrgyzstan', "Lao People's Democratic Republic", 'Latvia', 'Lebanon', 'Lesotho', 'Liberia', 'Libya', 'Liechtenstein', 'Lithuania', 'Luxembourg', 'Macao', 'Macedonia (the former Yugoslav Republic of)', 'Madagascar', 'Malawi', 'Malaysia', 'Maldives', 'Mali', 'Malta', 'Marshall Islands', 'Martinique', 'Mauritania', 'Mauritius', 'Mayotte', 'Mexico', 'Micronesia (Federated States of)', 'Moldova (Republic of)', 'Monaco', 'Mongolia', 'Montenegro', 'Montserrat', 'Morocco', 'Mozambique', 'Myanmar', 'Namibia', 'Nauru', 'Nepal', 'Netherlands', 'New Caledonia', 'New Zealand', 'Nicaragua', 'Niger', 'Nigeria', 'Niue', 'Norfolk Island', "Korea (Democratic People's Republic of)", 'Northern Mariana Islands', 'Norway', 'Oman', 'Pakistan', 'Palau', 'Palestine, State of', 'Panama', 'Papua New Guinea', 'Paraguay', 'Peru', 'Philippines', 'Pitcairn', 'Poland', 'Portugal', 'Puerto Rico', 'Qatar', 'Republic of Kosovo', 'Réunion', 'Romania', 'Russian Federation', 'Rwanda', 'Saint Barthélemy', 'Saint Helena, Ascension and Tristan da Cunha', 'Saint Kitts and Nevis', 'Saint Lucia', 'Saint Martin (French part)', 'Saint Pierre and Miquelon', 'Saint Vincent and the Grenadines', 'Samoa', 'San Marino', 'Sao Tome and Principe', 'Saudi Arabia', 'Senegal', 'Serbia', 'Seychelles', 'Sierra Leone', 'Singapore', 'Sint Maarten (Dutch part)', 'Slovakia', 'Slovenia', 'Solomon Islands', 'Somalia', 'South Africa', 'South Georgia and the South Sandwich Islands', 'Korea (Republic of)', 'South Sudan', 'Spain', 'Sri Lanka', 'Sudan', 'Suriname', 'Svalbard and Jan Mayen', 'Swaziland', 'Sweden', 'Switzerland', 'Syrian Arab Republic', 'Taiwan', 'Tajikistan', 'Tanzania, United Republic of', 'Thailand', 'Timor-Leste', 'Togo', 'Tokelau', 'Tonga', 'Trinidad and Tobago', 'Tunisia', 'Turkey', 'Turkmenistan', 'Turks and Caicos Islands', 'Tuvalu', 'Uganda', 'Ukraine', 'United Arab Emirates', 'United Kingdom of Great Britain and Northern Ireland', 'United States of America', 'Uruguay', 'Uzbekistan', 'Vanuatu', 'Venezuela (Bolivarian Republic of)', 'Viet Nam', 'Wallis and Futuna', 'Western Sahara', 'Yemen', 'Zambia', 'Zimbabwe']

doc = nlp_en("Czech Republic may help Slovakia protect its airspace. France is an ally of United States of America")

matcher = PhraseMatcher(nlp_en.vocab)
# Create pattern Doc objects and add them to the matcher
# This is the faster version of: [nlp(country) for country in COUNTRIES]
patterns = list(nlp_en.pipe(COUNTRIES))
matcher.add("COUNTRY", None, *patterns)

# Call the matcher on the test document and print the result
matches = matcher(doc)

print(matches)

print([doc[start:end] for match_id, start, end in matches])<jupyter_output>[(4000319556510314152, 0, 2), (4000319556510314152, 4, 5), (4000319556510314152, 9, 10), (4000319556510314152, 14, 18)]
[Czech Republic, Slovakia, France, United States of America]<jupyter_text>**Extract Countries and Relationship** - In the previous exercise, you wrote a script using spaCy’s PhraseMatcher to find country names in text. Let’s use that country matcher on a longer text, analyze the syntax and update the document’s entities with the matched countries.- Iterate over the matches and create a Span with the label "GPE" (geopolitical entity).- Overwrite the entities in doc.ents and add the matched span.- Get the matched span’s root head token.- Print the text of the head token and the span.<jupyter_code>matcher = PhraseMatcher(nlp_en.vocab)
patterns = list(nlp_en.pipe(COUNTRIES))
matcher.add("COUNTRY", None, *patterns)

# Create a doc and find matches in it
doc = nlp_en("Czech Republic may help Slovakia protect its airspace. France is an ally of United States of America")

# Iterate over the matches
for match_id, start, end in matcher(doc):
    # Create a Span with the label for "GPE"
    span = Span(doc, start, end, label="GPE")
	
    #print(doc.ents)
    # Overwrite the doc.ents and add the span
    doc.ents = list(doc.ents) + [span]
    
     # Get the span's root head token
    span_root_head = span.root.head
    # Print the text of the span root's head token and the span text
    print(span_root_head.text, "-->", span.text)

# Print the entities in the document
print([(ent.text, ent.label_) for ent in doc.ents if ent.label_ == "GPE"])<jupyter_output>Czech --> Czech Republic
Slovakia --> Slovakia
France --> France
United --> United States of America
[('Czech Republic', 'GPE'), ('Slovakia', 'GPE'), ('France', 'GPE'), ('United States of America', 'GPE')]
{'fraction_non_alphanumeric': 0.1323479786731457, 'fraction_numerical': 0.014780319902814336, 'mean_word_length': 5.550839964633068, 'pattern_counts': {'":': 13, '<': 53, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '583324', 'license': 'MIT', 'n_tokens_mistral': 5203, 'n_tokens_neox': 4847, 'n_words': 1763, 'path': 'spaCy_chapter2.ipynb', 'repo_name': 'diem-ai/natural-lanaguage-processing', 'size': '14842'}