<jupyter_start><jupyter_text>Redaction of Names from Documents Using SpaCy Applications of Named Entity Recognition- Sanitization is the process of removing sensitive information from a document or other message (or sometimes encrypting it), so that the document may be distributed to a broader audience  Purpose of Sanitization/Redaction of Document- For anonymity of source in document- To ensure there is no sensitive or personally identifiable information in the document- Censorship<jupyter_code>import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp(open('covid_research.txt').read())
for token in doc.ents:
    if token.label_ == 'PERSON':
        print('{:<30}{:<30}'.format(token.text, token.label_))
# Function to Sanitize/Redact Names
def sanitize_names(text):
    docx = nlp(text)
    redacted_sentences = []
    for ent in docx.ents:
        ent.merge()
    for token in docx:
        if token.ent_type_ == 'PERSON':
            redacted_sentences.append("[REDACTED]")
        else:
            redacted_sentences.append(token.string)
    return "".join(redacted_sentences)
ex1 = open('covid_research.txt').read()
# Redact the Names
sanitize_names(ex1)
from spacy import displacy
displacy.render(doc, style = 'ent')
mytext = nlp(sanitize_names(ex1))
displacy.render(mytext, style = 'ent')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09895052473763119, 'fraction_numerical': 0.005247376311844078, 'mean_word_length': 4.493827160493828, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '203864', 'license': 'MIT', 'n_tokens_mistral': 427, 'n_tokens_neox': 412, 'n_words': 124, 'path': 'Spacy/13_Spacy_Redaction_Names.ipynb', 'repo_name': 'sudevansujit/7_NLP', 'size': '1334'}