<jupyter_start><jupyter_text>Creating examples from existing datasets via perturbations<jupyter_code>editor = Editor()<jupyter_output><empty_output><jupyter_text>Let's start by creating a fictitious dataset:<jupyter_code>data = ['John is a very smart person, he lives in Ireland.',
        '<NAME> was born and raised in Chicago',
        '<NAME> has 3 sisters.',
        'Mary is not a nurse.',
        'Julianne is an engineer.',
        'My brother Andrew used to be a lawyer.']<jupyter_output><empty_output><jupyter_text>Writing your own perturbations Let's say we want to write a perturbation function to replace some professions with other professions:<jupyter_code>import re
def change_professions(x, *args, **kwargs):
    # Returns empty or a list of strings with profesions changed
    professions = ['doctor', 'nurse', 'engineer', 'lawyer']
    ret = []
    for p in professions:
        if re.search(r'\b%s\b' % p, x):
            ret.extend([re.sub(r'\b%s\b' % p, p2, x) for p2 in professions if p != p2])
    return ret
            
change_professions(data[3])<jupyter_output><empty_output><jupyter_text>We could use this function on every example in `data`, and keep only cases where it applies.There is an auxiliary function that does this (and more) for us, called `Perturb.perturb`:<jupyter_code>ret = Perturb.perturb(data, change_professions, keep_original=True)
ret.data<jupyter_output><empty_output><jupyter_text>Notice how `Perturb.perturb` automatically ignored examples in our dataset where the perturbation didn't return anything, e.g. 'John is a very smart person'.  We set `keep_original=True`, and therefore the original data point is kept as the first in every example list. This is typically what we want to do in perturbation tests. This is what we would get if we had set it to `False`:<jupyter_code>ret = Perturb.perturb(data, change_professions, keep_original=False)
ret.data<jupyter_output><empty_output><jupyter_text>We can also specify a number of samples if our dataset is too large:<jupyter_code>ret = Perturb.perturb(data, change_professions, keep_original=False, nsamples=2)
ret.data<jupyter_output><empty_output><jupyter_text>Finally, we may want our perturbation function to return some metadata. In our case, maybe we want to remember which profession was swapped into which profession. To do so, let's rewrite `change_professions` so that it returns an additional list with metadata:<jupyter_code>def change_professions(x, meta=False, *args, **kwargs):
    # Returns empty or a list of strings with profesions changed
    professions = ['doctor', 'nurse', 'engineer', 'lawyer']
    ret = []
    ret_meta = []
    for p in professions:
        if re.search(r'\b%s\b' % p, x):
            ret.extend([re.sub(r'\b%s\b' % p, p2, x) for p2 in professions if p != p2])
            ret_meta.extend([(p, p2) for p2 in professions if p != p2])
    if meta:
        return ret, ret_meta
    else:
        return ret
            
change_professions(data[3], meta=True)<jupyter_output><empty_output><jupyter_text>We can now call `Perturb.perturb` with `meta=True`. Whatever keyword arguments we use in `Perturb.perturb` get passed along to the perturbation function. In addition, if we set `meta=True`, `ret.meta` will have the metadata.<jupyter_code>ret = Perturb.perturb(data, change_professions, keep_original=True, nsamples=1, meta=True)
print('Data')
print(ret.data)
print('Metadata')
print(ret.meta)<jupyter_output>Data
[['Julianne is an engineer.', 'Julianne is an doctor.', 'Julianne is an nurse.', 'Julianne is an lawyer.']]
Metadata
[[None, ('engineer', 'doctor'), ('engineer', 'nurse'), ('engineer', 'lawyer')]]<jupyter_text>General-purpose perturbations We provide some general-purpose perturbation functions. Some assume you have preprocessed the data with spacy, so let's do that:<jupyter_code>import spacy
nlp = spacy.load('en_core_web_sm')
pdata = list(nlp.pipe(data))
pdata[:3]<jupyter_output><empty_output><jupyter_text>Punctuation `Perturb.strip_punctuation` removes punctuation:<jupyter_code>pdata[0], Perturb.strip_punctuation(pdata[0])<jupyter_output><empty_output><jupyter_text>`Perturb.punctuation` adds and / or removes punctuation (notice that we add it when it's not there and remove it when it is)<jupyter_code>ret = Perturb.perturb(pdata, Perturb.punctuation)
ret.data[:4]<jupyter_output><empty_output><jupyter_text>Typos<jupyter_code>data[0], Perturb.add_typos(data[0])
ret = Perturb.perturb(data, Perturb.add_typos, nsamples=1)
ret.data<jupyter_output><empty_output><jupyter_text>Contractions `Perturb.expand_contractions` and `Perturb.contract` act on a single string:<jupyter_code>data[3], Perturb.contract(data[3])
Perturb.expand_contractions('What\'s going on?')<jupyter_output><empty_output><jupyter_text>`Perturb.contractions` contracts AND expands contractions if possible:<jupyter_code>Perturb.contractions('What\'s going on? I am not happy')
ret = Perturb.perturb(data, Perturb.contractions)
ret.data<jupyter_output><empty_output><jupyter_text>Changing named entities The following functions all assume you have parsed the input with spacy.   Perturb.change_names allows you to replace person names automatically.  You can specify if you only want first names, first and last names, etc.<jupyter_code>ret = Perturb.perturb(pdata[2:3], Perturb.change_names, nsamples=1)
ret.data<jupyter_output><empty_output><jupyter_text>You can also specify how many replacements with `n` (default is 10):<jupyter_code>ret = Perturb.perturb(pdata, Perturb.change_names, nsamples=1, n=3)
ret.data
ret = Perturb.perturb(pdata, Perturb.change_names, nsamples=1, first_only=True, n=3)
ret.data
ret = Perturb.perturb(pdata, Perturb.change_names, nsamples=1, last_only=True, n=3)
ret.data<jupyter_output><empty_output><jupyter_text>You can also set `meta=True` if you want to save the change in the metadata:<jupyter_code>ret = Perturb.perturb(pdata, Perturb.change_names, nsamples=1, n=3, last_only=True, meta=True)
ret.data[0][1:], ret.meta[0][1:]<jupyter_output><empty_output><jupyter_text>Similarly, you can change locations with `Perturb.change_location`:<jupyter_code>ret = Perturb.perturb(pdata, Perturb.change_location, nsamples=1, n=3, meta=True)
ret.data[0], ret.meta[0]<jupyter_output><empty_output><jupyter_text>And numbers:<jupyter_code>ret = Perturb.perturb(pdata, Perturb.change_number, nsamples=1, n=3, meta=True)
ret.data[0], ret.meta[0]<jupyter_output><empty_output><jupyter_text>Negation We have some experimental functions (may or may not work) for adding or removing negations:<jupyter_code>data = []
ret = Perturb.perturb(pdata, Perturb.add_negation)
ret.data
for t in ['This is not good', 'He didn\'t play the guitar', 'He doesn\'t play anything', 'She wasn\'t sad']:
    print(t)
    print(Perturb.remove_negation(nlp(t)))
    print()<jupyter_output>This is not good
This is good

He didn't play the guitar
He played the guitar

He doesn't play anything
He plays anything

She wasn't sad
She was sad
{'fraction_non_alphanumeric': 0.11243367273770256, 'fraction_numerical': 0.006883694249247096, 'mean_word_length': 5.8641732283464565, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 81, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '75717', 'license': 'MIT', 'n_tokens_mistral': 2382, 'n_tokens_neox': 2317, 'n_words': 664, 'path': 'notebooks/tutorials/2. Perturbing data.ipynb', 'repo_name': 'ahzz1207/chinese_ck', 'size': '6983'}