<jupyter_start><jupyter_text>Simple Tools from Extracting Quantities from StringsSuppose we have a report and we want to find the sentences that are talking about numerical things....*Originally inspired by [When you get data in sentences: how to use a spreadsheet to extract numbers from phrases](https://onlinejournalismblog.com/2019/07/29/when-you-get-data-in-sentences-how-to-use-a-spreadsheet-to-extract-numbers-from-phrases/), <NAME>, Online Journalism blog, from which some of the example sentences (sic!) are taken.*Distribution: https://twitter.com/paulbradshaw/status/1158752556958519297  Potentially Useful Python Packages- `quantulum`: extract quantities from natural language text;- `ctparse`: extract time / date related quantities from natural language text;- `r1chardj0n3s/parse`: easy scrape / regex extraction from semi-structred text using `format()` like patterns; example use [here](https://github.com/psychemedia/parlihacks/blob/master/notebooks/MP%20Register%20of%20Interests.ipynb);- [`dateparser`](https://github.com/scrapinghub/dateparser) [[docs](https://dateparser.readthedocs.io/en/latest/)]: "easily parse localized dates in almost any string formats commonly found on web pages" (includes foreign language detection);- [invoice2data](https://github.com/invoice-x/invoice2data):  Example SentencesMake a start on some sample test sentences...<jupyter_code>sentences = [
    '4 years and 6 months’ imprisonment with a licence extension of 2 years and 6 months',
    'No quantities here',
    'I measured it as 2 meters and 30 centimeters.',
    "four years and six months' imprisonment with a licence extension of 2 years and 6 months",
    'it cost £250... bargain...',
    'it weighs four hundred kilograms.',
    'It weighs 400kg.',
    'three million, two hundred & forty, you say?',
    'it weighs four hundred and twenty kilograms.'
    
]<jupyter_output><empty_output><jupyter_text>`quantulum3`[`quantulum3`](https://github.com/nielstron/quantulum3) is a Python package *"for information extraction of quantities from unstructured text"*.<jupyter_code>#!pip3 install quantulum3
from quantulum3 import parser
for sent in sentences:
    print(sent)
    p = parser.parse(sent)
    if p:
        print('\tSpoken:',parser.inline_parse_and_expand(sent))
        print('\tNumeric elements:')
        for q in p:
            display(q)
            print('\t\t{} :: {}'.format(q.surface, q))
    print('\n---------\n')<jupyter_output>4 years and 6 months’ imprisonment with a licence extension of 2 years and 6 months
	Spoken: four years and six months’ imprisonment with a licence extension of two years and six months
	Numeric elements:<jupyter_text>Finding quantity statements in large textsIf we have a large block of text, we might want to quickly skim it for quantity containing sentences, we can do something like the following...<jupyter_code>import spacy
nlp = spacy.load('en_core_web_lg', disable = ['ner'])
text = '''
Once upon a time, there was a thing. The thing weighed forty kilogrammes and cost £250. 
It was blue. It took forty five minutes to get it home. 
What a day that was. I didn't get back until 2.15pm. Then I had some cake for tea.
'''
doc = nlp(text)
for sent in doc.sents:
    print(sent)
for sent in doc.sents:
    sent = sent.text
    p = parser.parse(sent)
    if p:
        print('\tSpoken:',parser.inline_parse_and_expand(sent))
        print('\tNumeric elements:')
        for q in p:
            display(q)
            print('\t\t{} :: {}'.format(q.surface, q))
    print('\n---------\n')<jupyter_output>Spoken: 
Once upon one instance, there was a thing.
	Numeric elements:<jupyter_text>Annotating a datasetCan we extract numbers from sentences in a CSV file? Yes we can...<jupyter_code>url = 'https://raw.githubusercontent.com/BBC-Data-Unit/unduly-lenient-sentences/master/ULS%20for%20Sankey.csv'
import pandas as pd

df = pd.read_csv(url)
df.head()
#get a row
df.iloc[1]
#and a, erm. sentence...
df.iloc[1]['Original sentence (refined)']
parser.parse(df.iloc[1]['Original sentence (refined)'])
def amountify(txt):
    #txt may be some flavout of nan...
    #handle scruffily for now...
    try:
        if txt:
            p = parser.parse(txt)
            x=[]
            for q in p:
                x.append( '{} {}'.format(q.value, q.unit.name))
            return '::'.join(x)
        return ''
    except:
        return
df['amounts'] = df['Original sentence (refined)'].apply(amountify)
df.head()<jupyter_output><empty_output><jupyter_text>We could then do something to split multiple amounts into multiple rows or columns...  Parsing Semi-Structured SentencesThe sentencing sentences look to have a reasonable degree of structure to them (or at least, there are some commenalities in the way some of them are structured).We can exploit this structure by writing some more specific pattern matches to pull out even more information.<jupyter_code>df['Original sentence (refined)'][:20].apply(print);<jupyter_output>3 years imprisonment
6 years imprisonment - Disqualified driving - 8 years
9 months imprisonment suspended for 2 years
4 years and 10 months imprisonment - consecutive to any other periods of imprisonment
unknown
unknown
3 year community sentence attend sex offenders group and pay surcharge of £60 within 2 months
£850 Fine
12-months disqualification
Community Sentence / SOPO for 5 years/ pay a surcharge of £60 within 3 months
Bound over in the sum of £100.00 for 12 months
18 months imprisonment suspended for 2 years
9 years imprisonment
13 months imprisonment
14 years and 6 months imprisonment
3 years and 9 months imprisonment
Life imprisonment with a minimum of 25 years
6 years and 3 months imprisonment
4 years imprisonment
12 months imprisonment - confiscation under POCA 2002
{'fraction_non_alphanumeric': 0.09941115344648424, 'fraction_numerical': 0.02216834083824039, 'mean_word_length': 3.978448275862069, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '46022', 'license': 'MIT', 'n_tokens_mistral': 1775, 'n_tokens_neox': 1633, 'n_words': 646, 'path': 'notebooks/Quantity Parsing.ipynb', 'repo_name': 'psychemedia/parlihacks', 'size': '5781'}