<jupyter_start><jupyter_text>Experiments Natural language processingWhile initial investigations on using NLP were made with NLTK, a quick wikipedia scan pointed to spaCy as an option that may be higher level. First impression is also that the documentation is slicker, and popular in so far as conting the guthub stars.resources (see also browser bookmarks)* [Natural Language Processing and Computational Linguistics](http://pqtech.safaribooksonline.com/book/programming/machine-learning/9781788838535)* [spaCy](https://spacy.io)<jupyter_code>import spacy
nlp = spacy.load('en_core_web_sm')<jupyter_output><empty_output><jupyter_text>```nd and dark clay.', u'clay orange, some granite',       u'clay and sand bars - some very hard granitic bands',       u'coffee rock', u'black clay.', u'grey, yellow and brown clay',       u'ironstone with clay', u'clay, green.  total depth 179ft',       u'sand - cream to orange, fine to coarse, subangular to rounded, poorly sorted, loose, clear, light grey and iron stained quartz grains, rare pink with purple quartz grains, 5-10% shell fragments increasing % and fragments size with depth, trace dark green black lithic fragments.',       u'light grey with 1.5mm white sands', u'yellowicream sandy clay',       u'leederville formation.', u'grey clay and granitic band',       u'sand, calcitareous, buff, mg.',       u'sand, granular to small pebbles mainly well sorted',       u'grey with black clays of granite bands',       u'shell bed - medium sand', u'shale.',       u'dark brown bands of coarse sand', u'yellow sand',       u'dark grey fine grained silty to clayey sand with organics to fine',       u'shale, sandstone and siltstone interbedded',       u'clay, blue with black', u'sub soil', u'sand', u'white clay',       u'loam.  munsell colour 2.5yr 4with3 reddish brown.', u'capstone',       u'crumbly limestone',       u'sandstone, grey greenish, medium to very coarse grained, moderately to well sorted.',       u'sand, minor clay dark brown iron cementation', u'red clay.',       u'sand.  clay at 12m', u'hard limestone', u'grey clay and shell.',       u'clay, gravelly, very slightly silty, grey-brown',       u'yellow to white fine sand', u'gravel & coarse orange sands',       u'white sand, coarse', u'light brown clay.',       u'fine grained sandy clay.', u'granite boulder',       u'black sticky clay', u'white sands', u'not logged',       u'clay.                        (log - quaternary))',       u'black shale, small sand stringers interbedded between layers.'],       dtype='<U473')  ```<jupyter_code>doc = nlp(u'this contains dark grey, fine grained silty to clayey sand with organics to fine')
for t in doc:
    print(t.text, t.pos_, t.dep_)
    
for t in doc.ents:
    print(t.text, t.start_char, t.end_char, t.label_)
doc.ents
doc = nlp(u'sand - cream to orange, fine to coarse, subangular to rounded, poorly sorted, loose, clear, light grey and iron stained quartz grains, rare pink with purple quartz grains, 5-10% shell fragments increasing % and fragments size with depth')
for t in doc:
    print(t.text, t.pos_, t.dep_)
for t in doc.ents:
    print(t.text, t.start_char, t.end_char, t.label_)
len(doc.ents[0].vector)
doc = nlp(u'Understanding structure in conversations and extracting information were key problems in early NLP, and the fruits of the research done are being very apparent in the 21st century')
for t in doc:
    print(t.text, t.pos_, t.dep_)
for t in doc.ents:
    print(t.text, t.start_char, t.end_char, t.label_)
doc = nlp(u"Sand, yellow brown, slightly clayey, fine-medium, mod sorted qtz sub ang to rounded with yellow brown staining; scattered coarse well rounded qtz grains and rare heavy mineral grains.")
for t in doc:
    print(t.text, t.pos_, t.dep_)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0997618417570786, 'fraction_numerical': 0.00952632971685631, 'mean_word_length': 3.909090909090909, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 9, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '566555', 'license': 'MIT', 'n_tokens_mistral': 1201, 'n_tokens_neox': 1106, 'n_words': 461, 'path': 'trials/spacy_trial.ipynb', 'repo_name': 'Sudhir22/pyela', 'size': '3779'}