<jupyter_start><jupyter_text>Collect any Internet pages |**[Overview](Overview)** |**[Installation](Installation)||**[Prior-steps](Prior-steps)|**[How-to-use](How-to-use)**|**[Next-steps](Next-steps)**| OverviewIn addition to the text files you have collected, there may be one or two internet pages which you want to add to the analysis. I have added this partly to show what happens when a document is added which is about the same subject, but comes at the subject from a very different angle. In order to compare the results with the Orange results, I have not run this block of code.  InstallationCheck installation has been made, as per the [READme](https://github.com/lawrencerowland/Data-Model-for-Project-Frameworks/blob/master/Project-frameworks-by-using-NLP-with-Python-libraries/README.md) Prior-stepsNone of the steps 1-3 are needed in order to generate these keywords.But to generate keywords of new documents, review these steps to see which you require. How-to-use Set directory<jupyter_code>import os
directory= '/Users/lawrence/Documents/GitHub/Data-Model-for-Project-Frameworks/Nuclear-project-data-for-use-in-generating-example-frameworks'
print (os.listdir(directory))
# Change directory location for your particular set-up.<jupyter_output>['.DS_Store', 'Corpus_as_one_string.txt', 'Keywords_for_whole_corpus.csv', 'Keywords_for_whole_corpus.txt']<jupyter_text>Bring in web page<jupyter_code>#hide
import requests
text = requests.get('https://archive.org/stream/ProjectManagementForTheOilAndGasIndustry/ProjectManagementForTheOilAndGasIndustry_djvu.txt').text
text=text[70000:300000] #here I have stripped the front text by inspection only.
#text=strip_multiple_whitespaces(text)  #may wish to turn this off for readability
filename="PM_guidance_for_Energy_Projects" # Set your own title
f= open(directory+'/'+filename+".txt","w+") 
f.write(filename+"\n"+text)
f.close()
#If you wish to collect many Internet pages, you will need to construct a loop.
# You also will need to check how well the text is extracted from  particular sources
# and consult the Python Requests module documentation if it needs adjustment.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10018467220683287, 'fraction_numerical': 0.006001846722068329, 'mean_word_length': 6.247491638795987, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '173245', 'license': 'MIT', 'n_tokens_mistral': 654, 'n_tokens_neox': 625, 'n_words': 210, 'path': 'Project-frameworks-by-using-NLP-with-Python-libraries/Jupyter-notebooks/Step-2-collect-internet.ipynb', 'repo_name': 'lawrencerowland/Automation-of-construction-contract-review', 'size': '2166'}