<jupyter_start><jupyter_text>Downloading the EP full-text data for text analytics data on a serverDocumentation available at:* [EP full-text data for text analytics data](https://www.epo.org/searching-for-patents/data/bulk-data-sets/text-analytics.html)* [Cloud Storage Client Libraries](https://cloud.google.com/storage/docs/reference/librariesclient-libraries-install-python)  Import model configuration<jupyter_code>import sys
sys.path.append("../settings")
import settings<jupyter_output><empty_output><jupyter_text>Installing new libraries<jupyter_code># Upgrading pip
!/usr/bin/python3.8 -m pip install --upgrade pip
from platform import python_version
print(python_version())
# Installing the Google cloud storage API
!pip3.6 install google
!pip3.6 install google-cloud-storage<jupyter_output>Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: google in /home/antoine/.local/lib/python3.6/site-packages (3.0.0)
Requirement already satisfied: beautifulsoup4 in /home/antoine/.local/lib/python3.6/site-packages (from google) (4.9.1)
Requirement already satisfied: soupsieve>1.2 in /home/antoine/.local/lib/python3.6/site-packages (from beautifulsoup4->google) (2.0.1)
[33mWARNING: You are using pip version 20.0.2; however, version 20.2.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.[0m
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: google-cloud-storage in /home/antoine/.local/lib/python3.6/site-packages (1.30.0)
Requirement already satisfied: google-cloud-core<2.0dev,>=1.2.0 in /home/antoine/.local/lib/python3.6/site-packages (from google-cloud-storage) (1.4.1)
Requirement already satisfied: google-aut[...]<jupyter_text>Authentification in the Google cloud service To run the client library, you must first set up authentication by creating a service account and setting an environment variable. > * Complete the steps of the Google page [Cloud Storage Client Libraries](https://cloud.google.com/storage/docs/reference/librariesclient-libraries-install-python) to set up authentication.* The keys and confidential settings are stored in a dedicated folder.* Run `gsutil config` in a proper terminal then follow the steps to authentificate> * /!\ As from now, the actions will be charged on the Google cloud account!   First glance at the content of the bucket<jupyter_code># Visualisation of the files in the EPO public repository
# Here we see that two versions of the database exist (the 2019 and the 2020 version)
!gsutil -u {settings.google_project_id} ls gs://epo-public
# Estimating the size of the 2019 edition of the EP data: 232 GB
!gsutil -u {settings.google_project_id} du -s gs://epo-public/EP-fulltext-for-text-analytics_2019week31/
# Estimating the size of the 2020 edition of the EP data
# Here we see that the size of the 2020 bucket is very small: 11 bytes!
!gsutil -u {settings.google_project_id} du -s gs://epo-public/EP-fulltext-for-text-analytics_2020week08/<jupyter_output>11           gs://epo-public/EP-fulltext-for-text-analytics_2020week08<jupyter_text>Copying a single file (6 GB) to the local repository* EP full-text data for text analytics comprises around 35 data files of about 5-8 GB each.* each file contains the publications associated with 100 000 publication numbers.<jupyter_code>!gsutil -u {settings.google_project_id} cp gs://epo-public/EP-fulltext-for-text-analytics_2019week31/EP1400000.txt {settings.local_storage_destination_sample}<jupyter_output>Copying gs://epo-public/EP-fulltext-for-text-analytics_2019week31/EP1400000.txt...
- [1 files][  5.9 GiB/  5.9 GiB]   59.0 MiB/s                                   
Operation completed over 1 objects/5.9 GiB.<jupyter_text>We visualise the data we have just downloaded!<jupyter_code>import pandas as pd
data = pd.read_csv(settings.local_storage_destination_sample + '/EP1400000.txt', sep = '\t',  header = None)
data.columns = ['publication_authority', # will always have the value "EP"
                'publication_number', # a seven-digit number
                'publication_kind', # see https://www.epo.org/searching-for-patents/helpful-resources/first-time-here/definitions.html for help.
                'publication_date', # in format YYYY-MM-DD
                'language_text_component', # de, en, fr; xx means unknown
                'text_type', # TITLE, ABSTR, DESCR, CLAIM, AMEND, ACSTM, SREPT, PDFEP
                'text' # it contains, where appropriate, XML tags for better structure. You will find the DTD applicable to all parts of the publication at: http://docs.epoline.org/ebd/doc/ep-patent-document-v1-5.dtd
               ]
data.head(10)<jupyter_output><empty_output><jupyter_text>Downloading the entire database on the local server<jupyter_code># 2019 edition
!gsutil -u {settings.google_project_id} cp gs://epo-public/EP-fulltext-for-text-analytics_2019week31/* {settings.local_storage_destination_2019}
# 2020 edition
!gsutil -u {settings.google_project_id} cp gs://epo-public/EP-fulltext-for-text-analytics_2020week08/* {settings.local_storage_destination_2020}<jupyter_output>Copying gs://epo-public/EP-fulltext-for-text-analytics_2020week08/$description.txt...
Copying gs://epo-public/EP-fulltext-for-text-analytics_2020week08/$license.txt...
Copying gs://epo-public/EP-fulltext-for-text-analytics_2020week08/EP0000000.txt...
| [3 files][  5.2 GiB/  5.2 GiB]   58.7 MiB/s                                   
==> NOTE: You are performing a sequence of gsutil operations that may
run significantly faster if you instead use gsutil -m cp ... Please
see the -m section under "gsutil help options" for further information
about when gsutil -m can be advantageous.

Copying gs://epo-public/EP-fulltext-for-text-analytics_2020week08/EP0100000.txt...
Copying gs://epo-public/EP-fulltext-for-text-analytics_2020week08/EP0200000.txt...
Copying gs://epo-public/EP-fulltext-for-text-analytics_2020week08/EP0300000.txt...
Copying gs://epo-public/EP-fulltext-for-text-analytics_2020week08/EP0400000.txt...
Copying gs://epo-public/EP-fulltext-for-text-analytics_2020week08/EP0500000.txt...
C[...]
{'fraction_non_alphanumeric': 0.10698725189607874, 'fraction_numerical': 0.04566725835081491, 'mean_word_length': 4.30196749358426, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 27, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '281767', 'license': 'MIT', 'n_tokens_mistral': 2113, 'n_tokens_neox': 1916, 'n_words': 538, 'path': 'notebooks/download_EP_full_text_data.ipynb', 'repo_name': 'amc-econ/nlp_patent_classification', 'size': '6197'}