<jupyter_start><jupyter_text>File Manipulation with Azure Blob StorageWe try a few file manipulation between a local computer and a blob storage on Azure. It requires [azure-sdk-for-python](https://github.com/Azure/azure-sdk-for-python) and [pyenbc](http://www.xavierdupre.fr/app/pyenbc/helpsphinx/index.html). We first create a dummy file.<jupyter_code>import pandas, random
mat = [ {"x":random.random(), "y":random.random()} for i in range(0,1000)]
df = pandas.DataFrame(mat)
df.to_csv("randomxy.txt", sep="\t", encoding="utf8")<jupyter_output><empty_output><jupyter_text>We need credentials and to avoid having them in clear in the notebook, we use a HTML form:<jupyter_code>import pyquickhelper.ipythonhelper as ipy
params={"blob_storage":"hdblobstorage", "password":""}
ipy.open_html_form(params=params,title="credentials",key_save="blobservice")<jupyter_output><empty_output><jupyter_text>We stored the values in two variables in the workspace:<jupyter_code>blobstorage = blobservice["blob_storage"]
blobpassword = blobservice["password"]<jupyter_output><empty_output><jupyter_text>We need pyensae >= 1.2:<jupyter_code>import pyensae
import pyenbc
%load_ext pyensae
%load_ext pyenbc
pyensae.__version__, pyenbc.__version__
%blob_open --help<jupyter_output>usage: blob_open [-h] [-b BLOBSTORAGE] [-p BLOBPASSWORD]

open a connection to an Azure blob storage, by default, the magic command
takes blobstorage and blobpassword local variables as default values

optional arguments:
  -h, --help            show this help message and exit
  -b BLOBSTORAGE, --blobstorage BLOBSTORAGE
                        blob storage name
  -p BLOBPASSWORD, --blobpassword BLOBPASSWORD
                        blob password
usage: blob_open [-h] [-b BLOBSTORAGE] [-p BLOBPASSWORD]<jupyter_text>We open a connection to the blob storage:<jupyter_code>cl, bs = %blob_open
cl, bs<jupyter_output><empty_output><jupyter_text>We extract the available containers:<jupyter_code>l = %blob_containers
l<jupyter_output><empty_output><jupyter_text>We get the content of one container:<jupyter_code>df = %blob_ls hdblobstorage
df.tail(n=5)
%hd_wasb_prefix
cl.wasb_to_file("hdblobstorage", "velib_several_days")<jupyter_output><empty_output><jupyter_text>We upload the file we created in the first cell:<jupyter_code>%blob_up randomxy.txt clusterensaeazure1/testpyenbc/randomxy.txt<jupyter_output><empty_output><jupyter_text>We check the file is over there:<jupyter_code>%blob_ls clusterensaeazure1/testpyenbc<jupyter_output><empty_output><jupyter_text>We try an extended version:<jupyter_code>%blob_lsl clusterensaeazure1/testpyenbc<jupyter_output><empty_output><jupyter_text>If you need information not accessible through a magic command, you can use the variable ``bs`` (type [azure.storage.blobservice.BlobService](http://www.xavierdupre.fr/app/azure-sdk-for-python/helpsphinx/storage/blobservice.htmlmodule-azure.storage.blobservice)):<jupyter_code>l=bs.get_block_list("clusterensaeazure1", "testpyenbc/randomxy.txt")
for _ in l.committed_blocks:
    print("size=",_.size, "id=",_.id)<jupyter_output>size= 43483 id= 00000000<jupyter_text>We download this again to the local computer:<jupyter_code>%blob_down clusterensaeazure1/testpyenbc/randomxy.txt randomxx_copy.txt --overwrite
%lsr r.*[.]txt<jupyter_output><empty_output><jupyter_text>PIG scripts usually produce more than one output and it is convenient to merge them while downloading them. To test that, we upload a second time our file with a different names:<jupyter_code>%blob_up randomxy.txt clusterensaeazure1/testpyenbc/randomxy2.txt
%blob_ls clusterensaeazure1/testpyenbc<jupyter_output><empty_output><jupyter_text>And we merge them:<jupyter_code>%blob_downmerge clusterensaeazure1/testpyenbc randomall.txt --overwrite<jupyter_output><empty_output><jupyter_text>We check the size of file ``randomall.txt`` is twice bigger:<jupyter_code>%lsr r.*[.]txt<jupyter_output><empty_output><jupyter_text>We finally remove the files from the blob storage:<jupyter_code>%blob_delete clusterensaeazure1/testpyenbc/randomxy.txt
%blob_delete clusterensaeazure1/testpyenbc/randomxy2.txt<jupyter_output><empty_output><jupyter_text>We check it disappeared:<jupyter_code>%blob_ls clusterensaeazure1/testpyenbc/<jupyter_output><empty_output><jupyter_text>And we close the connection:<jupyter_code>%blob_close<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10461114934618032, 'fraction_numerical': 0.008029364533149804, 'mean_word_length': 7.599605522682446, 'pattern_counts': {'":': 4, '<': 71, '<?xml version=': 0, '>': 72, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '361127', 'license': 'MIT', 'n_tokens_mistral': 1518, 'n_tokens_neox': 1515, 'n_words': 313, 'path': '_doc/notebooks/azure_blob_storage.ipynb', 'repo_name': 'sdpython/pyenbc', 'size': '4359'}