<commit_before>{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "notebook.ipynb",
      "provenance": [],
      "private_outputs": true,
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow IO Authors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "colab_type": "code",
        "id": "tuOe1ymfHZPu",
        "colab": {}
      },
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qFdPvlXBOdUN"
      },
      "source": [
        "# Azure blob storage with TensorFlow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MfBg1C5NB3X0"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/io/tutorials/azure\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/io/blob/master/docs/tutorials/azure.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/io/blob/master/docs/tutorials/azure.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "      <td>\n",
        "    <a href=\"https://raw.githubusercontent.com/tensorflow/io/master/docs/tutorials/azure.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xHxb-dlhMIzW"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This tutorial shows how to use read and write files on [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/) with TensorFlow, through TensorFlow IO's Azure file system integration.\n",
        "\n",
        "An Azure storage account is needed to read and write files on Azure Blob Storage. The Azure Storage Key should be provided through environmental variable:\n",
        "```\n",
        "os.environ['TF_AZURE_STORAGE_KEY'] = '<key>'\n",
        "```\n",
        "\n",
        "The storage account name and container name are part of the filename uri:\n",
        "```\n",
        "azfs://<storage-account-name>/<container-name>/<path>\n",
        "```\n",
        "\n",
        "In this tutorial, for demo purposes we also provides the optional setup of [Azurite](https://github.com/Azure/Azurite) which is a Azure Storage emulator. With Azurite emulator it is possible to read and write files through Azure blob storage interface with TensorFlow."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MUXex9ctTuDB"
      },
      "source": [
        "## Setup and usage"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "upgCc3gXybsA",
        "colab_type": "text"
      },
      "source": [
        "### Install required packages, and restart runtime"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uUDYyMZRfkX4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "try:\n",
        "  %tensorflow_version 2.x \n",
        "except Exception:\n",
        "  pass\n",
        "\n",
        "!pip install tensorflow-io"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yZmI7l_GykcW",
        "colab_type": "text"
      },
      "source": [
        "### Install and setup Azurite (optional)\n",
        "\n",
        "In case an Azure Storage Account is not available, the following is needed to install and setup Azurite that emulates the Azure Storage interface:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YUj0878jPyz7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!npm install -g azurite@2.7.0\n",
        "\n",
        "get_ipython().system_raw('azurite-blob -s &')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "acEST3amdyDI",
        "colab_type": "text"
      },
      "source": [
        "### Read and write files to Azure Storage with TensorFlow\n",
        "\n",
        "The following is an example of reading and writing files to Azure Storage with TensorFlow's API.\n",
        "\n",
        "It behaves the same way as other file systems (e.g., POSIX or GCS) in TensorFlow once `tensorflow-io` package is imported, as `tensorflow-io` will automatically register `azfs` scheme for use.\n",
        "\n",
        "The Azure Storage Key should be provided through `TF_AZURE_STORAGE_KEY` environmental variable. Otherwise `TF_AZURE_USE_DEV_STORAGE` could be set to `True` to use Azurite emulator instead:\n"<commit_msg>Maybe fix npm install. (#785)<commit_after>{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "notebook.ipynb",
      "provenance": [],
      "private_outputs": true,
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow IO Authors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "colab_type": "code",
        "id": "tuOe1ymfHZPu",
        "colab": {}
      },
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qFdPvlXBOdUN"
      },
      "source": [
        "# Azure blob storage with TensorFlow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MfBg1C5NB3X0"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/io/tutorials/azure\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/io/blob/master/docs/tutorials/azure.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/io/blob/master/docs/tutorials/azure.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "      <td>\n",
        "    <a href=\"https://raw.githubusercontent.com/tensorflow/io/master/docs/tutorials/azure.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MsyhTesh4LSy",
        "colab_type": "text"
      },
      "source": [
        "Caution: In addition to python packages this notebook uses `npm install --user` to install packages. Be careful when running locally. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xHxb-dlhMIzW"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This tutorial shows how to use read and write files on [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/) with TensorFlow, through TensorFlow IO's Azure file system integration.\n",
        "\n",
        "An Azure storage account is needed to read and write files on Azure Blob Storage. The Azure Storage Key should be provided through environmental variable:\n",
        "```\n",
        "os.environ['TF_AZURE_STORAGE_KEY'] = '<key>'\n",
        "```\n",
        "\n",
        "The storage account name and container name are part of the filename uri:\n",
        "```\n",
        "azfs://<storage-account-name>/<container-name>/<path>\n",
        "```\n",
        "\n",
        "In this tutorial, for demo purposes we also provides the optional setup of [Azurite](https://github.com/Azure/Azurite) which is a Azure Storage emulator. With Azurite emulator it is possible to read and write files through Azure blob storage interface with TensorFlow."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MUXex9ctTuDB"
      },
      "source": [
        "## Setup and usage"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "upgCc3gXybsA",
        "colab_type": "text"
      },
      "source": [
        "### Install required packages, and restart runtime"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uUDYyMZRfkX4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "try:\n",
        "  %tensorflow_version 2.x \n",
        "except Exception:\n",
        "  pass\n",
        "\n",
        "!pip install tensorflow-io"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yZmI7l_GykcW",
        "colab_type": "text"
      },
      "source": [
        "### Install and setup Azurite (optional)\n",
        "\n",
        "In case an Azure Storage Account is not available, the following is needed to install and setup Azurite that emulates the Azure Storage interface:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YUj0878jPyz7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!npm install azurite@2.7.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KXbiNLKY4kNM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Run `azurite-blob -s` as a background process. \n",
        "# IPython doesn't recognize `&` in inline bash cells.\n",
        "get_ipython().system_raw('azurite-blob -s &')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "acEST3amdyDI",
        "colab_type": "text"
      },
      "source": [
        "### Read and write files to Azure Storage with TensorFlow\n",
        "\n",
        "The following is an example of reading and writing files to Azure Storage with TensorFlow's API.\n",
        "\n",
        "It behaves the same way as other file systems (e.g., POSIX or GCS) in TensorFlow once `tensorflow-io` package is imported, as `tensorflow-io` will automatically register `azfs` scheme for use.\n",
        "\n",
        "The Azure Storage Key should be provided through `TF_AZURE_STORAGE_KEY` environmental variable. Otherwise `TF_AZURE_USE_DEV_STORAGE` could be set to `True` to use Azurite emulator instead:\n"
{'fraction_non_alphanumeric': 0.16890499538122647, 'fraction_numerical': 0.007390037660768848, 'mean_word_length': 2.0628944504896625, 'pattern_counts': {'":': 187, '<': 55, '<?xml version=': 0, '>': 55, 'https://': 22, 'lorem ipsum': 0, 'www.': 12, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '633128', 'n_tokens_mistral': 4706, 'n_tokens_neox': 4596, 'n_words': 1099}