<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Triggering a Cloud Composer Pipeline with a Google Cloud Function\n",
    "\n",
    "In this advanced lab you will learn how to create and run an [Apache Airflow](http://airflow.apache.org/) workflow in Cloud Composer that completes the following tasks:\n",
    "- Watches for new CSV data to be uploaded to a [Cloud Storage](https://cloud.google.com/storage/docs/) bucket\n",
    "- A [Cloud Function](https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf#getting_the_client_id) call triggers the [Cloud Composer Airflow DAG](https://cloud.google.com/composer/docs/how-to/using/writing-dags) to run when a new file is detected \n",
    "- The workflow finds the input file that triggered the workflow and executes a [Cloud Dataflow](https://cloud.google.com/dataflow/) job to transform and output the data to BigQuery  \n",
    "- Moves the original input file to a different Cloud Storage bucket for storing processed files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part One: Create Cloud Composer environment and workflow\n",
    "First, create a Cloud Composer environment by doing the following:\n",
    "1. In the Navigation menu under Big Data, select **Composer**\n",
    "2. Select **Create**\n",
    "3. Set the following parameters:\n",
    "    - Name: composer-workflow\n",
    "    - Location: us-central1\n",
    "    - Other values at defaults\n",
    "4. Select **Create**\n",
    "\n",
    "The environment creation process is completed when the green checkmark displays to the left of the environment name on the Environments page in the GCP Console.\n",
    "It can take up to 20 minutes for the environment to complete the setup process. Move on to the next section - Create Cloud Storage buckets and BigQuery dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set environment variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "PROJECT = 'PROJECT' # REPLACE WITH YOUR PROJECT ID\n",
    "REGION = 'us-central1' # REPLACE WITH YOUR REGION e.g. us-central1\n",<commit_msg>aligning composer names<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Triggering a Cloud Composer Pipeline with a Google Cloud Function\n",
    "\n",
    "In this advanced lab you will learn how to create and run an [Apache Airflow](http://airflow.apache.org/) workflow in Cloud Composer that completes the following tasks:\n",
    "- Watches for new CSV data to be uploaded to a [Cloud Storage](https://cloud.google.com/storage/docs/) bucket\n",
    "- A [Cloud Function](https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf#getting_the_client_id) call triggers the [Cloud Composer Airflow DAG](https://cloud.google.com/composer/docs/how-to/using/writing-dags) to run when a new file is detected \n",
    "- The workflow finds the input file that triggered the workflow and executes a [Cloud Dataflow](https://cloud.google.com/dataflow/) job to transform and output the data to BigQuery  \n",
    "- Moves the original input file to a different Cloud Storage bucket for storing processed files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part One: Create Cloud Composer environment and workflow\n",
    "First, create a Cloud Composer environment if you don't have one already by doing the following:\n",
    "1. In the Navigation menu under Big Data, select **Composer**\n",
    "2. Select **Create**\n",
    "3. Set the following parameters:\n",
    "    - Name: mlcomposer\n",
    "    - Location: us-central1\n",
    "    - Other values at defaults\n",
    "4. Select **Create**\n",
    "\n",
    "The environment creation process is completed when the green checkmark displays to the left of the environment name on the Environments page in the GCP Console.\n",
    "It can take up to 20 minutes for the environment to complete the setup process. Move on to the next section - Create Cloud Storage buckets and BigQuery dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set environment variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "PROJECT = 'PROJECT' # REPLACE WITH YOUR PROJECT ID\n",
    "REGION = 'us-central1' # REPLACE WITH YOUR REGION e.g. us-central1\n",
{'fraction_non_alphanumeric': 0.1307443365695793, 'fraction_numerical': 0.003883495145631068, 'mean_word_length': 3.2767527675276753, 'pattern_counts': {'":': 30, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4828389', 'n_tokens_mistral': 1421, 'n_tokens_neox': 1371, 'n_words': 507}