<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overall Firefox Engagement Ratio\n",
    "\n",
    "Compute the Engagement Ratio for the overall Firefox population as described in [Bug 1240849](https://bugzilla.mozilla.org/show_bug.cgi?id=1240849). The resulting data is shown on the [Firefox Dashboard](http://metrics.services.mozilla.com/firefox-dashboard/), and the more granular MAU and DAU values can be viewed via the [Diagnostic Data Viewer](https://metrics.services.mozilla.com/diagnostic-data-viewer).\n",
    "\n",
    "The actual Daily Active Users (DAU) and Monthly Active Users (MAU) computations are defined in [standards.py](https://github.com/mozilla/python_moztelemetry/blob/master/moztelemetry/standards.py) in the [python_moztelemetry](https://github.com/mozilla/python_moztelemetry) repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4 ms, sys: 4 ms, total: 8 ms\n",
      "Wall time: 21.9 s\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql import SQLContext\n",
    "from pyspark.sql.types import *\n",
    "from datetime import datetime as _datetime, timedelta, date\n",
    "import boto3\n",
    "import botocore\n",
    "import csv\n",
    "import os.path\n",
    "\n",
    "bucket = \"telemetry-parquet\"\n",
    "prefix = \"main_summary/v2\"\n",
    "%time dataset = sqlContext.read.load(\"s3://{}/{}\".format(bucket, prefix), \"parquet\")"
   ]
  },
  {
   "cell_type": "markdown",<commit_msg>Bug 1312493 - Use main_summary v3<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overall Firefox Engagement Ratio\n",
    "\n",
    "Compute the Engagement Ratio for the overall Firefox population as described in [Bug 1240849](https://bugzilla.mozilla.org/show_bug.cgi?id=1240849). The resulting data is shown on the [Firefox Dashboard](http://metrics.services.mozilla.com/firefox-dashboard/), and the more granular MAU and DAU values can be viewed via the [Diagnostic Data Viewer](https://metrics.services.mozilla.com/diagnostic-data-viewer).\n",
    "\n",
    "The actual Daily Active Users (DAU) and Monthly Active Users (MAU) computations are defined in [standards.py](https://github.com/mozilla/python_moztelemetry/blob/master/moztelemetry/standards.py) in the [python_moztelemetry](https://github.com/mozilla/python_moztelemetry) repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4 ms, sys: 4 ms, total: 8 ms\n",
      "Wall time: 21.9 s\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql import SQLContext\n",
    "from pyspark.sql.types import *\n",
    "from datetime import datetime as _datetime, timedelta, date\n",
    "import boto3\n",
    "import botocore\n",
    "import csv\n",
    "import os.path\n",
    "\n",
    "bucket = \"telemetry-parquet\"\n",
    "prefix = \"main_summary/v3\"\n",
    "%time dataset = sqlContext.read.load(\"s3://{}/{}\".format(bucket, prefix), \"parquet\")"
   ]
  },
  {
   "cell_type": "markdown",
{'fraction_non_alphanumeric': 0.16836124401913877, 'fraction_numerical': 0.01674641148325359, 'mean_word_length': 3.207547169811321, 'pattern_counts': {'":': 28, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2872576', 'n_tokens_mistral': 1207, 'n_tokens_neox': 1132, 'n_words': 275}