<commit_before>---
layout: page
permalink: /data/
title: "Data"
page-title: "Overview"
menu-item: true
breadcrumb: data
---

# Getting Started with M-Lab Data 

M-Lab stores our canonical data from all tools in Google's Big Query platform. We have prepared this guide to help new researchers to begin searching M-Lab data using Big Query's web interface or using the command line tools in Google's Cloud SDK. Data collected since January 2009 for three M-Lab tests is available in Big Query: [Network Diagnostic Tool (NDT)](https://console.developers.google.com/storage/browser/m-lab/ndt/), [Network Path and Application Diagnosis (NPAD)](https://console.developers.google.com/storage/browser/m-lab/npad/), and [Paris-Traceroute](https://console.developers.google.com/storage/browser/m-lab/paris-traceroute/). Big Query tables are updated every day with data from M-Lab logs collected the day before. As a consequence, there is typically at least a 24-hour delay between data collection and data publication.

## Subscribe to the M-Lab Discuss List to Whitelist Your Account

Before searching M-Lab data in Big Query, you first need to subscribe to the M-Lab Discuss List to whitelist your account.

When an account is subscribed to [M-Lab's Discuss group](https://groups.google.com/a/measurementlab.net/forum/#!forum/discuss), it is automatically whitelisted to query our data in Big Query. Any Gmail, Google Apps account (Business, Non-profit, etc.), can be used. Subscribe to the Discuss Google Group using the Gmail account or developer service account you wish to use to query M-Lab data.   

After subscribing to M-Lab Discuss, you will be able to query M-Lab's data at no cost.

## Choose how you want to access M-Lab data

M-Lab data can be queried using the Big Query web interface, using the command line tools available in the Google Cloud SDK, or by a custom application that uses the Google Cloud SDK. Raw data is also available. 

### Big Query Web Interface

M-Lab data is easily queried using Google’s web interface to Big Query. Learn more on our [BigQuery QuickStart page](/data/bq/quickstart/). 

### Big Query tools in the Google Cloud SDK

While M-Lab data is easily queried using Google’s web interface to Big Query, researchers may wish to run queries at the command line.

Download and Install [Google’s Cloud SDK](https://cloud.google.com/sdk/) using the Installation and Quick Start instructions for your operating system. 

After installation, authentication and restarting your terminal, Big Query’s command line tools are available in your shell. New to Big Query? We suggest that you next  consult [Google’s Big Query documentation](https://cloud.google.com/bigquery/what-is-bigquery) to get started.

Researchers should also use the [bq Command-Line Tool Quickstart](https://cloud.google.com/bigquery/bq-command-line-tool-quickstart) as reference documentation. 

#### Custom Google Cloud Application using the Big Query API

Given the resources and expertise, you can develop your own web application that uses M-Lab data and tools. To learn more about building your own custom application in the [Google Cloud Platform documentation](https://cloud.google.com/docs/).<commit_msg>updated data overview per reviewable comments<commit_after>---
layout: page
permalink: /data/
title: "Data"
page-title: "Overview"
menu-item: true
breadcrumb: data
---

# About M-Lab's Data  

M-Lab stores our canonical raw data in [Google Cloud Storage](https://console.developers.google.com/storage/browser/m-lab/) and makes that same data available for querying via [Google's BigQuery platform](https://bigquery.cloud.google.com/queries/measurement-lab). 

Data collected since January 2009 for four M-Lab tests is available:

  * [Network Diagnostic Tool (NDT)](/tools/ndt/)
  * [Network Path and Application Diagnosis (NPAD)](/tools/npad/)
  * [Paris-Traceroute](/tools/paris-traceroute/)
  * [SideSream](/tools/sidestream)

 Raw data is updated hourly and BigQuery tables are updated every day with data from M-Lab logs collected the day before. As a consequence, there is typically at least a 24-hour delay between data collection and data publication.

## Getting Started with M-Lab Data

### Subscribe to the M-Lab Discuss List to Whitelist Your Account

Whether you want to access M-Lab's raw data, search it using BigQuery, or build your own application using it, follow these quick steps to get started:

  * Choose a Google account to use for your queries or application
  * Subscribe that account to [M-Lab's Discuss group](https://groups.google.com/a/measurementlab.net/forum/#!forum/discuss). 
  * Enable billing for the account in the [Google Cloud Platform Console](https://console.developers.google.com/)

Any Gmail, Google Apps account (Business, Non-profit, etc.), can be used, however developer "service accounts" are not supported. In any case, the email address you choose should be the one also used with your Google Cloud Platform Console.

Subscribing to [M-Lab's Discuss group](https://groups.google.com/a/measurementlab.net/forum/#!forum/discuss) whitelists the account so no query charges are incurred when searching the M-Lab dataset.

Because BigQuery is a part of the Google Cloud Platform, in order to use BigQuery you must have a Google account and enable billing from within [Google Cloud Platform Console](https://console.developers.google.com/).

You will *not* be charged for queries against tables in the M-Lab dataset. M-Lab is committed to open data and offering our data free of charge is part of that commitment.

## Choose how you want to access M-Lab data

M-Lab data can be queried using the BigQuery web interface, using the command line tools available in the Google Cloud SDK, or by a custom application that uses the Google Cloud SDK. Raw data is also available. 

### BigQuery Web Interface

M-Lab data is easily queried using Google’s web interface to BigQuery. Learn more on our [BigQuery QuickStart page](/data/bq/quickstart/). 

### BigQuery tools in the Google Cloud SDK

You may also run queries at the command line using BigQuery tools in the Google Cloud SDK.

Download and Install [Google’s Cloud SDK](https://cloud.google.com/sdk/) using the Installation and Quick Start instructions for your operating system. 

After installation, authentication and restarting your terminal, BigQuery’s command line tools are available in your shell. If you are new to BigQuery, we suggest that you next consult the resources below to get started: 

  * [M-Lab's BigWuery QuickStart Guide](/data/bq/quickstart/)
  * [Google’s BigQuery documentation](https://cloud.google.com/bigquery/what-is-bigquery)
  * [Google's bq Command-Line Tool Quickstart documentation](https://cloud.google.com/bigquery/bq-command-line-tool-quickstart)

### Custom Google Cloud Application using the Big Query API

Given the resources and expertise, you can develop your own web application that uses M-Lab data and tools. To learn more about building your own custom application in the [Google Cloud Platform documentation](https://cloud.google.com/docs/).

_Telescope_ is an example of an application that M-Lab developed which uses the BigQuery Python API to download M-Lab data.

Telescope is available at Github here: https://github.com/m-lab/telescope
{'fraction_non_alphanumeric': 0.07678595913890031, 'fraction_numerical': 0.0016454134101192926, 'mean_word_length': 4.463670411985019, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 19, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3170299', 'n_tokens_mistral': 1990, 'n_tokens_neox': 1861, 'n_words': 938}