<commit_before>---
layout: page
permalink: /data/
title: "Data"
page-title: "Overview"
menu-item: true
breadcrumb: data
---

# About M-Lab's Data  

M-Lab stores our canonical raw data in [Google Cloud Storage](https://console.developers.google.com/storage/browser/m-lab/). Google Cloud Storage contains all of our data in raw form (no parsing). Additionally, we parse a copy of our raw data for a subset of our tests into [BigQuery tables](https://bigquery.cloud.google.com/queries/measurement-lab), so that the data is queryable through a SQL interface. 

Raw data is updated hourly and BigQuery tables are updated every day with data from M-Lab logs collected the day before. As a consequence, there is typically at least a 24-hour delay between data collection and data publication.

## Data License and Citing M-Lab Data

All data collected by M-Lab tests are available to the public without restriction under a [No Rights Reserved Creative Commons Zero Waiver](http://creativecommons.org/about/cc0).

Please cite M-Lab datasets as follows:
 
- The M-Lab <pre>test name</pre> Dataset, <pre>date range used</pre> <pre>M-Lab test URL</pre>

For example:

- The M-Lab NDT Dataset 2009-02-11 - 2015-12-21 [https://measurementlab.net/tools/ndt](https://measurementlab.net/tools/ndt)

## Getting Started with M-Lab Data

### Subscribe to the M-Lab Discuss List to Whitelist Your Account

Whether you want to access M-Lab's raw data, search it using BigQuery, or build your own application using it, follow these quick steps to get started:

- Choose a Google account to use for your queries or application
- Subscribe that account to [M-Lab's Discuss group](https://groups.google.com/a/measurementlab.net/forum/#!forum/discuss). 
- Enable billing for the account in the [Google Cloud Platform Console](https://console.developers.google.com/)

Any Gmail, Google Apps account (Business, Non-profit, etc.), can be used, however developer "service accounts" are not supported. In any case, the email address you choose should be the one also used with your Google Cloud Platform Console.

Subscribing to [M-Lab's Discuss group](https://groups.google.com/a/measurementlab.net/forum/#!forum/discuss) whitelists the account so no query charges are incurred when searching the M-Lab dataset.

Because BigQuery is a part of the Google Cloud Platform, in order to use BigQuery you must have a Google account and enable billing from within [Google Cloud Platform Console](https://console.developers.google.com/).

You will **not** be charged for queries against tables in the M-Lab dataset. M-Lab is committed to open data and offering our data free of charge is part of that commitment.

## Choose how you want to access M-Lab data

M-Lab data can be queried using the BigQuery web interface, using the command line tools available in the Google Cloud SDK, or by a custom application that uses the Google Cloud SDK. Raw data is also available. <commit_msg>formatting adjustment on citation format<commit_after>---
layout: page
permalink: /data/
title: "Data"
page-title: "Overview"
menu-item: true
breadcrumb: data
---

# About M-Lab's Data  

M-Lab stores our canonical raw data in [Google Cloud Storage](https://console.developers.google.com/storage/browser/m-lab/). Google Cloud Storage contains all of our data in raw form (no parsing). Additionally, we parse a copy of our raw data for a subset of our tests into [BigQuery tables](https://bigquery.cloud.google.com/queries/measurement-lab), so that the data is queryable through a SQL interface. 

Raw data is updated hourly and BigQuery tables are updated every day with data from M-Lab logs collected the day before. As a consequence, there is typically at least a 24-hour delay between data collection and data publication.

## Data License and Citing M-Lab Data

All data collected by M-Lab tests are available to the public without restriction under a [No Rights Reserved Creative Commons Zero Waiver](http://creativecommons.org/about/cc0).

Please cite M-Lab datasets as follows:
 
- The M-Lab `test name` Dataset, `date range used`. `M-Lab test URL`

For example:

- The M-Lab NDT Dataset 2009-02-11 - 2015-12-21 [https://measurementlab.net/tools/ndt](https://measurementlab.net/tools/ndt)

## Getting Started with M-Lab Data

### Subscribe to the M-Lab Discuss List to Whitelist Your Account

Whether you want to access M-Lab's raw data, search it using BigQuery, or build your own application using it, follow these quick steps to get started:

- Choose a Google account to use for your queries or application
- Subscribe that account to [M-Lab's Discuss group](https://groups.google.com/a/measurementlab.net/forum/#!forum/discuss). 
- Enable billing for the account in the [Google Cloud Platform Console](https://console.developers.google.com/)

Any Gmail, Google Apps account (Business, Non-profit, etc.), can be used, however developer "service accounts" are not supported. In any case, the email address you choose should be the one also used with your Google Cloud Platform Console.

Subscribing to [M-Lab's Discuss group](https://groups.google.com/a/measurementlab.net/forum/#!forum/discuss) whitelists the account so no query charges are incurred when searching the M-Lab dataset.

Because BigQuery is a part of the Google Cloud Platform, in order to use BigQuery you must have a Google account and enable billing from within [Google Cloud Platform Console](https://console.developers.google.com/).

You will **not** be charged for queries against tables in the M-Lab dataset. M-Lab is committed to open data and offering our data free of charge is part of that commitment.

## Choose how you want to access M-Lab data

M-Lab data can be queried using the BigQuery web interface, using the command line tools available in the Google Cloud SDK, or by a custom application that uses the Google Cloud SDK. Raw data is also available. 
{'fraction_non_alphanumeric': 0.07944321846885079, 'fraction_numerical': 0.006450517738923782, 'mean_word_length': 4.312894499549143, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 16, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '574071', 'n_tokens_mistral': 1665, 'n_tokens_neox': 1538, 'n_words': 788}