<commit_before>---
layout: page
title: Datasets: Wikipedia pagecounts-raw sorted by page (years 2007-2016)
modified: 2018-12-06T02:16:21.453254177+01:00
excerpt: "This dataset contains the pagecounts statistics for all the Wikimedia projects sorted by page 2007-2016"
image:
  feature: mountains-7.jpg
  credit: Dolomites, Mont<NAME> by <NAME> @ Flickr (CC-BY 2.0)
  creditlink: http://bit.ly/1ZbMlDJ
---

This dataset consists of hourly pagecounts for Wikipedia pages sorted by article, , ordered by `(project, page, timestamp)`. It has been created by processing Wikimedia's [`pagecounts-raw`](https://dumps.wikimedia.org/other/pagecounts-raw/) dataset.

The original dataset holds the desktop sites' pageview data (separately for every page) for the period from December 2007 to July 2016, with hourly granularity for all Wikipedia editions. More info about the original dataset are available on [Wikitech](https://wikitech.wikimedia.org/wiki/Analytics/Archive/Data/Pagecounts-raw). Note that **these are not unique visits**.

The CSV uses *spaces* as delimiter, without any form of escaping because it is not needed. It has 5 columns:

* `project`: the project name
* `page`: the page requested, url-escaped
* `timestamp`: the timestamp of the hour (format: `%Y%m%d-%H%M%S`)
* `count`: the number of times the page has been requested (in that hour)
* `bytes`: the number of bytes transferred (in that hour)

The original dataset has been normalized in the following ways:

* the `project` column has been converted to lowercase
* the `page` columns has been unquoted and then re-quoted according to RFC 1308,
using the following equivalent Python 3 code:
{% highlight python %}
import urllib.parse
page_unquoted = urllib.parse.unquote(page, encoding='utf-8',
    errors='replace')
page_requoted = urllib.parse.quote(page_unquoted)
{% endhighlight %}
* if two lines are now equal because of this normalization, their `count` and
`bytes` columns are summed up.

This dataset is split in many gzip'd files, each of them containing 1,000,000 records.

An `index` folder is included: for every month there is a file describing the the first record of every partial file.

Here's an excerpt of the file [`pagecounts-20071210-000000.gz`](https://dumps.wikimedia.org/other/pagecounts-raw/2007/2007-12/pagecounts-20071210-000000.gz) (lines 379737--379752)<a class="collapsible inactive" id='pagecounts-raw_sample' href="#"></a>:
<div class="collapsible" id='pagecounts-raw_sample'>
{% highlight text %}
en Albert_Einstein 540 540
en Albert_Einstein#_note-76 1 1
en Albert_Einstein%27s_brain 23 23
en Albert_Einstein's_brain 14 14
en Albert_Einstein_-_Wikipedia%2C_the_free_encyclopedia_files/shared.css 1 1
en Albert_Einstein_Academy_Charter_School 1 1
en Albert_Einstein_Award 8 8
en Albert_Einstein_Award#_note-tellerbio 1 1
en Albert_Einstein_College_of_Medicine 1 1
en Albert_Einstein_Medal 1 1
en Albert_Einstein_Memorial 5 5
en Albert_Einstein_Peace_Prize 9 9
en Albert_Einstein_Peace_Prize#_note-0 1 1
en Albert_Einstein_Peace_Prize#_note-2 1 1
en Albert_Einstein_Society 1 1
en Albert_Einstein_in_popular_culture 5 5
{% endhighlight %}
</div>

and in the processed dataset in file `2007-12/part-00082.gz` (lines 352686--352695) you can find:<a class="collapsible inactive" id='pagecounts-raw-sorted_sample' href="#"></a>.
<div class="collapsible" id='pagecounts-raw-sorted_sample'>
{% highlight text %}
en Albert_Einstein 20071209-190000 471 471
en Albert_Einstein 20071209-200000 545 545
en Albert_Einstein 20071209-210000 546 546
en Albert_Einstein 20071209-220000 497 497
en Albert_Einstein 20071209-230000 564 564
en Albert_Einstein 20071210-000000 540 540
en Albert_Einstein 20071210-010000 567 567
en Albert_Einstein 20071210-020000 547 547
en Albert_Einstein 20071210-030000 557 557
en Albert_Einstein 20071210-040000 624 624
{% endhighlight %}
</div>

## Download

This dataset can be downloaded in two different ways:

### dat (preferred method)

* coming soon

### HTTP

You can find the dataset on: [`cricca.disi.unitn.it/datasets/pagecounts-raw-sorted/`](http://cricca.disi.unitn.it/datasets/pagecounts-raw-sorted/).

## Code

* This dataset has been produced using [Apache Spark](https://spark.apache.org/)
  on [Microsoft ](http://doc.science.unitn.it/wiki/Il_Cluster_di_Scienze_%28Cisca-Cluster%29)
  with [this script](https://gist.github.com/youtux/13353862ac4227c4baf2) by <NAME>.
* The Python module [pagecounts-search](https://github.com/youtux/pagecounts-search)
  provides a command-line utility to query this dataset.
* The repository [wikipedia-pageviews-extraction](https://github.com/NGI4eu/engineroom-wikipedia-pageviews-extraction) contains a collection of utility to extract pageviews
data for groups of articles taking into account the existence of redirects.

## Authors

* <NAME> ([cristian.consonni(at)unitn.it](mailto:cristian.consonni(at)unitn(dot)it)), DISI - University of Trento
* <NAME>, DISI - University of Trento

### License

The [original dataset](https://dumps.wikimedia.org/other/pagecounts-raw/) was
published in the Public Domain ([Public Domain Mark 1.0](http://creativecommons.org/publicdomain/mark/1.0/)).

You can reuse this dataset under the same license.

### How to cite

Please cite this dataset as:

***<NAME>, <NAME>.***
***Wikipedia pagecounts-raw sorted by article.***
***doi coming soon***<commit_msg>Fix link in datasets/wikipedia-pagecounts-raw-sorted/index.md<commit_after>---
layout: page
title: Datasets: Wikipedia pagecounts-raw sorted by page (years 2007-2016)
modified: 2018-12-06T02:16:21.453254177+01:00
excerpt: "This dataset contains the pagecounts statistics for all the Wikimedia projects sorted by page 2007-2016"
image:
  feature: mountains-7.jpg
  credit: Dolomites, <NAME> by <NAME> @ Flickr (CC-BY 2.0)
  creditlink: http://bit.ly/1ZbMlDJ
---

This dataset consists of hourly pagecounts for Wikipedia pages sorted by article, , ordered by `(project, page, timestamp)`. It has been created by processing Wikimedia's [`pagecounts-raw`](https://dumps.wikimedia.org/other/pagecounts-raw/) dataset.

The original dataset holds the desktop sites' pageview data (separately for every page) for the period from December 2007 to July 2016, with hourly granularity for all Wikipedia editions. More info about the original dataset are available on [Wikitech](https://wikitech.wikimedia.org/wiki/Analytics/Archive/Data/Pagecounts-raw). Note that **these are not unique visits**.

The CSV uses *spaces* as delimiter, without any form of escaping because it is not needed. It has 5 columns:

* `project`: the project name
* `page`: the page requested, url-escaped
* `timestamp`: the timestamp of the hour (format: `%Y%m%d-%H%M%S`)
* `count`: the number of times the page has been requested (in that hour)
* `bytes`: the number of bytes transferred (in that hour)

The original dataset has been normalized in the following ways:

* the `project` column has been converted to lowercase
* the `page` columns has been unquoted and then re-quoted according to RFC 1308,
using the following equivalent Python 3 code:
{% highlight python %}
import urllib.parse
page_unquoted = urllib.parse.unquote(page, encoding='utf-8',
    errors='replace')
page_requoted = urllib.parse.quote(page_unquoted)
{% endhighlight %}
* if two lines are now equal because of this normalization, their `count` and
`bytes` columns are summed up.

This dataset is split in many gzip'd files, each of them containing 1,000,000 records.

An `index` folder is included: for every month there is a file describing the the first record of every partial file.

Here's an excerpt of the file [`pagecounts-20071210-000000.gz`](https://dumps.wikimedia.org/other/pagecounts-raw/2007/2007-12/pagecounts-20071210-000000.gz) (lines 379737--379752)<a class="collapsible inactive" id='pagecounts-raw_sample' href="#"></a>:
<div class="collapsible" id='pagecounts-raw_sample'>
{% highlight text %}
en Albert_Einstein 540 540
en Albert_Einstein#_note-76 1 1
en Albert_Einstein%27s_brain 23 23
en Albert_Einstein's_brain 14 14
en Albert_Einstein_-_Wikipedia%2C_the_free_encyclopedia_files/shared.css 1 1
en Albert_Einstein_Academy_Charter_School 1 1
en Albert_Einstein_Award 8 8
en Albert_Einstein_Award#_note-tellerbio 1 1
en Albert_Einstein_College_of_Medicine 1 1
en Albert_Einstein_Medal 1 1
en Albert_Einstein_Memorial 5 5
en Albert_Einstein_Peace_Prize 9 9
en Albert_Einstein_Peace_Prize#_note-0 1 1
en Albert_Einstein_Peace_Prize#_note-2 1 1
en Albert_Einstein_Society 1 1
en Albert_Einstein_in_popular_culture 5 5
{% endhighlight %}
</div>

and in the processed dataset in file `2007-12/part-00082.gz` (lines 352686--352695) you can find:<a class="collapsible inactive" id='pagecounts-raw-sorted_sample' href="#"></a>.
<div class="collapsible" id='pagecounts-raw-sorted_sample'>
{% highlight text %}
en Albert_Einstein 20071209-190000 471 471
en Albert_Einstein 20071209-200000 545 545
en Albert_Einstein 20071209-210000 546 546
en Albert_Einstein 20071209-220000 497 497
en Albert_Einstein 20071209-230000 564 564
en Albert_Einstein 20071210-000000 540 540
en Albert_Einstein 20071210-010000 567 567
en Albert_Einstein 20071210-020000 547 547
en Albert_Einstein 20071210-030000 557 557
en Albert_Einstein 20071210-040000 624 624
{% endhighlight %}
</div>

## Download

This dataset can be downloaded in two different ways:

### dat (preferred method)

* coming soon

### HTTP

You can find the dataset on: [`cricca.disi.unitn.it/datasets/pagecounts-raw-sorted/`](http://cricca.disi.unitn.it/datasets/pagecounts-raw-sorted/).

## Code

* This dataset has been produced using [Apache Spark](https://spark.apache.org/)
  on [Microsoft Azure](https://azure.microsoft.com/)
  with [this script](https://gist.github.com/youtux/13353862ac4227c4baf2) by <NAME>.
* The Python module [pagecounts-search](https://github.com/youtux/pagecounts-search)
  provides a command-line utility to query this dataset.
* The repository [wikipedia-pageviews-extraction](https://github.com/NGI4eu/engineroom-wikipedia-pageviews-extraction) contains a collection of utility to extract pageviews
data for groups of articles taking into account the existence of redirects.

## Authors

* <NAME> ([cristian.consonni(at)unitn.it](mailto:cristian.consonni(at)unitn(dot)it)), DISI - University of Trento
* <NAME>, DISI - University of Trento

### License

The [original dataset](https://dumps.wikimedia.org/other/pagecounts-raw/) was
published in the Public Domain ([Public Domain Mark 1.0](http://creativecommons.org/publicdomain/mark/1.0/)).

You can reuse this dataset under the same license.

### How to cite

Please cite this dataset as:

***<NAME>, <NAME>.***
***Wikipedia pagecounts-raw sorted by article.***
***doi coming soon***
{'fraction_non_alphanumeric': 0.10146277834715364, 'fraction_numerical': 0.07639988819528556, 'mean_word_length': 5.370326409495549, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 17, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4620170', 'n_tokens_mistral': 4198, 'n_tokens_neox': 3337, 'n_words': 1126}