<commit_before>---
title: Progeny of Ten Socrata Datasets
created_at: 2013-07-19
kind: article
twitter_image: family.jpg
facebook_image: family.jpg
tweet_text: The World Bank contract award dataset has been forked 345 times. http://thomaslevine.com/!/socrata-genealogies/ @WBOpenFinances
description: It's cool what you can do when everyone's data analysis is logged and exposed publicly over the web.
twitter_description: It's cool what you can do when everyone's data analysis is logged and exposed publicly over the web.
facebook_text: More @socrata analysis! The World Bank major contract award dataset has been forked 345 times. http://thomaslevine.com/!/socrata-genealogies/ @WBOpenFinances
facebook_description: It's cool what you can do when everyone's data analysis is logged and exposed publicly over the web.
tags: ['socrata']
---
Governments and other organizations have recently been trying to open up their
data. Socrata's [Open Data Portal](http://www.socrata.com/open-data-portal/) software
is one tool that tries to help with this; an organization using Socrata is given a website
("portal") hosted by Socrata where they can upload their datasets and where
the public can download them.

I recently downloaded all of the metadata about all of the datasets from all
of the Socrata portals and then posted this [summary](/!/socrata-summary) of
the data. Now on to some deeper analysis.

## What is a dataset?
As the Twitters have pointed out,the dataset counts that I presented in my
initial summary are somewhat deceptive.


[![@tomschenkjr Tweets about Chicago's filters](tomschenkjr.png)](https://twitter.com/tomschenkjr/status/354010005504147456)

[![@SR_spatial Tweets about patterns of derived datasets](SR_spatial.png)](https://twitter.com/SR_spatial/status/354088265344749568)

Many of the things that I was calling a dataset can be seen as a
copy or a derivative of another dataset. In this post, I'll discuss

1. Socrata concepts and terminology
2. Ways that we can arrive at apparent duplicates in Socrata data
3. The progeny of ten Socrata datasets

## Socrata terminology
Most of my work on this for the past week has been figuring out
Socrata's terminology and schema. Let's define some Socrata terms.

### Everything is a view
{:#term-view}

When you go to the home page of a Socrata portal, you can
"Search & Browse Datasets and Views". This phrasing is sort
of wrong. "**view**" is just a generic concept that refers
to any sort of file or data that is presented to a user.

Everything in that list on the home page is a view. I haven't
yet explained what a dataset is, but a dataset is a type of
view. For example, the top two views in
[explore.data.gov](https://explore.data.gov/) are currently (July 17)
[White House Visitor Records Requests](https://explore.data.gov/dataset/White-House-Visitor-Records-Requests/644b-gaut)
and [U.S. Overseas Loans and Grants (Greenbook)](https://explore.data.gov/dataset/White-House-Visitor-Records-Requests/644b-gaut).

[![Search & Browse Datasets and Views](search-browse.png){:.wide}](https://explore.data.gov/)

You also get a list of "View Types". Below, I define some
of these view types.

### Datasets
{:#term-dataset}

Let's start with the **dataset**.
A dataset is when you get when you upload data to Socrata in one of
its supported tabular formats.

### Filtered views
{:#term-filtered-view}

Before I define "filtered views", I want to explain why they exist.
Socrata helps people publish their data by providing various APIs
for importing from different data sources, and Socrata helps people
consume data by providing a data analysis suite inside the web browser.
This includes maps and graphs and whatnot that you can embed in
websites rather than just in PDF documents.

Socrata also allows you to "Filter"
datasets. For example, here I filter the list of
[Public Works Volunteer Opportunities](https://data.oaklandnet.com/Environmental/Public-Works-Volunteer-Opportunities/sduu-bfki)
to include only opportunities on July 29.

![Filtering on date July 29](filter.png){:.wide}

[Here](https://data.oaklandnet.com/Environmental/Volunteer-Opportunities-on-July-29/vyhb-nqtw)'s the resulting filtered view.

**Filtered views** are queries on a dataset. The queries are represented internally in the
[SODA filter query language](http://dev.socrata.com/deprecated/querying-datasets).

### Charts and maps
{:#term-chart-and-map}

**Charts** and **maps** are also queries on a dataset.
The difference between filtered views, charts and maps is quite subtle.
They are all queries on datasets; they just display a different
visualization when you view them on the Socrata website.

### And more
{:#term-other-view-types}

There are other types of views, including views of non-tabular information,
but we don't need to know about them for now.

### Tables
{:#term-table}

![A table family, containing a dataset and several filtered views, charts and maps](<%= @item.identifier %>family.jpg){:.wide}
<!-- Icons from https://explore.data.gov/stylesheets/images/icons/type_icons_30.png?1 -->

There is also a concept of a **table**, and
it is somewhat abstract. Here are two ways of thinking of it.

First, a more conceptual explanation.
After someone uploads a dataset, a variety of filtered views,
charts and maps can emerge. I see this as a family of views,
with the parent being the original dataset and the ancestors
being all of the filtered views, charts and maps that make
SODA queries on the original dataset. In Socrata, this family
is called a table.

Next, a more technical explanation.
The data are stored in a table, and this table is not exposed directly to users. 
The most raw form of the table is exposed through a dataset, which is an empty
query on the dataset (equivalent to `SELECT * FROM table_name;`). Filtered views,
charts and maps act on the table rather than on the source dataset; they're just
like datasets, except that they include a query.

### Federation
Socrata doesn't provide a particularly obvious means for searching multiple
data portals at once. (This was part of my motivation for downloading all of
the datasets.) But it is possible for one data portal to include all of
another portal's datasets.

Sometimes, you'll see a view in the search & browse pane with a gray background,
instead of white. Hawaii has a bunch of these.

[![Hawaii data portal](hawaii.png){:.wide}](https://data.hawaii.gov/)

These views are "provided" by other portals through a process called
"federation". The destination portal (data.hawaii.gov in the above screenshot)
makes a request to the source portal (data.explore.gov in the above screenshot)
to federate the source portal's data.

This request shows up in the administrator interface for the source portal.
If the source portal accepts the request, all of the views from the source portal
are provided to the destination portal as in the screenshot above. Here are
[two](http://www.socrata.com/video/socrata-open-data-federation-demonstration/)
[videos](http://www.socrata.com/datagov/open-data-federation-video/) about that.

If you look closely, you'll notice that the federated views are actually just
links to the source portal; the views show up in the search, but they aren't
otherwise copied to the destination portal.

## Types of duplicate datasets
<commit_msg>federation<commit_after>---
title: Progeny of Ten Socrata Datasets
created_at: 2013-07-19
kind: article
twitter_image: family.jpg
facebook_image: family.jpg
tweet_text: The World Bank contract award dataset has been forked 345 times. http://thomaslevine.com/!/socrata-genealogies/ @WBOpenFinances
description: It's cool what you can do when everyone's data analysis is logged and exposed publicly over the web.
twitter_description: It's cool what you can do when everyone's data analysis is logged and exposed publicly over the web.
facebook_text: More @socrata analysis! The World Bank major contract award dataset has been forked 345 times. http://thomaslevine.com/!/socrata-genealogies/ @WBOpenFinances
facebook_description: It's cool what you can do when everyone's data analysis is logged and exposed publicly over the web.
tags: ['socrata']
---
Governments and other organizations have recently been trying to open up their
data. Socrata's [Open Data Portal](http://www.socrata.com/open-data-portal/) software
is one tool that tries to help with this; an organization using Socrata is given a website
("portal") hosted by Socrata where they can upload their datasets and where
the public can download them.

I recently downloaded all of the metadata about all of the datasets from all
of the Socrata portals and then posted this [summary](/!/socrata-summary) of
the data. Now on to some deeper analysis.

## What is a dataset?
As the Twitters have pointed out,the dataset counts that I presented in my
initial summary are somewhat deceptive.


[![@tomschenkjr Tweets about Chicago's filters](tomschenkjr.png)](https://twitter.com/tomschenkjr/status/354010005504147456)

[![@SR_spatial Tweets about patterns of derived datasets](SR_spatial.png)](https://twitter.com/SR_spatial/status/354088265344749568)

Many of the things that I was calling a dataset can be seen as a
copy or a derivative of another dataset. In this post, I'll discuss

1. Socrata concepts and terminology
2. Ways that we can arrive at apparent duplicates in Socrata data
3. The progeny of ten Socrata datasets

## Socrata terminology
Most of my work on this for the past week has been figuring out
Socrata's terminology and schema. Let's define some Socrata terms.

### Everything is a view
{:#term-view}

When you go to the home page of a Socrata portal, you can
"Search & Browse Datasets and Views". This phrasing is sort
of wrong. "**view**" is just a generic concept that refers
to any sort of file or data that is presented to a user.

Everything in that list on the home page is a view. I haven't
yet explained what a dataset is, but a dataset is a type of
view. For example, the top two views in
[explore.data.gov](https://explore.data.gov/) are currently (July 17)
[White House Visitor Records Requests](https://explore.data.gov/dataset/White-House-Visitor-Records-Requests/644b-gaut)
and [U.S. Overseas Loans and Grants (Greenbook)](https://explore.data.gov/dataset/White-House-Visitor-Records-Requests/644b-gaut).

[![Search & Browse Datasets and Views](search-browse.png){:.wide}](https://explore.data.gov/)

You also get a list of "View Types". Below, I define some
of these view types.

### Datasets
{:#term-dataset}

Let's start with the **dataset**.
A dataset is when you get when you upload data to Socrata in one of
its supported tabular formats.

### Filtered views
{:#term-filtered-view}

Before I define "filtered views", I want to explain why they exist.
Socrata helps people publish their data by providing various APIs
for importing from different data sources, and Socrata helps people
consume data by providing a data analysis suite inside the web browser.
This includes maps and graphs and whatnot that you can embed in
websites rather than just in PDF documents.

Socrata also allows you to "Filter"
datasets. For example, here I filter the list of
[Public Works Volunteer Opportunities](https://data.oaklandnet.com/Environmental/Public-Works-Volunteer-Opportunities/sduu-bfki)
to include only opportunities on July 29.

![Filtering on date July 29](filter.png){:.wide}

[Here](https://data.oaklandnet.com/Environmental/Volunteer-Opportunities-on-July-29/vyhb-nqtw)'s the resulting filtered view.

**Filtered views** are queries on a dataset. The queries are represented internally in the
[SODA filter query language](http://dev.socrata.com/deprecated/querying-datasets).

### Charts and maps
{:#term-chart-and-map}

**Charts** and **maps** are also queries on a dataset.
The difference between filtered views, charts and maps is quite subtle.
They are all queries on datasets; they just display a different
visualization when you view them on the Socrata website.

### And more
{:#term-other-view-types}

There are other types of views, including views of non-tabular information,
but we don't need to know about them for now.

### Tables
{:#term-table}

![A table family, containing a dataset and several filtered views, charts and maps](<%= @item.identifier %>family.jpg){:.wide}
<!-- Icons from https://explore.data.gov/stylesheets/images/icons/type_icons_30.png?1 -->

There is also a concept of a **table**, and
it is somewhat abstract. Here are two ways of thinking of it.

First, a more conceptual explanation.
After someone uploads a dataset, a variety of filtered views,
charts and maps can emerge. I see this as a family of views,
with the parent being the original dataset and the ancestors
being all of the filtered views, charts and maps that make
SODA queries on the original dataset. In Socrata, this family
is called a table.

Next, a more technical explanation.
The data are stored in a table, and this table is not exposed directly to users. 
The most raw form of the table is exposed through a dataset, which is an empty
query on the dataset (equivalent to `SELECT * FROM table_name;`). Filtered views,
charts and maps act on the table rather than on the source dataset; they're just
like datasets, except that they include a query.

### Federation
{:#term-federation}

Socrata doesn't provide a particularly obvious means for searching multiple
data portals at once. (This was part of my motivation for downloading all of
the datasets.) But it is possible for one data portal to include all of
another portal's datasets.

Sometimes, you'll see a view in the search & browse pane with a gray background,
instead of white. Hawaii has a bunch of these.

[![Hawaii data portal](hawaii.png){:.wide}](https://data.hawaii.gov/)

These views are "provided" by other portals through a process called
"federation". The destination portal (data.hawaii.gov in the above screenshot)
makes a request to the source portal (data.explore.gov in the above screenshot)
to federate the source portal's data.

This request shows up in the administrator interface for the source portal.
If the source portal accepts the request, all of the views from the source portal
are provided to the destination portal as in the screenshot above. Here are
[two](http://www.socrata.com/video/socrata-open-data-federation-demonstration/)
[videos](http://www.socrata.com/datagov/open-data-federation-video/) about that.

If you look closely, you'll notice that the federated views are actually just
links to the source portal; the views show up in the search, but they aren't
otherwise copied to the destination portal.

## Types of duplicate datasets

{'fraction_non_alphanumeric': 0.07701227830832197, 'fraction_numerical': 0.009549795361527967, 'mean_word_length': 4.65188897455667, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 20, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '395761', 'n_tokens_mistral': 4284, 'n_tokens_neox': 3982, 'n_words': 1999}