<commit_before>---
layout: post
author: csiu
date: 2017-05-03
title: "Day68: csv,conf,pdx (day2)"
categories: update
tags:
  - 100daysofcode
excerpt: csv,conf,pdx conference notes (day 2)
header-img: img/header/2017-05-03_portland.jpg
header-img-source: <NAME> (in Portland, csv,conf,v3)
---

*Today is the second (and final) day of the [CSV Conference](https://csvconf.com) in Portland and the following are notes I’ve taken regarding the talks I’ve attended:*

- *Contexts & Institutions (Keynote)*
- *Emoji Data Science*
- *Trends in Data in Time and Space*
- *Using Web Archives to Enrich the Live Web Experience Through storytelling*
- *Introducing d3.express, IDE for D3 (Keynote)*
- *Opinionated Analysis Development*
- *Applying software engineering to data analysis*
- *Dat (Data tables)*
- *Javascript for numeric computing*

## **Keynote |** Contexts & Institutions

Saving and indexing research quality federal environmental and climate data.

> <NAME>\\
> <a href="https://twitter.com/librlaurie" class="twitter-follow-button" data-show-count="false">Follow @librlaurie</a><script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>

- Laurie is a librarian involved with data rescue and is a "rogue scientist" to save climate data from Trump
- [DataRefuge](https://www.datarefuge.org)
  - formed over the concerns about the disappearance of research quality ("real") federal environmental and climate data. The risk is real.
  - The goal is to save copies of federal data
  - Would like to see events regarding designated communities (to identity data in communities that should be saved and to save it)
- How do you slice and dice federal information?
  - There is no list
  - You can organize by data format & size, agency that produced the data, covered by federal records laws, or most valuable & vulnerable first
  - No perfect way to slice and dice federal information
  - Types of vulnerabilities: legal, technical, political, use & uniqueness of the data
- Event paths
  - Seed & sort webpages (from WayBack Machine)
  - DataRefuge: harvest, check, bag, describe the data
  - Storytelling (one way of saving data, is to understand the data)
  - long trail (what to do in future to make it more sustainable)
- Codification: spreadsheets -> workflow -> web app -> system
  - Still work to be done eg. description is not quite clear what data is for
  - Shout out to Environmental Data & Governance Initiative (EDGI)
- [Dat Project](https://datproject.org) & notion of [Frictionless data](http://frictionlessdata.io)
- Multiple meaning of metadata
  - Metadata, paradata (universe of things one need to know to work with data) and software preservation
  - When data is given to another person, there are assumptions in place to what the other person knows<commit_msg>Add space<commit_after>---
layout: post
author: csiu
date: 2017-05-03
title: "Day68: csv,conf,pdx (day2)"
categories: update
tags:
  - 100daysofcode
excerpt: csv,conf,pdx conference notes (day 2)
header-img: img/header/2017-05-03_portland.jpg
header-img-source: <NAME> (in Portland, csv,conf,v3)
---

*Today is the second (and final) day of the [CSV Conference](https://csvconf.com) in Portland and the following are notes I’ve taken regarding the talks I’ve attended:*

- *Contexts & Institutions (Keynote)*
- *Emoji Data Science*
- *Trends in Data in Time and Space*
- *Using Web Archives to Enrich the Live Web Experience Through storytelling*
- *Introducing d3.express, IDE for D3 (Keynote)*
- *Opinionated Analysis Development*
- *Applying software engineering to data analysis*
- *Dat (Data tables)*
- *Javascript for numeric computing*

<br>

## **Keynote |** Contexts & Institutions

Saving and indexing research quality federal environmental and climate data.

> <NAME>\\
> <a href="https://twitter.com/librlaurie" class="twitter-follow-button" data-show-count="false">Follow @librlaurie</a><script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>

- Laurie is a librarian involved with data rescue and is a "rogue scientist" to save climate data from Trump
- [DataRefuge](https://www.datarefuge.org)
  - formed over the concerns about the disappearance of research quality ("real") federal environmental and climate data. The risk is real.
  - The goal is to save copies of federal data
  - Would like to see events regarding designated communities (to identity data in communities that should be saved and to save it)
- How do you slice and dice federal information?
  - There is no list
  - You can organize by data format & size, agency that produced the data, covered by federal records laws, or most valuable & vulnerable first
  - No perfect way to slice and dice federal information
  - Types of vulnerabilities: legal, technical, political, use & uniqueness of the data
- Event paths
  - Seed & sort webpages (from WayBack Machine)
  - DataRefuge: harvest, check, bag, describe the data
  - Storytelling (one way of saving data, is to understand the data)
  - long trail (what to do in future to make it more sustainable)
- Codification: spreadsheets -> workflow -> web app -> system
  - Still work to be done eg. description is not quite clear what data is for
  - Shout out to Environmental Data & Governance Initiative (EDGI)
- [Dat Project](https://datproject.org) & notion of [Frictionless data](http://frictionlessdata.io)
- Multiple meaning of metadata
  - Metadata, paradata (universe of things one need to know to work with data) and software preservation
  - When data is given to another person, there are assumptions in place to what the other person knows
{'fraction_non_alphanumeric': 0.08974358974358974, 'fraction_numerical': 0.009615384615384616, 'mean_word_length': 4.289077212806026, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 26, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3621899', 'n_tokens_mistral': 1621, 'n_tokens_neox': 1530, 'n_words': 713}