<commit_before># datapypes -  Pythonic abstractions for data pipelines

**datapypes** provides the building blocks to enable the following:

```python
News(query='stuff').latest(GoogleNews()).save(MongoDB())
```

```python
Post(text='I did something interesting').save(
  Twitter(username='tweet_user', password='<PASSWORD>'),
  Tumblr(username='tumblr_user', password='<PASSWORD>'))
```

```python
Track(artist="Meshuggah", title="New Millenium Cyanide Christ").get(
  SoundCloud(), Youtube()).save(
    Twitter(username='tweet_user', password='<PASSWORD>'),
    MongoDB(host='127.0.0.1')
```

See examples.py for implementation details.

## Classes

Build your pipeline by subclassing these classes:

* **Attribute** - A model attribute, i.e. title, author, date
* **Model** - A data model, i.e. Article, Video
* **Set** - A set of data models, i.e. News, Videos
* **Source** - A data source, i.e. Google News, Twitter<commit_msg>README updates<commit_after># datapypes -  Pythonic abstractions for data pipelines

**datapypes** provides the building blocks to enable the following:

```python
# Retrieve news from GoogleNews, store in MongoDB
News(query='stuff').latest(GoogleNews()).save(MongoDB())
```

```python
# Create a post, send to Twitter & Tumblr
Post(text='I did something interesting').save(
  Twitter(username='tweet_user', password='<PASSWORD>'),
  Tumblr(username='tumblr_user', password='<PASSWORD>'))
```

```python
# Get SoundCloud, YouTube links for a track, post to Twitter, save to MongoDB
Track(artist="Meshuggah", title="New Millenium Cyanide Christ").get(
  SoundCloud(), Youtube()).save(
    Twitter(username='tweet_user', password='<PASSWORD>'),
    MongoDB(host='127.0.0.1'))
```

See examples.py for implementation details.

## Classes

Build your pipeline by subclassing these classes:

* **Attribute** - A model attribute, i.e. title, author, date
* **Model** - A data model, i.e. Article, Video
* **Set** - A set of data models, i.e. News, Videos
* **Source** - A data source, i.e. Google News, Twitter
{'fraction_non_alphanumeric': 0.16110019646365423, 'fraction_numerical': 0.005893909626719057, 'mean_word_length': 5.466666666666667, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5945533', 'n_tokens_mistral': 668, 'n_tokens_neox': 629, 'n_words': 193}