<commit_before># Twitterverse
> Explore the Twitter conversations through users and their tweets and countries and their trending topics.

Application to harvest data from the Twitter API (using Python `tweepy` package) and to make that data easy to navigate, search and make sense of. SQLite is used for storage, with `SQLObject` as the ORM.

This project will be upgraded to _Python 3_.

## Features

Use the project for any of the features below.

### Fetch tweets

Use the _tweets_ section of this project.

Lookup tweets from the Twitter API. Get the tweet timeline of watched users. Or search for tweets which match a query such as phrases or hashtag - this data is only available from the API for a 7 day window.

Use the database to store and update search queries, you can easily rerun the same queries daily (or a few times a week if you need to always be up to date).

### Get trending topics

Using the trends scripts to find what topics are trending in your country or town at the current moment. See what other places are also talking about this topics.

There 62 countries and over 400 cities that Twitter provides trending data for. This project supports getting for all of them and groups places by country and continent.

Use the job manager tool in this project to get data just for the places you need,

Run the trending tool as a daily cron job. Then you have a history of trend data for places in the database and can create reports using your recent or historical data.

### Stream

Do a live stream of a search query and print out the tweets to the console.

## Documentation

Get started using these guides:

- [Installation](installation.md)
- [Usage](usage.md)

Use the menu sidebar for more details.<commit_msg>docs: Update docs/README.md<commit_after># Twitterverse
> Explore the Twitter conversations through users and their tweets and countries and their trending topics.

Application to harvest data from the Twitter API (using Python `tweepy` package) and to make that data easy to navigate, search and make sense of. SQLite is used for storage, with `SQLObject` as the ORM.

## Future

This project has been worked on and used for over 2 years so the ORM and command-line tools have been added to and improved a lot.

The project has been adapted to manager higher volumes of data.

Unit tests have been added but a lot more could be added.

The command-line tools could be refactor to be more consistent and easier to use.

The plan is to upgrade this project to Python 3 by 2020.


## Features

Access to the project is all through command-line tools. Use the project for any of the features below.

### Fetch tweets

Use the _tweets_ section of this project.

Lookup tweets from the Twitter API. Get the tweet timeline of watched users. Or search for tweets which match a query such as phrases or hashtag - this data is only available from the API for a 7 day window.

Use the database to store and update search queries, you can easily rerun the same queries daily (or a few times a week if you need to always be up to date).

You can use the _insert_ tool to add tweets directly to the DB using the ORM. There are in-progress _extract_ tools to handle saving to a CSV and later importing the CSV, to make fetching data faster. This means that searches on Twitter are shorter and therefore you can do more searches in a period, you can handle volumes of thousands of tweets easier.

### Get trending topics

Using the trends scripts to find what topics are trending in your country or town at the current moment. See what other places are also talking about this topics.

There 62 countries and over 400 cities that Twitter provides trending data for. This project supports getting for all of them and groups places by country and continent.

Use the job manager tool in this project to get data just for the places you need,

Run the trending tool as a daily cron job. Then you have a history of trend data for places in the database and can create reports using your recent or historical data.

### Stream

Do a live stream of a search query and print out the tweets to the console.
{'fraction_non_alphanumeric': 0.030810286268801553, 'fraction_numerical': 0.004609412906356138, 'mean_word_length': 4.021924482338611, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3340108', 'n_tokens_mistral': 1022, 'n_tokens_neox': 970, 'n_words': 709}