<commit_before># Tweepy (fork) Read

Unlike Facebook, Twitter is admirably transparent and open about its data.
Its [Terms of Service](https://twitter.com/tos?PHPSESSID=57a411f70b1964a2bc78b82638ba1843)
state that what you put up there is instantly publicly available and contain the
'tip' that "What you say on Twitter may be viewed all around the world instantly. You are what you Tweet!" 
This allows the company to offer open access methods
for people to tap into the 'live stream' and use the information for 3rd party applications.

This openness and the growing interest in Tweets as a useful
data source about peoples' perceptions of the world has led to a mass of 3rd party
applications to help access this vast stream of data. Many of these are
open source and most take advantage of either the
[Twitter REST API](https://dev.twitter.com/docs/api) (which is
severely [rate limited](https://dev.twitter.com/docs/rate-limiting/1.1)) or the
[Twitter Streaming API](https://dev.twitter.com/docs/api/streaming), 
which allows more information to be taken in, based on either a search term
or a bounding box - a spatial filter.

In this post we demonstrate a few methods of accessing this rich Twitter data.
The majority are open source and many are evolving thanks largely to the social
coding site GitHub. In the final section we demonstrate some interesting potential
applications of harvesting Twitter data, with an illustrated example of Tweets about
road safety.

## Tweepy

The first method we use is [Tweepy](https://github.com/tweepy/tweepy),
a Python library for accessing the Twitter APIs (I'm not sure if
this is can access only the REST or Streaming API, or both).
Installation is easy. Once Python's package manager is installed
(via `sudo apt-get install python-pip` in Debian/Ubuntu systems), 
installation of Tweepy and all its dependencies is a doddle:

```{python}
pip install tweepy
```

Now that it is installed, how does it work?
Well Python must *import* the Tweepy package before it is used.
A critical stage is to find and fill-in your Twitter API
user details into the `streaming.py` example file in the Tweety
GitHub repository:

```{}
consumer_key=""
consumer_secret=""
access_token=""
access_token_secret=""
```

All four codes must be entered correctly between the quotation "" symbols.
Then the rest of the `streaming.py` file can be altered for your needs
A basic example is provided [here](https://github.com/Robinlovelace/tweepy/blob/master/streaming-leeds.py),
which contains the following line of code to filter the tweets by geographical
location:

```{}<commit_msg>updating readme<commit_after># Twitter Listener for Python

**PLease note:** this is a collection of scripts and libraries to use Python to connect to the Twitter Streaming API. Most of the work is done by the fantastic [tweepy](https://github.com/tweepy/tweepy) library ([http://www.tweepy.org/](http://www.tweepy.org/)) which I have had nothing to do with and can't take any credit for! It also contains work from <NAME> [tweepy fork](https://github.com/Robinlovelace/tweepy), particularly R stuff and some of the text in this README.

This README has some instructions about how to use the scripts here and start listenning to the Streaming API. It assumes a reasonable knowledge of how to use the command line and run Python programs.

## About Twitter data

_(Skip this section if you're not interested, it doesn't contain any instructions about how to use the software)_.

Twitter is admirably transparent and open about its data. Its [Terms of Service](https://twitter.com/tos?PHPSESSID=57a411f70b1964a2bc78b82638ba1843)
state that what you put up there is instantly publicly available and contain the 'tip' that "What you say on Twitter may be viewed all around the world instantly. You are what you Tweet!" This allows the company to offer open access methods for people to tap into the 'live stream' and use the information for 3rd party applications.

Twitter provides two APIs to access the data. The [REST API](https://dev.twitter.com/docs/api) (which is severly [rate limited](https://dev.twitter.com/docs/rate-limiting/1.1)) can be used to run searches on recent tweets. On the other hand, the
[Streaming API](https://dev.twitter.com/docs/api/streaming) provides tweets in real time that match some query. Queries are quite flexible and can be set up to listen for particular words/phrases, for messages from particular users, or for messages that originate in certain locations. We primarily use location as the search parameter below, but the tweepy library provides a mechanism to run any kind of query supported by the API.

## Pre-requisites

### Download this program!

Download all the ```twitter_listener``` files, by clicking on the 'download' link in GitHub, or following [this link](https://github.com/nickmalleson/twitter_listener/archive/master.zip). Unzip the files somewhere sensible, then navigate the directory using the command-line.

### Python

Before running the program, you need to get some software. Firstly, you'll need [Python](https://www.python.org/). On a mac or linux you probably have this already, but on Windows you'll probably need to download it. Follow the instructions on the [download page](https://www.python.org/downloads/).

### Python libraries

Once you have python, you need to install an extra library called <code>configparser</code>. Python has a package manager that you can use. On a Mac or Linux try:

```
sudo pip install configparser
```

and if that doesn't work, try:

```
sudo easy_install configparser
```

If neither of those work, you need to install the [setuptools](https://pypi.python.org/pypi/setuptools) package manager first (although I'm surprised that it didn't come with your Python distribution.

I'm not sure how to install Python libraries on Windows. It looks like [pip-Win](https://pypi.python.org/pypi/pip-Win/1.4) is what you need.

Once you have installed configparser, see if Python is ready to go by trying to run the ```streaming.py``` program that was part of the zip file you downloaded:

```
python streaming.py
```

If you get the following error: 
```Error ./credentials.ini doesn't look like a file. See the README for details. ``` 
then Python is working fine.

## Twitter API Key

Before you can access the Twitter API, you need a key. The details from the key need to be stored in a file called ```credentials.ini'''.





```{}
consumer_key=""
consumer_secret=""
access_token=""
access_token_secret=""
```

All four codes must be entered correctly between the quotation "" symbols.
Then the rest of the `streaming.py` file can be altered for your needs
A basic example is provided [here](https://github.com/Robinlovelace/tweepy/blob/master/streaming-leeds.py),
which contains the following line of code to filter the tweets by geographical
location:

```{}
{'fraction_non_alphanumeric': 0.07631808913486746, 'fraction_numerical': 0.00801048645499563, 'mean_word_length': 4.670520231213873, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 18, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1285955', 'n_tokens_mistral': 1932, 'n_tokens_neox': 1814, 'n_words': 951}